Shallow syntax
Shallow syntactic tasks provide an analysis of a text on the level of the syntactic structure of the text.
Chunking
Chunking, also known as shallow parsing, identifies continuous spans of tokens that form syntactic units such as noun phrases or verb phrases.
Example:
| Vinken | , | 61 | years | old |
|---|---|---|---|---|
| B-NLP | I-NP | I-NP | I-NP | I-NP |
Penn Treebank
The Penn Treebank is typically used for evaluating chunking. Sections 15-18 are used for training, section 19 for development, and and section 20 for testing. Models are evaluated based on F1.
| Model | F1 score | Paper | Source |
|---|---|---|---|
| ACE + fine-tune (Wang et al., 2020) | 97.30 | Automated Concatenation of Embeddings for Structured Prediction | Official |
| Flair embeddings (Akbik et al., 2018) | 96.72 | Contextual String Embeddings for Sequence Labeling | Flair |
| JMT (Hashimoto et al., 2017) | 95.77 | A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks | |
| Low supervision (Søgaard and Goldberg, 2016) | 95.57 | Deep multi-task learning with low level tasks supervised at lower layers | |
| Suzuki and Isozaki (2008) | 95.15 | Semi-Supervised Sequential Labeling and Segmentation using Giga-word Scale Unlabeled Data | |
| NCRF++ (Yang and Zhang, 2018) | 95.06 | NCRF++: An Open-source Neural Sequence Labeling Toolkit | NCRF++ |
CoNLL 2003
Though the CoNLL 2003 datasets are typically used for evaluating NER, the datasets can be used for evaluating chunking as well. The dataset split is official standard split. Models are evaluated based on F1.
| Model | English | German | Paper | Source |
|---|---|---|---|---|
| ACE + fine-tune (Wang et al., 2020) | 92.5 | 95.0 | Automated Concatenation of Embeddings for Structured Prediction | Official |
| Flair + BERT + Word + Char embeddings (Wang et al., 2020) | 92.0 | 94.4 | More Embeddings, Better Sequence Labelers? | |
| Word + Char + MFVI (Wang et al., 2020) | 91.71 | 94.03 | AIN: Fast and Accurate Sequence Labeling with Approximate Inference Network | Official |
Resolving the Scope and focus of negation
Scope of negation is the part of the meaning that is negated and focus the part of the scope that is most prominently negated (Huddleston and Pullum 2002).
Example:
[John had] never [said %as much% before].
Scope is enclosed in square brackets and focus marked between % signs.
The CD-SCO (Conan Doyle Scope) dataset is for scope detection. The PB-FOC (PropBank Focus) dataset is for focus detection. The public leaderboard is available on the *SEM Shared Task 2012 website.