View on GitHub

NLP-progress

Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the most common NLP tasks.

Shallow syntax

Shallow syntactic tasks provide an analysis of a text on the level of the syntactic structure of the text.

Chunking

Chunking, also known as shallow parsing, identifies continuous spans of tokens that form syntactic units such as noun phrases or verb phrases.

Example:

Vinken , 61 years old
B-NLP I-NP I-NP I-NP I-NP

Penn Treebank

The Penn Treebank is typically used for evaluating chunking. Sections 15-18 are used for training, section 19 for development, and and section 20 for testing. Models are evaluated based on F1.

Model F1 score Paper Source
ACE + fine-tune (Wang et al., 2020) 97.30 Automated Concatenation of Embeddings for Structured Prediction Official
Flair embeddings (Akbik et al., 2018) 96.72 Contextual String Embeddings for Sequence Labeling Flair
JMT (Hashimoto et al., 2017) 95.77 A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks  
Low supervision (Søgaard and Goldberg, 2016) 95.57 Deep multi-task learning with low level tasks supervised at lower layers  
Suzuki and Isozaki (2008) 95.15 Semi-Supervised Sequential Labeling and Segmentation using Giga-word Scale Unlabeled Data  
NCRF++ (Yang and Zhang, 2018) 95.06 NCRF++: An Open-source Neural Sequence Labeling Toolkit NCRF++

CoNLL 2003

Though the CoNLL 2003 datasets are typically used for evaluating NER, the datasets can be used for evaluating chunking as well. The dataset split is official standard split. Models are evaluated based on F1.

Model English German Paper Source
ACE + fine-tune (Wang et al., 2020) 92.5 95.0 Automated Concatenation of Embeddings for Structured Prediction Official
Flair + BERT + Word + Char embeddings (Wang et al., 2020) 92.0 94.4 More Embeddings, Better Sequence Labelers?  
Word + Char + MFVI (Wang et al., 2020) 91.71 94.03 AIN: Fast and Accurate Sequence Labeling with Approximate Inference Network Official

Resolving the Scope and focus of negation

Scope of negation is the part of the meaning that is negated and focus the part of the scope that is most prominently negated (Huddleston and Pullum 2002).

Example:

[John had] never [said %as much% before].

Scope is enclosed in square brackets and focus marked between % signs.

The CD-SCO (Conan Doyle Scope) dataset is for scope detection. The PB-FOC (PropBank Focus) dataset is for focus detection. The public leaderboard is available on the *SEM Shared Task 2012 website.

Go back to the README