View on GitHub


Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the most common NLP tasks.

Shallow syntax

Shallow syntactic tasks provide an analysis of a text on the level of the syntactic structure of the text.


Chunking, also known as shallow parsing, identifies continuous spans of tokens that form syntactic units such as noun phrases or verb phrases.


Vinken , 61 years old

Penn Treebank

The Penn Treebank is typically used for evaluating chunking. Sections 15-18 are used for training, section 19 for development, and and section 20 for testing. Models are evaluated based on F1.

Model F1 score Paper / Source  
JMT (Hashimoto et al., 2017) 95.77 A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks  
Low supervision (Søgaard and Goldberg, 2016) 95.57 Deep multi-task learning with low level tasks supervised at lower layers  
Suzuki and Isozaki (2008) 95.15 Semi-Supervised Sequential Labeling and Segmentation using Giga-word Scale Unlabeled Data  
NCRF++ (Yang and Zhang, 2018) 95.06 NCRF++: An Open-source Neural Sequence Labeling Toolkit NCRF++

Resolving the Scope and focus of negation

Scope of negation is the part of the meaning that is negated and focus the part of the scope that is most prominently negated (Huddleston and Pullum 2002).


[John had] never [said %as much% before].

Scope is enclosed in square brackets and focus marked between % signs.

The CD-SCO (Conan Doyle Scope) dataset is for scope detection. The PB-FOC (PropBank Focus) dataset is for focus detection. The public leaderboard is available on the *SEM Shared Task 2012 website.

Go back to the README