drop

• Description:

With system performance on existing reading comprehension benchmarks nearing or surpassing human performance, we need a new, hard dataset that improves systems' capabilities to actually read paragraphs of text. DROP is a crowdsourced, adversarially-created, 96k-question benchmark, in which a system must resolve references in a question, perhaps to multiple input positions, and perform discrete operations over them (such as addition, counting, or sorting). These operations require a much more comprehensive understanding of the content of paragraphs than what was necessary for prior datasets.

• Additional Documentation: Explore on Papers With Code

• Homepage: https://allennlp.org/drop

• Source code: tfds.text.drop.Drop

• Versions:

  • 1.0.0: Initial release.
  • 2.0.0 (default): Add all options for the answers.

• Download size: 7.92 MiB

• Dataset size: 116.24 MiB

• Auto-cached (documentation): Yes

• Splits:

Split	Examples
'dev'	9,536
'train'	77,409

• Feature structure:

FeaturesDict({
    'answer': Text(shape=(), dtype=string),
    'passage': Text(shape=(), dtype=string),
    'query_id': Text(shape=(), dtype=string),
    'question': Text(shape=(), dtype=string),
    'validated_answers': Sequence(Text(shape=(), dtype=string)),
})

• Feature documentation:

Feature	Class	Shape	Dtype	Description
	FeaturesDict
answer	Text		string
passage	Text		string
query_id	Text		string
question	Text		string
validated_answers	Sequence(Text)	(None,)	string

• Supervised keys (See as_supervised doc): None

• Figure (tfds.show_examples): Not supported.

• Examples (tfds.as_dataframe):

• Citation:

@inproceedings{Dua2019DROP,
  author={Dheeru Dua and Yizhong Wang and Pradeep Dasigi and Gabriel Stanovsky and Sameer Singh and Matt Gardner},
  title={  {DROP}: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs},
  booktitle={Proc. of NAACL},
  year={2019}
}