TFDS now supports the Croissant 🥐 format! Read the documentation to know more.

blimp

Description:

BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics. The data is automatically generated according to expert-crafted grammars.

Additional Documentation: Explore on Papers With Code
Homepage: https://github.com/alexwarstadt/blimp/tree/master/
Source code: tfds.datasets.blimp.Builder
Versions:
- 0.1.0 (default): No release notes.
Auto-cached (documentation): Yes
Splits:

Split	Examples
`'train'`	1,000

Feature structure:

FeaturesDict({
    'UID': Text(shape=(), dtype=string),
    'field': Text(shape=(), dtype=string),
    'lexically_identical': bool,
    'linguistics_term': Text(shape=(), dtype=string),
    'one_prefix_method': bool,
    'pair_id': int32,
    'sentence_bad': Text(shape=(), dtype=string),
    'sentence_good': Text(shape=(), dtype=string),
    'simple_LM_method': bool,
    'two_prefix_method': bool,
})

Feature documentation:

Feature	Class	Dtype
	FeaturesDict
UID	Text	string
field	Text	string
lexically_identical	Tensor	bool
linguistics_term	Text	string
one_prefix_method	Tensor	bool
pair_id	Tensor	int32
sentence_bad	Text	string
sentence_good	Text	string
simple_LM_method	Tensor	bool
two_prefix_method	Tensor	bool

Supervised keys (See as_supervised doc): None
Figure (tfds.show_examples): Not supported.
Citation:

@article{warstadt2019blimp,
  title={BLiMP: A Benchmark of Linguistic Minimal Pairs for English},
  author={Warstadt, Alex and Parrish, Alicia and Liu, Haokun and Mohananey, Anhad and Peng, Wei, and Wang, Sheng-Fu and Bowman, Samuel R},
  journal={arXiv preprint arXiv:1912.00582},
  year={2019}
}

blimp/adjunct_island (default config)

Config description: This configuration includes the paradigm adjunct_island.
Download size: 350.86 KiB
Dataset size: 366.47 KiB
Examples (tfds.as_dataframe):

blimp/anaphor_gender_agreement

Config description: This configuration includes the paradigm anaphor_gender_agreement.
Download size: 426.51 KiB
Dataset size: 332.31 KiB
Examples (tfds.as_dataframe):

blimp/anaphor_number_agreement

Config description: This configuration includes the paradigm anaphor_number_agreement.
Download size: 440.29 KiB
Dataset size: 341.07 KiB
Examples (tfds.as_dataframe):

blimp/animate_subject_passive

Config description: This configuration includes the paradigm animate_subject_passive.
Download size: 451.46 KiB
Dataset size: 345.50 KiB
Examples (tfds.as_dataframe):

blimp/animate_subject_trans

Config description: This configuration includes the paradigm animate_subject_trans.
Download size: 422.95 KiB
Dataset size: 329.27 KiB
Examples (tfds.as_dataframe):

blimp/causative

Config description: This configuration includes the paradigm causative.
Download size: 309.73 KiB
Dataset size: 324.36 KiB
Examples (tfds.as_dataframe):

blimp/complex_NP_island

Config description: This configuration includes the paradigm complex_NP_island.
Download size: 383.17 KiB
Dataset size: 398.78 KiB
Examples (tfds.as_dataframe):

blimp/coordinate_structure_constraint_complex_left_branch

Config description: This configuration includes the paradigm coordinate_structure_constraint_complex_left_branch.
Download size: 558.30 KiB
Dataset size: 410.43 KiB
Examples (tfds.as_dataframe):

blimp/coordinate_structure_constraint_object_extraction

Config description: This configuration includes the paradigm coordinate_structure_constraint_object_extraction.
Download size: 357.47 KiB
Dataset size: 372.10 KiB
Examples (tfds.as_dataframe):

blimp/determiner_noun_agreement_1

Config description: This configuration includes the paradigm determiner_noun_agreement_1.
Download size: 457.66 KiB
Dataset size: 356.93 KiB
Examples (tfds.as_dataframe):

blimp/determiner_noun_agreement_2

Config description: This configuration includes the paradigm determiner_noun_agreement_2.
Download size: 477.40 KiB
Dataset size: 357.01 KiB
Examples (tfds.as_dataframe):

blimp/determiner_noun_agreement_irregular_1

Config description: This configuration includes the paradigm determiner_noun_agreement_irregular_1.
Download size: 463.80 KiB
Dataset size: 365.08 KiB
Examples (tfds.as_dataframe):

blimp/determiner_noun_agreement_irregular_2

Config description: This configuration includes the paradigm determiner_noun_agreement_irregular_2.
Download size: 479.35 KiB
Dataset size: 361.76 KiB
Examples (tfds.as_dataframe):

blimp/determiner_noun_agreement_with_adj_2

Config description: This configuration includes the paradigm determiner_noun_agreement_with_adj_2.
Download size: 514.46 KiB
Dataset size: 379.92 KiB
Examples (tfds.as_dataframe):

blimp/determiner_noun_agreement_with_adj_irregular_1

Config description: This configuration includes the paradigm determiner_noun_agreement_with_adj_irregular_1.
Download size: 487.95 KiB
Dataset size: 384.67 KiB
Examples (tfds.as_dataframe):

blimp/determiner_noun_agreement_with_adj_irregular_2

Config description: This configuration includes the paradigm determiner_noun_agreement_with_adj_irregular_2.
Download size: 516.14 KiB
Dataset size: 384.54 KiB
Examples (tfds.as_dataframe):

blimp/determiner_noun_agreement_with_adjective_1

Config description: This configuration includes the paradigm determiner_noun_agreement_with_adjective_1.
Download size: 492.85 KiB
Dataset size: 385.25 KiB
Examples (tfds.as_dataframe):

blimp/distractor_agreement_relational_noun

Config description: This configuration includes the paradigm distractor_agreement_relational_noun.
Download size: 513.33 KiB
Dataset size: 391.45 KiB
Examples (tfds.as_dataframe):

blimp/distractor_agreement_relative_clause

Config description: This configuration includes the paradigm distractor_agreement_relative_clause.
Download size: 551.53 KiB
Dataset size: 416.14 KiB
Examples (tfds.as_dataframe):

blimp/drop_argument

Config description: This configuration includes the paradigm drop_argument.
Download size: 297.07 KiB
Dataset size: 311.70 KiB
Examples (tfds.as_dataframe):

blimp/ellipsis_n_bar_1

Config description: This configuration includes the paradigm ellipsis_n_bar_1.
Download size: 402.32 KiB
Dataset size: 417.06 KiB
Examples (tfds.as_dataframe):

blimp/ellipsis_n_bar_2

Config description: This configuration includes the paradigm ellipsis_n_bar_2.
Download size: 417.53 KiB
Dataset size: 432.47 KiB
Examples (tfds.as_dataframe):

blimp/existential_there_object_raising

Config description: This configuration includes the paradigm existential_there_object_raising.
Download size: 537.77 KiB
Dataset size: 423.00 KiB
Examples (tfds.as_dataframe):

blimp/existential_there_quantifiers_1

Config description: This configuration includes the paradigm existential_there_quantifiers_1.
Download size: 348.95 KiB
Dataset size: 363.58 KiB
Examples (tfds.as_dataframe):

blimp/existential_there_quantifiers_2

Config description: This configuration includes the paradigm existential_there_quantifiers_2.
Download size: 350.80 KiB
Dataset size: 365.43 KiB
Examples (tfds.as_dataframe):

blimp/existential_there_subject_raising

Config description: This configuration includes the paradigm existential_there_subject_raising.
Download size: 385.21 KiB
Dataset size: 399.84 KiB
Examples (tfds.as_dataframe):

blimp/expletive_it_object_raising

Config description: This configuration includes the paradigm expletive_it_object_raising.
Download size: 573.88 KiB
Dataset size: 437.58 KiB
Examples (tfds.as_dataframe):

blimp/inchoative

Config description: This configuration includes the paradigm inchoative.
Download size: 291.71 KiB
Dataset size: 306.34 KiB
Examples (tfds.as_dataframe):

blimp/intransitive

Config description: This configuration includes the paradigm intransitive.
Download size: 298.33 KiB
Dataset size: 312.96 KiB
Examples (tfds.as_dataframe):

blimp/irregular_past_participle_adjectives

Config description: This configuration includes the paradigm irregular_past_participle_adjectives.
Download size: 434.10 KiB
Dataset size: 345.74 KiB
Examples (tfds.as_dataframe):

blimp/irregular_past_participle_verbs

Config description: This configuration includes the paradigm irregular_past_participle_verbs.
Download size: 410.27 KiB
Dataset size: 327.21 KiB
Examples (tfds.as_dataframe):

blimp/irregular_plural_subject_verb_agreement_1

Config description: This configuration includes the paradigm irregular_plural_subject_verb_agreement_1.
Download size: 449.91 KiB
Dataset size: 366.17 KiB
Examples (tfds.as_dataframe):

blimp/irregular_plural_subject_verb_agreement_2

Config description: This configuration includes the paradigm irregular_plural_subject_verb_agreement_2.
Download size: 442.75 KiB
Dataset size: 354.70 KiB
Examples (tfds.as_dataframe):

blimp/left_branch_island_echo_question

Config description: This configuration includes the paradigm left_branch_island_echo_question.
Download size: 471.31 KiB
Dataset size: 348.84 KiB
Examples (tfds.as_dataframe):

blimp/left_branch_island_simple_question

Config description: This configuration includes the paradigm left_branch_island_simple_question.
Download size: 335.40 KiB
Dataset size: 351.01 KiB
Examples (tfds.as_dataframe):

blimp/matrix_question_npi_licensor_present

Config description: This configuration includes the paradigm matrix_question_npi_licensor_present.
Download size: 447.08 KiB
Dataset size: 354.14 KiB
Examples (tfds.as_dataframe):

blimp/npi_present_1

Config description: This configuration includes the paradigm npi_present_1.
Download size: 427.75 KiB
Dataset size: 339.68 KiB
Examples (tfds.as_dataframe):

blimp/npi_present_2

Config description: This configuration includes the paradigm npi_present_2.
Download size: 412.24 KiB
Dataset size: 329.11 KiB
Examples (tfds.as_dataframe):

blimp/only_npi_licensor_present

Config description: This configuration includes the paradigm only_npi_licensor_present.
Download size: 448.41 KiB
Dataset size: 349.50 KiB
Examples (tfds.as_dataframe):

blimp/only_npi_scope

Config description: This configuration includes the paradigm only_npi_scope.
Download size: 570.04 KiB
Dataset size: 408.52 KiB
Examples (tfds.as_dataframe):

blimp/passive_1

Config description: This configuration includes the paradigm passive_1.
Download size: 332.30 KiB
Dataset size: 346.93 KiB
Examples (tfds.as_dataframe):

blimp/passive_2

Config description: This configuration includes the paradigm passive_2.
Download size: 301.12 KiB
Dataset size: 315.75 KiB
Examples (tfds.as_dataframe):

blimp/principle_A_c_command

Config description: This configuration includes the paradigm principle_A_c_command.
Download size: 515.32 KiB
Dataset size: 388.54 KiB
Examples (tfds.as_dataframe):

blimp/principle_A_case_1

Config description: This configuration includes the paradigm principle_A_case_1.
Download size: 466.05 KiB
Dataset size: 370.87 KiB
Examples (tfds.as_dataframe):

blimp/principle_A_case_2

Config description: This configuration includes the paradigm principle_A_case_2.
Download size: 481.42 KiB
Dataset size: 370.88 KiB
Examples (tfds.as_dataframe):

blimp/principle_A_domain_1

Config description: This configuration includes the paradigm principle_A_domain_1.
Download size: 488.15 KiB
Dataset size: 371.62 KiB
Examples (tfds.as_dataframe):

blimp/principle_A_domain_2

Config description: This configuration includes the paradigm principle_A_domain_2.
Download size: 481.63 KiB
Dataset size: 365.92 KiB
Examples (tfds.as_dataframe):

blimp/principle_A_domain_3

Config description: This configuration includes the paradigm principle_A_domain_3.
Download size: 501.84 KiB
Dataset size: 359.74 KiB
Examples (tfds.as_dataframe):

blimp/principle_A_reconstruction

Config description: This configuration includes the paradigm principle_A_reconstruction.
Download size: 337.40 KiB
Dataset size: 353.00 KiB
Examples (tfds.as_dataframe):

blimp/regular_plural_subject_verb_agreement_1

Config description: This configuration includes the paradigm regular_plural_subject_verb_agreement_1.
Download size: 441.26 KiB
Dataset size: 359.56 KiB
Examples (tfds.as_dataframe):

blimp/regular_plural_subject_verb_agreement_2

Config description: This configuration includes the paradigm regular_plural_subject_verb_agreement_2.
Download size: 445.78 KiB
Dataset size: 354.47 KiB
Examples (tfds.as_dataframe):

blimp/sentential_negation_npi_licensor_present

Config description: This configuration includes the paradigm sentential_negation_npi_licensor_present.
Download size: 479.49 KiB
Dataset size: 372.30 KiB
Examples (tfds.as_dataframe):

blimp/sentential_negation_npi_scope

Config description: This configuration includes the paradigm sentential_negation_npi_scope.
Download size: 600.52 KiB
Dataset size: 431.18 KiB
Examples (tfds.as_dataframe):

blimp/sentential_subject_island

Config description: This configuration includes the paradigm sentential_subject_island.
Download size: 357.25 KiB
Dataset size: 372.86 KiB
Examples (tfds.as_dataframe):

blimp/superlative_quantifiers_1

Config description: This configuration includes the paradigm superlative_quantifiers_1.
Download size: 372.25 KiB
Dataset size: 360.02 KiB
Examples (tfds.as_dataframe):

blimp/superlative_quantifiers_2

Config description: This configuration includes the paradigm superlative_quantifiers_2.
Download size: 504.02 KiB
Dataset size: 360.07 KiB
Examples (tfds.as_dataframe):

blimp/tough_vs_raising_1

Config description: This configuration includes the paradigm tough_vs_raising_1.
Download size: 334.99 KiB
Dataset size: 349.62 KiB
Examples (tfds.as_dataframe):

blimp/tough_vs_raising_2

Config description: This configuration includes the paradigm tough_vs_raising_2.
Download size: 355.54 KiB
Dataset size: 370.17 KiB
Examples (tfds.as_dataframe):

blimp/transitive

Config description: This configuration includes the paradigm transitive.
Download size: 449.50 KiB
Dataset size: 334.45 KiB
Examples (tfds.as_dataframe):

blimp/wh_island

Config description: This configuration includes the paradigm wh_island.
Download size: 438.12 KiB
Dataset size: 343.47 KiB
Examples (tfds.as_dataframe):

blimp/wh_questions_object_gap

Config description: This configuration includes the paradigm wh_questions_object_gap.
Download size: 378.35 KiB
Dataset size: 392.99 KiB
Examples (tfds.as_dataframe):

blimp/wh_questions_subject_gap

Config description: This configuration includes the paradigm wh_questions_subject_gap.
Download size: 380.84 KiB
Dataset size: 395.47 KiB
Examples (tfds.as_dataframe):

blimp/wh_questions_subject_gap_long_distance

Config description: This configuration includes the paradigm wh_questions_subject_gap_long_distance.
Download size: 451.82 KiB
Dataset size: 466.53 KiB
Examples (tfds.as_dataframe):

blimp/wh_vs_that_no_gap

Config description: This configuration includes the paradigm wh_vs_that_no_gap.
Download size: 374.28 KiB
Dataset size: 388.91 KiB
Examples (tfds.as_dataframe):

blimp/wh_vs_that_no_gap_long_distance

Config description: This configuration includes the paradigm wh_vs_that_no_gap_long_distance.
Download size: 431.08 KiB
Dataset size: 445.74 KiB
Examples (tfds.as_dataframe):

blimp/wh_vs_that_with_gap

Config description: This configuration includes the paradigm wh_vs_that_with_gap.
Download size: 359.16 KiB
Dataset size: 373.79 KiB
Examples (tfds.as_dataframe):

blimp/wh_vs_that_with_gap_long_distance

Config description: This configuration includes the paradigm wh_vs_that_with_gap_long_distance.
Download size: 416.00 KiB
Dataset size: 430.63 KiB
Examples (tfds.as_dataframe):