clinc_oos

• Description:

Task-oriented dialog systems need to know when a query falls outside their range of supported intents, but current text classification corpora only define label sets that cover every example. We introduce a new dataset that includes queries that are out-of-scope (OOS), i.e., queries that do not fall into any of the system's supported intents. This poses a new challenge because models cannot assume that every query at inference time belongs to a system-supported intent class. Our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production task-oriented agent must handle. It offers a way of more rigorously and realistically benchmarking text classification in task-driven dialog systems.

• Additional Documentation: Explore on Papers With Code

• Homepage: https://github.com/clinc/oos-eval/

• Source code: tfds.text.ClincOOS

• Versions:

  • 0.1.0 (default): No release notes.

• Download size: 256.01 KiB

• Dataset size: 3.40 MiB

• Auto-cached (documentation): Yes

• Splits:

Split	Examples
'test'	4,500
'test_oos'	1,000
'train'	15,000
'train_oos'	100
'validation'	3,000
'validation_oos'	100

• Feature structure:

FeaturesDict({
    'domain': int32,
    'domain_name': Text(shape=(), dtype=string),
    'intent': int32,
    'intent_name': Text(shape=(), dtype=string),
    'text': Text(shape=(), dtype=string),
})

• Feature documentation:

Feature	Class	Shape	Dtype	Description
	FeaturesDict
domain	Tensor		int32
domain_name	Text		string
intent	Tensor		int32
intent_name	Text		string
text	Text		string

• Supervised keys (See as_supervised doc): ('text', 'intent')

• Figure (tfds.show_examples): Not supported.

• Examples (tfds.as_dataframe):

• Citation:

@inproceedings{larson-etal-2019-evaluation,
    title = "An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction",
    author = "Larson, Stefan  and
      Mahendran, Anish  and
      Peper, Joseph J.  and
      Clarke, Christopher  and
      Lee, Andrew  and
      Hill, Parker  and
      Kummerfeld, Jonathan K.  and
      Leach, Kevin  and
      Laurenzano, Michael A.  and
      Tang, Lingjia  and
      Mars, Jason",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D19-1131",
    doi = "10.18653/v1/D19-1131",
    pages = "1311--1316",
}