ML Community Day is November 9! Join us for updates from TensorFlow, JAX, and more Learn more

gref

  • Description:

The Google RefExp dataset is a collection of text descriptions of objects in images which builds on the publicly available MS-COCO dataset. Whereas the image captions in MS-COCO apply to the entire image, this dataset focuses on text descriptions that allow one to uniquely identify a single object or region within an image. See more details in this paper: Generation and Comprehension of Unambiguous Object Descriptions.

  • Homepage: https://github.com/mjhucla/Google_Refexp_toolbox

  • Source code: tfds.vision_language.gref.Gref

  • Versions:

    • 1.0.0 (default): Initial release.
  • Download size: Unknown size

  • Dataset size: 4.60 GiB

  • Manual download instructions: This dataset requires you to download the source data manually into download_config.manual_dir (defaults to ~/tensorflow_datasets/downloads/manual/):
    Follow instructions at https://github.com/mjhucla/Google_Refexp_toolbox to download and pre-process the data into aligned format with COCO. The directory contains 2 files and one folder:

  • google_refexp_train_201511_coco_aligned_catg.json

  • google_refexp_val_201511_coco_aligned_catg.json

  • coco_train2014/

The coco_train2014 folder contains all of COCO 2014 training images.

Split Examples
'train' 24,698
'validation' 4,650
  • Features:
FeaturesDict({
    'image': Image(shape=(None, None, 3), dtype=tf.uint8),
    'image/id': tf.int64,
    'objects': Sequence({
        'area': tf.int64,
        'bbox': BBoxFeature(shape=(4,), dtype=tf.float32),
        'id': tf.int64,
        'label': tf.int64,
        'label_name': ClassLabel(shape=(), dtype=tf.int64, num_classes=80),
        'refexp': Sequence({
            'raw': Text(shape=(), dtype=tf.string),
            'referent': Text(shape=(), dtype=tf.string),
            'refexp_id': tf.int64,
            'tokens': Sequence(Text(shape=(), dtype=tf.string)),
        }),
    }),
})

Visualization

  • Citation:
@inproceedings{mao2016generation,
  title={Generation and Comprehension of Unambiguous Object Descriptions},
  author={Mao, Junhua and Huang, Jonathan and Toshev, Alexander and Camburu, Oana and Yuille, Alan and Murphy, Kevin},
  booktitle={CVPR},
  year={2016}
}