• Description:

A re-labeled version of CIFAR-10 with real human annotation errors. For every pair (image, label) in the original CIFAR-10 train set, it provides several additional labels given by real human annotators.

  • Homepage:

  • Source code: tfds.image_classification.cifar10_n.Cifar10N

  • Versions:

    • 1.0.0: Initial release.
    • 1.0.1: Fixed typo in worse_label key.
    • 1.0.2: Fixed correspondence between annotations and images.
    • 1.0.3: Fixed files in MANUAL_DIR.
    • 1.0.4 (default): Fixed loading of side information.
  • Download size: 162.17 MiB

  • Dataset size: 147.91 MiB

  • Manual download instructions: This dataset requires you to download the source data manually into download_config.manual_dir (defaults to ~/tensorflow_datasets/downloads/manual/):
    Download 'side_info_cifar10N.csv', 'CIFAR-10_human_ordered.npy' and 'image_order_c10.npy' from

Then convert 'CIFAR-10_human_ordered.npy' into a CSV file 'CIFAR-10_human_annotations.csv'. This can be done with the following code:

import numpy as np
import pandas as pd
import tensorflow as tf

human_labels_np_path = '<local_path>/CIFAR-10_human_ordered.npy'
human_labels_csv_path = '<local_path>/CIFAR-10_human_annotations.csv'

with, "rb") as f:
  human_annotations = np.load(f, allow_pickle=True)

df = pd.DataFrame(human_annotations[()])

with, "w") as f:
  df.to_csv(f, index=False)
Split Examples
'test' 10,000
'train' 50,000
  • Feature structure:
    'aggre_label': ClassLabel(shape=(), dtype=tf.int64, num_classes=10),
    'id': Text(shape=(), dtype=tf.string),
    'image': Image(shape=(32, 32, 3), dtype=tf.uint8),
    'label': ClassLabel(shape=(), dtype=tf.int64, num_classes=10),
    'random_label1': ClassLabel(shape=(), dtype=tf.int64, num_classes=10),
    'random_label2': ClassLabel(shape=(), dtype=tf.int64, num_classes=10),
    'random_label3': ClassLabel(shape=(), dtype=tf.int64, num_classes=10),
    'worker1_id': tf.int64,
    'worker1_time': tf.float32,
    'worker2_id': tf.int64,
    'worker2_time': tf.float32,
    'worker3_id': tf.int64,
    'worker3_time': tf.float32,
    'worse_label': ClassLabel(shape=(), dtype=tf.int64, num_classes=10),
  • Feature documentation:
Feature Class Shape Dtype Description
aggre_label ClassLabel tf.int64
id Text tf.string
image Image (32, 32, 3) tf.uint8
label ClassLabel tf.int64
random_label1 ClassLabel tf.int64
random_label2 ClassLabel tf.int64
random_label3 ClassLabel tf.int64
worker1_id Tensor tf.int64
worker1_time Tensor tf.float32
worker2_id Tensor tf.int64
worker2_time Tensor tf.float32
worker3_id Tensor tf.int64
worker3_time Tensor tf.float32
worse_label ClassLabel tf.int64


  • Citation:
  title={Learning with Noisy Labels Revisited: A Study Using Real-World Human
  author={Jiaheng Wei and Zhaowei Zhu and Hao Cheng and Tongliang Liu and Gang
  Niu and Yang Liu},
  booktitle={International Conference on Learning Representations},