dolma

Keterangan :

Dolma: Korpus Terbuka Tiga Triliun Token untuk Penelitian Pra-Pelatihan Model Bahasa

Beranda : https://github.com/allenai/dolma
Kode sumber : tfds.datasets.dolma.Builder
Versi :
- 1.0.0 (default): Rilis awal.
Ukuran unduhan : Unknown size
Ukuran kumpulan data : 9.61 TiB
Cache otomatis ( dokumentasi ): Tidak
Perpecahan :

Membelah	Contoh
`'train'`	3.403.336.408

Struktur fitur :

FeaturesDict({
    'added': Text(shape=(), dtype=string),
    'created': Text(shape=(), dtype=string),
    'id': Text(shape=(), dtype=string),
    'source': Text(shape=(), dtype=string),
    'text': Text(shape=(), dtype=string),
})

Dokumentasi fitur :

Fitur	Kelas	Tipe D
	FiturDict
ditambahkan	Teks	rangkaian
dibuat	Teks	rangkaian
pengenal	Teks	rangkaian
sumber	Teks	rangkaian
teks	Teks	rangkaian

Kunci yang diawasi (Lihat dokumen as_supervised ): None
Gambar ( tfds.show_examples ): Tidak didukung.
Contoh ( tfds.as_dataframe ):

Kutipan :

@article{dolma,
  title = { {Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research} },
  author = {
    Luca Soldaini and Rodney Kinney and Akshita Bhagia and Dustin Schwenk and David Atkinson and
    Russell Authur and Ben Bogin and Khyathi Chandu and Jennifer Dumas and Yanai Elazar and
    Valentin Hofmann and Ananya Harsh Jha and Sachin Kumar and Li Lucy and Xinxi Lyu and Ian Magnusson and
    Jacob Morrison and Niklas Muennighoff and Aakanksha Naik and Crystal Nam and Matthew E. Peters and
    Abhilasha Ravichander and Kyle Richardson and Zejiang Shen and Emma Strubell and Nishant Subramani and
    Oyvind Tafjord and Evan Pete Walsh and Hannaneh Hajishirzi and Noah A. Smith and Luke Zettlemoyer and
    Iz Beltagy and Dirk Groeneveld and Jesse Dodge and Kyle Lo
},
  year = {2024},
  journal={arXiv preprint},
}