Attend the Women in ML Symposium on December 7 Register now

para_crawl

References:

enbg

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enbg')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 1039885
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "bg"
        ],
        "id": null,
        "_type": "Translation"
    }
}

encs

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/encs')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 2981949
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "cs"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enda

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enda')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 2414895
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "da"
        ],
        "id": null,
        "_type": "Translation"
    }
}

ende

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/ende')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 16264448
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "de"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enel

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enel')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 1985233
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "el"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enes

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enes')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 21987267
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "es"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enet

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enet')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 853422
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "et"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enfi

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enfi')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 2156069
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "fi"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enfr

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enfr')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 31374161
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "fr"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enga

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enga')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 357399
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "ga"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enhr

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enhr')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 1002053
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "hr"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enhu

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enhu')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 1901342
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "hu"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enit

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enit')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 12162239
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "it"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enlt

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enlt')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 844643
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "lt"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enlv

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enlv')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 553060
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "lv"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enmt

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enmt')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 195502
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "mt"
        ],
        "id": null,
        "_type": "Translation"
    }
}

ennl

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/ennl')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 5659268
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "nl"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enpl

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enpl')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 3503276
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "pl"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enpt

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enpt')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 8141940
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "pt"
        ],
        "id": null,
        "_type": "Translation"
    }
}

enro

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/enro')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 1952043
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "ro"
        ],
        "id": null,
        "_type": "Translation"
    }
}

ensk

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/ensk')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 1591831
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "sk"
        ],
        "id": null,
        "_type": "Translation"
    }
}

ensl

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/ensl')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 660161
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "sl"
        ],
        "id": null,
        "_type": "Translation"
    }
}

ensv

Use the following command to load this dataset in TFDS:

ds = tfds.load('huggingface:para_crawl/ensv')
  • Description:
Web-Scale Parallel Corpora for Official European Languages.
  • License: No known license
  • Version: 1.0.0
  • Splits:
Split Examples
'train' 3476729
  • Features:
{
    "translation": {
        "languages": [
            "en",
            "sv"
        ],
        "id": null,
        "_type": "Translation"
    }
}