ML Community Day is November 9! Join us for updates from TensorFlow, JAX, and more Learn more


TensorFlow 1 version

Converts a text to a sequence of words (or tokens).


text: Input text (string).
filters: list (or concatenation) of characters to filter out, such as
    punctuation. Default: ``!"#$%&()*+,-./:;<=>?@[\]^_`{|}~\t\n``,
    includes basic punctuation, tabs, and newlines.
lower: boolean. Whether to convert the input to lowercase.
split: str. Separator for word splitting.


A list of words (or tokens).