tft.experimental.document_frequency

Maps the terms in x to their document frequency in the same order.

tft.experimental.document_frequency(
    x: tf.SparseTensor, vocab_size: int, name: Optional[str] = None
) -> tf.SparseTensor

The document frequency of a term is the number of documents that contain the term in the entire dataset. Each unique vocab term has a unique document frequency.

Example usage:

def preprocessing_fn(inputs):
  integerized = tft.compute_and_apply_vocabulary(inputs['x'])
  vocab_size = tft.get_num_buckets_for_transformed_feature(integerized)
  return {
     'df': tft.experimental.document_frequency(integerized, vocab_size),
     'integerized': integerized,
  }
raw_data = [dict(x=["I", "like", "pie", "pie", "pie"]),
            dict(x=["yum", "yum", "pie"])]
feature_spec = dict(x=tf.io.VarLenFeature(tf.string))
raw_data_metadata = tft.DatasetMetadata.from_feature_spec(feature_spec)
with tft_beam.Context(temp_dir=tempfile.mkdtemp()):
  transformed_dataset, transform_fn = (
      (raw_data, raw_data_metadata)
      | tft_beam.AnalyzeAndTransformDataset(preprocessing_fn))
transformed_data, transformed_metadata = transformed_dataset
transformed_data
[{'df': array([1, 1, 2, 2, 2]), 'integerized': array([3, 2, 0, 0, 0])},
 {'df': array([1, 1, 2]), 'integerized': array([1, 1, 0])}]
 
  example strings: [["I", "like", "pie", "pie", "pie"], ["yum", "yum", "pie]]
  in: SparseTensor(indices=[[0, 0], [0, 1], [0, 2], [0, 3], [0, 4],
                            [1, 0], [1, 1], [1, 2]],
                   values=[1, 2, 0, 0, 0, 3, 3, 0])
  out: SparseTensor(indices=[[0, 0], [0, 1], [0, 2], [0, 3], [0, 4],
                            [1, 0], [1, 1], [1, 2]],
                   values=[1, 1, 2, 2, 2, 1, 1, 2])

x A 2D SparseTensor representing int64 values (most likely that are the result of calling compute_and_apply_vocabulary on a tokenized string).
vocab_size An int - the count of vocab used to turn the string into int64s including any OOV buckets.
name (Optional) A name for this operation.

SparseTensors with indices [index_in_batch, index_in_local_sequence] and values document_frequency. Same shape as the input x.

ValueError if x does not have 2 dimensions.