RSVP for your your local TensorFlow Everywhere event today!

tff.simulation.CSVMetricsManager

Utility class for saving/loading experiment metrics via a CSV file.

Inherits From: MetricsManager

csv_filepath A string specifying the file to write and read metrics from.

ValueError If csv_filepath is an empty string.
ValueError If the specified metrics csv file already exists but does not contain a round_num column.

metrics_filename

Methods

clear_all_rounds

View source

Existing metrics for all rounds are cleared out.

This method will atomically update the stored CSV file.

clear_rounds_after

View source

Metrics for rounds greater than last_valid_round_num are cleared out.

By using this method, this class can be used upon restart of an experiment at last_valid_round_num to ensure that no duplicate rows of data exist in the CSV file. This method will atomically update the stored CSV file.

Args
last_valid_round_num All metrics for rounds later than this are expunged.

Raises
RuntimeError If metrics do not exist (none loaded during construction ' nor recorded via update_metrics() and last_valid_round_num is not zero.
ValueError If last_valid_round_num is negative.

get_metrics

View source

Retrieve the stored experiment metrics data for all rounds.

Returns
A sequence representing all possible keys for the metrics, and a list containing experiment metrics data for all rounds. Each entry in the list is a dictionary corresponding to a given round. The data has been flattened, with the column names equal to the path in the original nested metric structure. There is a fieldname round_num to indicate the round number.

update_metrics

View source

Updates the stored metrics data with metrics for a specific round.

The specified round_num must be later than the latest round number for which metrics exist in the stored metrics data. This method will atomically update the stored CSV file. Also, if stored metrics already exist and metrics_to_append contains a new, previously unseen metric name, a new column in the dataframe will be added for that metric, and all previous rows will fill in with NaN values for the metric.

The metrics written are the leaf node tensors of the metrics_to_append structure. Purely scalar tensors will be written as scalars in the CSV, while tensors with non-zero rank will be written as a list of lists. For example, the tensor tf.ones([2, 2]) will be written to the CSV as '[[1.0, 1.0], [1.0, 1.0]'.

Args
round_num Communication round at which metrics_to_append was collected.
metrics_to_append A nested structure of metrics collected during round_num. The nesting will be flattened for storage in the CSV (with the new keys equal to the paths in the nested structure).

Returns
A collections.OrderedDict of the metrics used to update the manager. Compared with the input metrics_to_append, this data is flattened, with the key names equal to the path in the nested structure, and round_num has been added as an additional key (overwriting the value if already present in the input metrics_to_append). The OrderedDict is sorted by the flattened keys.

Raises
ValueError If the provided round number is negative.
ValueError If the provided round number is less than or equal to the latest round number in the stored metrics data.