scalarstop.datablob_metadata

Tools for storing metadata about DataBlob s.

Module Contents

Classes

DataBlobMetadata

Represents the metadata from a DataBlob
class DataBlobMetadata(*, name: str, group_name: str, hyperparams: scalarstop.hyperparams.HyperparamsType, save_load_version: int, num_shards: int)

Represents the metadata from a DataBlob that is saved to or loaded from the filesystem.

When we save metadata to the filesystem, we save the same information in two files–one in JSON format and the other in Python Pickle format. The JSON file is human-readable and can be parsed by non-Python programs. The Pickle file is kept to ensure that the DataBlob ‘s hyperparams can be accurately deserialized–even if the hyperparams are not JSON-serializable.

Creates a DataBlobMetadata object in memory.

Parameters

  • name – The DataBlob name.

  • group_name – The DataBlob group name.

  • hyperparams – The Hyperparams object for the DataBlob. This has to be an instance of HyperparamsType and not a Python dictionary.

  • save_load_version – The protocol version used to save or load this DataBlob to/from the filesystem.

  • num_shards – The number of shards to divide the DataBlob into when saving to the filesystem.

classmethod load(cls, path: str) → DataBlobMetadata

Loads metadata from a DataBlob ‘s directory on the filesystem.

classmethod from_datablob(cls, datablob: scalarstop.datablob.DataBlob, *, save_load_version: int, num_shards: int)

Creates a DataBlobMetadata object in memory from a DataBlob instance.

Parameters

  • datablob – The DataBlob for which this DataBlobMetadata object is being created for.

  • save_load_version – The protocol version used to save or load this DataBlob to/from the filesystem.

  • num_shards – The number of shards to divide the DataBlob into when saving to the filesystem.

to_dict(self, *, hyperparams_as_dict: bool = False) → Dict[str, Any]

Return the metadata as a Python dictionary.

Parameters

hyperparams_as_dict – Set to True to convert a

:param HyperparamsType: :param object to a Python dictionary.:

save(self, path: str)

Save the metadata to a given DataBlob directory on the filesystem.

Parameters

path – The DataBlob directory on the filesystem to save the metadata to.