scalarstop.datablob_metadata
¶
Tools for storing metadata about DataBlob
s.
Module Contents¶
Classes¶
Represents the metadata from a |
- class DataBlobMetadata(*, name: str, group_name: str, hyperparams: scalarstop.hyperparams.HyperparamsType, save_load_version: int, num_shards: int)¶
Represents the metadata from a
DataBlob
that is saved to or loaded from the filesystem.When we save metadata to the filesystem, we save the same information in two files–one in JSON format and the other in Python Pickle format. The JSON file is human-readable and can be parsed by non-Python programs. The Pickle file is kept to ensure that the
DataBlob
‘s hyperparams can be accurately deserialized–even if the hyperparams are not JSON-serializable.Creates a
DataBlobMetadata
object in memory.- Parameters
name – The
DataBlob
name.group_name – The
DataBlob
group name.hyperparams – The
Hyperparams
object for theDataBlob
. This has to be an instance ofHyperparamsType
and not a Python dictionary.save_load_version – The protocol version used to save or load this
DataBlob
to/from the filesystem.num_shards – The number of shards to divide the
DataBlob
into when saving to the filesystem.
- classmethod load(cls, path: str) DataBlobMetadata ¶
Loads metadata from a
DataBlob
‘s directory on the filesystem.
- classmethod from_datablob(cls, datablob: scalarstop.datablob.DataBlob, *, save_load_version: int, num_shards: int)¶
Creates a
DataBlobMetadata
object in memory from aDataBlob
instance.- Parameters
datablob – The
DataBlob
for which thisDataBlobMetadata
object is being created for.save_load_version – The protocol version used to save or load this
DataBlob
to/from the filesystem.num_shards – The number of shards to divide the
DataBlob
into when saving to the filesystem.
- to_dict(self, *, hyperparams_as_dict: bool = False) Dict[str, Any] ¶
Return the metadata as a Python dictionary.
- Parameters
hyperparams_as_dict – Set to
True
to convert a
:param
HyperparamsType
: :param object to a Python dictionary.: