File System Store¶
This file defines a filesystem store. This stores data in a specified directory in a filesystem. Data files are stored in files with name key+’.data’ and metadata files with name key+’.metadata’.
Create the magic file for the shared store. Useful to initialize the store for the first time.
- Parameters
path – The directory that will be used for the file store.
magic_fname – The name of the magic file in that directory,
-
class
encore.storage.filesystem_store.
FileSystemStore
(path, magic_fname='.FSStore')¶ A store that uses a Shared file system to store the data/metadata.
-
__init__
(path, magic_fname='.FSStore')¶ Initializes the store given a path to a store.
- Parameters
path (str:) – A path to the root of the file system store.
magic_fname – The name of the magic file in that directory,
-
connect
(credentials=None)¶ Connect to the key-value store.
- Parameters
credentials – These are not used by default.
-
delete
(key)¶ Delete a key from the repsository.
This may be left unimplemented by subclasses that represent a read-only key-value store.
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
- Events
StoreDeleteEvent - On successful completion of a transaction, a StoreDeleteEvent should be emitted with the key.
-
disconnect
()¶ Disconnect from the key-value store
This store does not authenticate, and has no external resources, so this does nothing
-
exists
(key)¶ Test whether or not a key exists in the key-value store
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
- Returns
exists (bool) - Whether or not the key exists in the key-value store.
-
from_bytes
(key, data, buffer_size=1048576)¶ Efficiently store a bytes object as the data associated with a key.
This method can be optionally overriden by subclasses to proved a more efficient way of copy the data from a bytes object to the underlying data store. The default implementation uses the set() method together with a cStringIO.
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
data (bytes) – The data as a bytes object.
buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).
-
from_file
(key, path, buffer_size=1048576)¶ Efficiently read data from a file into a key in the key-value store.
This method can be optionally overriden by subclasses to proved a more efficient way of copy the data from a path in the filesystem to the underlying data store. The default implementation uses the set() method together with chunked reads from the disk which are fed into the data stream.
This makes no attempt to set metadata.
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
path (string) – A file system path to read the data from.
buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).
-
get
(key)¶ Retrieve a stream of data and metdata from a given key in the key-value store.
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
- Returns
data (file-like) - A readable file-like object that provides stream of data from the key-value store
metadata (dictionary) - A dictionary of metadata for the key.
- Raises
KeyError - If the key is not found in the store, a KeyError is raised.
-
get_data
(key)¶ Retrieve a stream from a given key in the key-value store.
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
- Returns
data (file-like) - A readable file-like object the that provides stream of data from the key-value store.
- Raises
KeyError - This will raise a key error if the key is not present in the store.
-
get_metadata
(key, select=None)¶ Retrieve the metadata for a given key in the key-value store.
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
select (iterable of strings or None) – Which metadata keys to populate in the result. If unspecified, then return the entire metadata dictionary.
- Returns
metadata (dict) - A dictionary of metadata associated with the key. The dictionary has keys as specified by the select argument. If a key specified in select is not present in the metadata, then it will not be present in the returned value.
- Raises
KeyError - This will raise a key error if the key is not present in the store.
-
glob
(pattern)¶ Return keys which match glob-style patterns
- Parameters
pattern (string) – Glob-style pattern to match keys with.
- Returns
result (iterable) - A iterable of keys which match the glob pattern.
-
is_connected
()¶ Whether or not the store is currently connected
- Returns
connected (bool) - Whether or not the store is currently connected.
-
multiget
(keys)¶ Retrieve the data and metadata for a collection of keys.
- Parameters
keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.
- Returns
result (iterator of (file-like, dict) tuples) - An iterator of (data, metadata) pairs.
- Raises
KeyError - This will raise a key error if the key is not present in the store.
-
multiget_data
(keys)¶ Retrieve the data for a collection of keys.
- Parameters
keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.
- Returns
result (iterator of file-like) - An iterator of file-like data objects corresponding to the keys.
- Raises
KeyError - This will raise a key error if the key is not present in the store.
-
multiget_metadata
(keys, select=None)¶ Retrieve the metadata for a collection of keys in the key-value store.
- Parameters
keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.
select (iterable of strings or None) – Which metadata keys to populate in the results. If unspecified, then return the entire metadata dictionary.
- Returns
metadatas (iterator of dicts) - An iterator of dictionaries of metadata associated with the key. The dictionaries have keys as specified by the select argument. If a key specified in select is not present in the metadata, then it will not be present in the returned value.
- Raises
KeyError - This will raise a key error if the key is not present in the store.
-
multiset
(keys, values, buffer_size=1048576)¶ Set the data and metadata for a collection of keys.
Where supported by an implementation, this should perform the whole collection of sets as a single transaction.
Like zip() if keys and values have different lengths, then any excess values in the longer list should be silently ignored.
- Parameters
keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.
values (iterable of (file-like, dict) tuples) – An iterator that provides the (data, metadata) pairs for the corresponding keys.
buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).
- Events
StoreProgressStartEvent - For buffering implementations, this event should be emitted prior to writing any data to the underlying store.
StoreProgressStepEvent - For buffering implementations, this event should be emitted periodically as data is written to the underlying store.
StoreProgressEndEvent - For buffering implementations, this event should be emitted after finishing writing to the underlying store.
StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.
-
multiset_data
(keys, datas, buffer_size=1048576)¶ Set the data for a collection of keys.
Where supported by an implementation, this should perform the whole collection of sets as a single transaction.
Like zip() if keys and datas have different lengths, then any excess values in the longer list should be silently ignored.
- Parameters
keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.
datas (iterable of file-like objects) – An iterator that provides the data file-like objects for the corresponding keys.
buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).
- Events
StoreProgressStartEvent - For buffering implementations, this event should be emitted prior to writing any data to the underlying store.
StoreProgressStepEvent - For buffering implementations, this event should be emitted periodically as data is written to the underlying store.
StoreProgressEndEvent - For buffering implementations, this event should be emitted after finishing writing to the underlying store.
StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.
-
multiset_metadata
(keys, metadatas)¶ Set the metadata for a collection of keys.
Where supported by an implementation, this should perform the whole collection of sets as a single transaction.
Like zip() if keys and metadatas have different lengths, then any excess values in the longer list should be silently ignored.
- Parameters
keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.
metadatas (iterable of dicts) – An iterator that provides the metadata dictionaries for the corresponding keys.
- Events
StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.
-
multiupdate_metadata
(keys, metadatas)¶ Update the metadata for a collection of keys.
Where supported by an implementation, this should perform the whole collection of sets as a single transaction.
Like zip() if keys and metadatas have different lengths, then any excess values in the longer list should be silently ignored.
- Parameters
keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.
metadatas (iterable of dicts) – An iterator that provides the metadata dictionaries for the corresponding keys.
- Events
StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.
-
query
(select=None, **kwargs)¶ Query for keys and metadata matching metadata provided as keyword arguments
This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of metadata for the key-value store.
- Parameters
select (iterable of strings or None) – An optional list of metadata keys to return. If this is not None, then the metadata dictionaries will only have values for the specified keys populated.
kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.
- Returns
result (iterable) - An iterable of (key, metadata) tuples where metadata matches all the specified values for the specified metadata keywords. If a key specified in select is not present in the metadata of a particular key, then it will not be present in the returned value.
-
query_keys
(**kwargs)¶ Query for keys matching metadata provided as keyword arguments
This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of keys for the key-value store.
This is equivalent to
self.query(**kwargs).keys()
, but potentially more efficiently implemented.- Parameters
kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.
- Returns
result (iterable) - An iterable of key-value store keys whose metadata matches all the specified values for the specified metadata keywords.
-
set
(key, value, buffer_size=1048576)¶ Store a stream of data into a given key in the key-value store.
This may be left unimplemented by subclasses that represent a read-only key-value store.
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
value (tuple of file-like, dict) – A pair of objects, the first being a readable file-like object that provides stream of data from the key-value store. The second is a dictionary of metadata for the key.
buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).
- Events
StoreProgressStartEvent - For buffering implementations, this event should be emitted prior to writing any data to the underlying store.
StoreProgressStepEvent - For buffering implementations, this event should be emitted periodically as data is written to the underlying store.
StoreProgressEndEvent - For buffering implementations, this event should be emitted after finishing writing to the underlying store
StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata
-
set_data
(key, data, buffer_size=1048576)¶ Replace the data for a given key in the key-value store.
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
data (file-like) – A readable file-like object the that provides stream of data from the key-value store.
buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).
- Events
StoreProgressStartEvent - For buffering implementations, this event should be emitted prior to writing any data to the underlying store.
StoreProgressStepEvent - For buffering implementations, this event should be emitted periodically as data is written to the underlying store.
StoreProgressEndEvent - For buffering implementations, this event should be emitted after finishing writing to the underlying store.
StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata
-
set_metadata
(key, metadata)¶ Set new metadata for a given key in the key-value store.
This replaces the existing metadata set for the key with a new set of metadata.
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.
- Events
StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata
-
to_bytes
(key, buffer_size=1048576)¶ Efficiently store the data associated with a key into a bytes object.
This method can be optionally overriden by subclasses to proved a more efficient way of copy the data from the underlying data store to a bytes object. The default implementation uses the get() method together with chunked reads from the returned data stream and join.
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).
- Returns
bytes - The contents of the file-like object as bytes.
- Events
StoreProgressStartEvent - For buffering implementations, this event should be emitted prior to extracting the data.
StoreProgressStepEvent - For buffering implementations, this event should be emitted periodically as data is extracted.
StoreProgressEndEvent - For buffering implementations, this event should be emitted after extracting the data.
-
to_file
(key, path, buffer_size=1048576)¶ Efficiently store the data associated with a key into a file.
This method can be optionally overriden by subclasses to proved a more efficient way of copy the data from the underlying data store to a path in the filesystem. The default implementation uses the get() method together with chunked reads from the returned data stream to the disk.
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
path (string) – A file system path to store the data to.
buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).
- Events
StoreProgressStartEvent - For buffering implementations, this event should be emitted prior to writing any data to disk.
StoreProgressStepEvent - For buffering implementations, this event should be emitted periodically as data is written to disk.
StoreProgressEndEvent - For buffering implementations, this event should be emitted after finishing writing to disk.
-
transaction
(notes)¶ Provide a transaction context manager
This class does not support transactions, so it returns a dummy object.
-
update_metadata
(key, metadata)¶ Update the metadata for a given key in the key-value store.
This performs a dictionary update on the existing metadata with the provided metadata keys and values
- Parameters
key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.
metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.
- Events
StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata
-