Dynamic URL Store

This module contains the DynamicURLStore store that communicates with a remote HTTP server which provides the actual data storage. This is a store which implements the basic operations via HTTP GET, POST, PUT and DELETE commands as described in the class documentation.

The implementation relies on the third-party requests library to handle the HTTP operations.

class encore.storage.dynamic_url_store.DynamicURLStore(base_url, query_url, url_format='{base}/{key}/{part}', url_format_no_part='{base}/{key}', parts={'data': 'data', 'metadata': 'metadata', 'permissions': 'auth'})

Store implementation which gets and sets from a web server

This store expects a server which exposes URLs for each key. By default these URLs are of the form:

<base>/<key>/<part>

Where <base> is a common prefix, <key> is the key of interest, and <part> is one of “data”, “metadata” or “auth”. If the store does not follow this format, you can provide a differnt url_format argument and a different mapping of <part> to aspects of the key.

The server is expected to respond to queries against these URLS in the following ways:

GET <base>/<key>/data

return the bytes in the body of the response

PUT <base>/<key>/data

accept the data bytes from the body of the request

GET <base>/<key>/metadata

return metadata as JSON

PUT <base>/<key>/metadata

set the metadata based on JSON contained in the body of the request

POST <base>/<key>/metadata

update the metadata based on JSON contained in the body of the request (as dict.update())

GET <base>/<key>/auth

return permissions information as JSON

PUT <base>/<key>/auth

set the permissions based on JSON contained in the body of the request

POST <base>/<key>/metadata

update the permissions based on JSON contained in the body of the request

In addition, a DELETE request to a URL of the form <base>/<key> should remove the key from the remote store. This pattern is configurable via the url_format_no_part argument to the constructor.

In addition, the server should have a query URL which accepts GET reuqests containing a JSON data structure of metadata key, value pairs to filter with, and should return a list of macthing keys, one per line.

connect(credentials=None)

Connect to a DynamicURLStore

Parameters

credentials ((user_tag, requests.Session)) – The credentials are a tuple containing ther user’s permission tag and a requests Session initialized with appropriate authentication.

delete(key)

Delete a key from the repsository.

This may be left unimplemented by subclasses that represent a read-only key-value store.

Parameters

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Events

StoreDeleteEvent - On successful completion of a transaction, a StoreDeleteEvent should be emitted with the key.

disconnect()

Disconnect from the key-value store

This method disposes or disconnects to any long-lived resources that the store requires.

get(key)

Retrieve a stream of data and metdata from a given key in the key-value store.

Parameters

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Returns

value (instance of Value) - An instance of a Value subclass which holds references to the data, metadata and other information about the key.

Raises

KeyError - If the key is not found in the store, a KeyError is raised.

get_data(key)

Retrieve a stream from a given key in the key-value store.

Parameters

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Returns

data (file-like) - A readable file-like object the that provides stream of data from the key-value store.

Raises

KeyError - This will raise a key error if the key is not present in the store.

get_metadata(key, select=None)

Retrieve the metadata for a given key in the key-value store.

Parameters
  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

  • select (iterable of strings or None) – Which metadata keys to populate in the result. If unspecified, then return the entire metadata dictionary.

Returns

metadata (dict) - A dictionary of metadata associated with the key. The dictionary has keys as specified by the select argument. If a key specified in select is not present in the metadata, then it will not be present in the returned value.

Raises

KeyError - This will raise a key error if the key is not present in the store.

get_permissions(key)

Return the set of permissions the user has

Parameters

key (str) – The key for the resource which you want to know the permissions.

Returns

permissions (dict of str: set of str) - A dictionary whose keys are the permissions and values are sets of tags which have that permission.

Raises
  • KeyError - This error will be raised if the key does not exist or the user is not authorized to see it.

  • AuthorizationError - This error will be raised if user is authorized to see the key, but is not an owner.

info()

Get information about the key-value store

Returns

metadata (dict) - A dictionary of metadata giving information about the key-value store.

is_connected()

Whether or not the store is currently connected

Returns

connected (bool) - Whether or not the store is currently connected.

query(select=None, **kwargs)

Query for keys and metadata matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of metadata for the key-value store.

Parameters
  • select (iterable of strings or None) – An optional list of metadata keys to return. If this is not None, then the metadata dictionaries will only have values for the specified keys populated.

  • kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.

Returns

result (iterable) - An iterable of (key, metadata) tuples where metadata matches all the specified values for the specified metadata keywords. If a key specified in select is not present in the metadata of a particular key, then it will not be present in the returned value.

query_keys(**kwargs)

Query for keys matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of keys for the key-value store.

This is equivalent to self.query(**kwargs).keys(), but potentially more efficiently implemented.

Parameters

kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.

Returns

result (iterable) - An iterable of key-value store keys whose metadata matches all the specified values for the specified metadata keywords.

set(key, value, buffer_size=1048576)

Store a stream of data into a given key in the key-value store.

This may be left unimplemented by subclasses that represent a read-only key-value store.

Parameters
  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

  • value (instance of Value) – An instance of a Value subclass.

  • buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

Events
  • StoreProgressStartEvent - For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

  • StoreProgressStepEvent - For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

  • StoreProgressEndEvent - For buffering implementations, this event should be emitted after finishing writing to the underlying store.

  • StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

set_data(key, data, buffer_size=1048576)

Replace the data for a given key in the key-value store.

Parameters
  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

  • data (file-like) – A readable file-like object the that provides stream of data from the key-value store.

  • buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

Events
  • StoreProgressStartEvent - For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

  • StoreProgressStepEvent - For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

  • StoreProgressEndEvent - For buffering implementations, this event should be emitted after finishing writing to the underlying store.

  • StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

set_metadata(key, metadata)

Set new metadata for a given key in the key-value store.

This replaces the existing metadata set for the key with a new set of metadata.

Parameters
  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

  • metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.

Events

StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

set_permissions(key, permissions)

Set the permissions on a key the user owns

Parameters
  • key (str) – The key for the resource which you want to know the permissions.

  • permissions (dict of str: set of str) – A dictionary whose keys are the permissions and values are sets of tags which have that permission. There must be an ‘owned’ permission with at least one tag.

Raises
  • KeyError - This error will be raised if the key does not exist or the user is not authorized to see it.

  • AuthorizationError - This error will be raised if user is authorized to see the key, but is not an owner.

transaction(notes)

Provide a transaction context manager

This class does not support transactions, so it returns a dummy object.

update_metadata(key, metadata)

Update the metadata for a given key in the key-value store.

This performs a dictionary update on the existing metadata with the provided metadata keys and values

Parameters
  • key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

  • metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.

Events

StoreSetEvent - On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

update_permissions(key, permissions)

Add permissions on a key the user owns

The tags provided in the permissions dictionary will be added to the existing set of tags for each permission.

Parameters
  • key (str) – The key for the resource which you want to know the permissions.

  • permissions (dict of str: set of str) – A dictionary whose keys are the permissions and values are sets of tags which have that permission.

Raises
  • KeyError - This error will be raised if the key does not exist or the user is not authorized to see it.

  • AuthorizationError - This error will be raised if user is authorized to see the key, but is not an owner.

user_tag()

A tag that represents the user