fluidml.storage

class fluidml.storage.File(path, mode, save_fn, load_fn, open_fn=None, load_kwargs=None, **open_kwargs)

Bases: object

A file like wrapper class to support opening files using the LocalFileStore.

Parameters:

• path (str) – The path to the file to open.

• mode (str) – The open mode, e.g. “r”, “w”, etc.

• save_fn (Callable) – A callable used for file.save() calls.

• load_fn (Callable) – A callable used for file.load() calls.

• open_fn (Optional[Callable]) – An optional callable used for file opening. The default is the inbuilt open() function.

• load_kwargs (Optional[Dict]) – Additional keyword arguments passed to the open function.

close()

property closed

flush()

classmethod from_promise(promise)

Creates a File object from a ´FilePromise`.

Parameters:

promise (FilePromise) –

load(**kwargs)

read(size=-1)

Parameters:

size (int) –

readable()

readline(size=-1)

Parameters:

size (int) –

readlines(hint=-1)

Parameters:

hint (int) –

save(obj, **kwargs)

seek(offset, whence=0)

Parameters:

offset (int) –

seekable()

tell()

truncate(size=None)

Parameters:

size (Optional[int]) –

writable()

write(obj)

Parameters:

obj (AnyStr) –

writelines(lines)

Parameters:

lines (List) –

class fluidml.storage.FilePromise(name, path, save_fn, load_fn, open_fn=None, mode=None, load_kwargs=None, **open_kwargs)

Bases: Promise

Parameters:

• name (str) –

• path (str) –

• save_fn (Callable) –

• load_fn (Callable) –

• open_fn (Optional[Callable]) –

• mode (Optional[str]) –

• load_kwargs (Optional[Dict]) –

load(**kwargs)

Loads the actual object.

class fluidml.storage.InMemoryStore(manager=None)

Bases: ResultsStore

An in-memory results store implemented using a dict.

When multiprocessing is used a manager.dict() is used for inter process communication. If no result store is provided to flow.run() this in-memory store is the default.

Parameters:

manager (Optional[Manager]) –

delete(name, task_name, task_unique_config)

Method to delete any artifact.

Parameters:

• name (str) – The object name.

• task_name (str) – The task name that saved the object.

• task_unique_config (Dict) – Unique config which specifies the run of the object.

delete_run(task_name, task_unique_config)

Method to delete all task results from a given run config

Parameters:

• task_name (str) – Task name which saved the object.

• task_unique_config (Dict) – Unique config which specifies the run of the object.

get_context(task_name, task_unique_config)

Wrapper to get store specific storage context, e.g. the current run directory for Local File Store

Parameters:

• task_name (str) – Task name.

• task_unique_config (Dict) – Unique config which specifies the run.

Returns:

The store context object holding information like the current run dir.

Return type:

StoreContext

load(name, task_name, task_unique_config, **kwargs)

Loads the given object from results store based on its name, task_name and task_config if it exists.

Parameters:

• name (str) – A unique name given to this object.

• task_name (str) – Task name which saved the loaded object.

• task_unique_config (Dict) – Unique config which specifies the run of the loaded object.

• **kwargs – Additional keyword argumentss.

Returns:

The loaded object.

Return type:

Optional[Any]

save(obj, name, task_name, task_unique_config, type_=None, **kwargs)

In-memory save function. Adds individual object to in-memory store.

Parameters:

• obj (Any) –

• name (str) –

• task_name (str) –

• task_unique_config (Dict) –

• type_ (Optional[str]) –

class fluidml.storage.LazySweep(value, config)

Bases: object

A lazy variation of the Sweep class.

Parameters:

• value (Promise) –

• config (MetaDict) –

config: MetaDict

The unique config of the object#s task.

value: Promise

The lazy value of the object.

class fluidml.storage.LocalFileStore(base_dir)

Bases: ResultsStore

A local file store that allows to easily save and load task results to/from a base directory in a file system.

Out of the box the local file store supports three common file types, “json”, “pickle” and “text”. It can be easily extended to arbitrary file types by subclassing the LocalFileStore and registering new Types to the self._type_registry dictionary. A new type needs to register a load and save function using the TypeInfo data class.

Parameters:

base_dir (str) – The base directory that is used to store results from tasks.

base_dir

The base directory that is used to store results from tasks.

type_registry

The dictionary to register new types with a save and load function.

delete(name, task_name, task_unique_config)

Deletes an object from the local file store.

The object is deleted based on the name and the provided task name and unique task config.

Parameters:

• name (str) – The name of the to be deleted object.

• task_name (str) – Task name which saved the object.

• task_unique_config (Dict) – Unique config which specifies the run of the object.

delete_run(task_name, task_unique_config)

Deletes an entire task run directory from the local file store.

The run is deleted based on the task name and the unique task config.

Parameters:

• task_name (str) – The name of the task.

• task_unique_config (Dict) – Unique config which specifies the run of the object.

get_context(task_name, task_unique_config)

Method to get the current task’s storage context.

E.g. the current run directory in case of LocalFileStore. Creates a new run dir if none exists.

Parameters:

• task_name (str) – Task name.

• task_unique_config (Dict) – Unique config which specifies the run.

Return type:

StoreContext

load(name, task_name, task_unique_config, lazy=False, **kwargs)

Loads an object from the local file store.

The object is retrieved based on the name and the provided task name and unique task config.

Parameters:

• name (str) – An unique name given to this object.

• task_name (str) – Task name which saves the object.

• task_unique_config (Dict) – Unique config which specifies the run of the object.

• lazy (bool) – A boolean whether the object should be lazily loaded. If True, a FilePromise object will be returned, that can be loaded into memory on demand with the promise.load() function.

• **kwargs – Additional keyword arguments passed to the registered load() function.

Returns:

The specified object if it is found.

Return type:

Optional[Any]

open(name=None, task_name=None, task_unique_config=None, mode=None, promise=None, type_=None, sub_dir=None, **open_kwargs)

Wrapper to open a file from Local File Store (only available for Local File Store).

It returns a file like object that has additional save() and load() methods that can be used to save/load objects with a registered type to/from the file store. The File like object allows for an incremental write or read process of objects that for example don’t fit into memory.

Parameters:

• name (Optional[str]) – An unique name given to this object.

• task_name (Optional[str]) – Task name which saved the object.

• task_unique_config (Optional[Dict]) – Unique config which specifies the run of the object.

• mode (Optional[str]) – The open mode, e.g. “r”, “w”, etc.

• promise (Optional[FilePromise]) – An optional Promise object used for creating the returned file like object.

• type_ (Optional[str]) – Additional type specification (e.g. json, which is to be passed to results store).

• sub_dir (Optional[str]) – A path of a subdirectory used for opening the file.

• **open_kwargs – Additional keyword arguments passed to the registered open() function.

Returns:

A File object that wraps a file like object and enables incremental result store reading and writing.

Return type:

Optional[File]

property run_info

The current run info of a task, which is used for naming newly created directories.

save(obj, name, type_, task_name, task_unique_config, sub_dir=None, mode=None, open_kwargs=None, load_kwargs=None, **kwargs)

Saves an object to the local file store.

If no task and run directory for the provided unique config exists, a new directory will be created.

Parameters:

• obj (Any) – The object to save.

• name (str) – An unique name given to this object.

• type_ (str) – Additional type specification (e.g. json, which is to be passed to results store).

• task_name (str) – Task name which saves the object.

• task_unique_config (Dict) – Unique config which specifies the run of the object.

• sub_dir (Optional[str]) – A path of a subdirectory used for saving the file.

• mode (Optional[str]) – The mode to save the file, e.g. “w” or “wb”.

• open_kwargs (Optional[Dict[str, Any]]) – Additional keyword arguments passed to the registered open() function.

• load_kwargs (Optional[Dict[str, Any]]) – Additional keyword arguments passed to the registered load() function.

• **kwargs – Additional keyword arguments passed to the registered save() function.

class fluidml.storage.MongoDBStore(db, collection_name=None, host=None)

Bases: ResultsStore

A mongo db result store implementation.

Parameters:

• db (str) – The database name to be used.

• collection_name (Optional[str]) – The name of the collection.

• host (Optional[str]) – The host name of the :program: mongod instance.

delete(name, task_name, task_unique_config)

Query method to delete an object based on its name, task_name and task_config if it exists

Parameters:

• name (str) –

• task_name (str) –

• task_unique_config (Dict) –

delete_run(task_name, task_unique_config)

Query method to delete a run document based on its task_name and task_config if it exists

Parameters:

• task_name (str) –

• task_unique_config (Dict) –

get_context(task_name, task_unique_config)

Wrapper to get store specific storage context, e.g. the current run directory for Local File Store

Parameters:

• task_name (str) – Task name.

• task_unique_config (Dict) – Unique config which specifies the run.

Returns:

The store context object holding information like the current run dir.

Return type:

StoreContext

load(name, task_name, task_unique_config, lazy=False)

Query method to load an object based on its name, task_name and task_config if it exists

Parameters:

• name (str) –

• task_name (str) –

• task_unique_config (Dict) –

• lazy (bool) –

Return type:

Optional[Any]

save(obj, name, type_, task_name, task_unique_config, **kwargs)

Method to save/update any artifact

Parameters:

• obj (Any) –

• name (str) –

• type_ (str) –

• task_name (str) –

• task_unique_config (Dict) –

class fluidml.storage.Promise

Bases: ABC

An interface for future objects, that can be lazy loaded.

abstract load(**kwargs)

Loads the actual object.

class fluidml.storage.ResultsStore

Bases: ABC

The interface of a results store.

abstract delete(name, task_name, task_unique_config)

Method to delete any artifact.

Parameters:

• name (str) – The object name.

• task_name (str) – The task name that saved the object.

• task_unique_config (Dict) – Unique config which specifies the run of the object.

abstract delete_run(task_name, task_unique_config)

Method to delete all task results from a given run config

Parameters:

• task_name (str) – Task name which saved the object.

• task_unique_config (Dict) – Unique config which specifies the run of the object.

abstract get_context(task_name, task_unique_config)

Wrapper to get store specific storage context, e.g. the current run directory for Local File Store

Parameters:

• task_name (str) – Task name.

• task_unique_config (Dict) – Unique config which specifies the run.

Returns:

The store context object holding information like the current run dir.

Return type:

StoreContext

get_results(task_name, task_unique_config)

Collects all saved results based that have been tracked when using Task.save().

Parameters:

• task_name (str) – Task name which saved the object.

• task_unique_config (Dict) – Unique config which specifies the run of the object.

Returns:

A dictionary of all saved result objects.

Return type:

Optional[Dict]

is_finished(task_name, task_unique_config)

Checks if a task is finished.

Parameters:

• task_name (str) – Task name which saved the object.

• task_unique_config (Dict) – Unique config which specifies the run of the object.

Returns:

A boolean indicating whether the task is finished or not.

Return type:

bool

abstract load(name, task_name, task_unique_config, **kwargs)

Loads the given object from results store based on its name, task_name and task_config if it exists.

Parameters:

• name (str) – A unique name given to this object.

• task_name (str) – Task name which saved the loaded object.

• task_unique_config (Dict) – Unique config which specifies the run of the loaded object.

• **kwargs – Additional keyword argumentss.

Returns:

The loaded object.

Return type:

Optional[Any]

property lock

open(name=None, task_name=None, task_unique_config=None, mode=None, promise=None, type_=None, sub_dir=None, **open_kwargs)

Method to open a file from Local File Store (only available for file stores).

Parameters:

• name (Optional[str]) – An unique name given to this object.

• task_name (Optional[str]) – Task name which saved the object.

• task_unique_config (Optional[Dict]) – Unique config which specifies the run of the object.

• mode (Optional[str]) – The open mode, e.g. “r”, “w”, etc.

• promise (Optional[Promise]) – An optional Promise object used for creating the returned file like object.

• type_ (Optional[str]) – Additional type specification (e.g. json, which is to be passed to results store).

• sub_dir (Optional[str]) – A path of a subdirectory used for opening the file.

• **open_kwargs – Additional keyword arguments passed to the registered open() function.

Returns:

A File object that behaves like a file like object.

abstract save(obj, name, type_, task_name, task_unique_config, **kwargs)

Method to save/update any artifact.

Parameters:

• obj (Any) – The object to save/update

• name (str) – The object name.

• type_ (str) – The type of the object. Only required for file stores.

• task_name (str) – The task name that saves/updates the object.

• task_unique_config (Dict) – The unique config of that task.

• **kwargs – Additional keyword arguments.

class fluidml.storage.StoreContext(run_dir=None, sweep_counter=None)

Bases: object

The store context of the current task.

Parameters:

• run_dir (Optional[str]) –

• sweep_counter (Optional[str]) –

run_dir: Optional[str] = None

The run directory of the task. Relevant for File Stores.

sweep_counter: Optional[str] = None

The sweep counter of the task. Relevant for File Stores. A dynamically created counter to distinguish different task instances with the same run name in the results store.

class fluidml.storage.Sweep(value, config)

Bases: object

A sweep class holding the value and config of a specific task result.

List of Sweeps are the standard inputs for reduce tasks that gather results from grid search expanded tasks.

Parameters:

• value (Any) –

• config (MetaDict) –

config: MetaDict

The unique config of the object#s task.

value: Any

The value of the object.

class fluidml.storage.TypeInfo(save_fn, load_fn, extension=None, is_binary=None, open_fn=None, needs_path=False)

Bases: object

Initializes saving and loading information for a specific type.

Parameters:

• save_fn (Callable) –

• load_fn (Callable) –

• extension (Optional[str]) –

• is_binary (Optional[bool]) –

• open_fn (Optional[Callable]) –

• needs_path (bool) –

extension: Optional[str] = None

File extension the object is saved with.

is_binary: Optional[bool] = None

Read, write and append in binary mode.

load_fn: Callable

Load function used to load the object from store.

needs_path: bool = False

false.

Type:

Whether save and load fn should operate on path and not on file like object. Default

open_fn: Optional[Callable] = None

Function used to open a file object (default is builtin open()).

save_fn: Callable

Save function used to save the object to store.