config — YAML-based Configuration Dictionaries

This module provides structured, YAML based, deep dictionary configuration objects. The objects have a built-in deep-update function and use deep-update behavior by default. They act otherwise like dictionaries, and handle thier internal operation using a storage dictionary. The objects also provide a YAML configuration file reading and writing interface.

pyshell.config.reformat(d, nt)

Recursive extraction method for changing the type of nested dictionary objects.

Parameters:

• d (mapping) – The dictionary to re-type.
• nt (mapping-type) – The new mapping type to use.

pyshell.config.force_yaml_unicode()

This method forces the PyYAML library to construct unicode objects when reading YAML instead of producing regular strings.

It is designed to imporove compatibility in Python2.x using unicode objects.

pyshell.config.deepmerge(d, u, s, invert=False, inplace=True)

Merge deep collection-like structures.

When this function encounters a sequence, the entire sequence from u is considered a single value which replaces any value from d. To allow for merging sequences in u and d, see function advanceddeepmerge().

Parameters:

• d (dict-like) – Deep Structure
• u (dict-like) – Updated Structure
• s (dict-like-type) – Default structure to use when a new deep structure is required.
• invert (bool) – Whether to do an inverse merge.

Inverse Merge causes u to only update missing values of d, but does so in a deep fashion.

pyshell.config.advanceddeepmerge(d, u, s, sequence=True, invert=False, inplace=True)

Merge deep collection-like structures.

This function will merge sequence structures when they are found. When used with sequence=False, it behaves like deepmerge().

Parameters:

• d (dict-like) – Deep Structure
• u (dict-like) – Updated Structure
• s (dict-like-type) – Default structure to use when a new deep structure is required.
• sequence (bool) – Control sequence merging
• invert (bool) – Whether to do an inverse merge.

Inverse Merge causes u to only update missing values of d, but does so in a deep fashion.

pyshell.config.flatten(d, stump=u'', sequence=False, separator=u'.', dt=<type 'dict'>)

Flatten a given nested dictionary.

Parameters:

• d – Dictionary to flatten.
• stump – The base stump for flattened keys. Setting this applies a universal starting value to each key.
• sequence – Whether to expand sequences.
• separator – The string separator to use in flat keys.
• dt – The final output type for the dictionary.

Each nested key will become a root level key in the final dictionary. The root level keys will be the set of nested keys, joined by the separator keyword argument.

pyshell.config.expand(d, sequence=False, separator=u'.', dt=<type 'dict'>)

Expand a flattened dictionary into a nested one.

Parameters:

• d – Dictionary to expand.
• sequence – Whether to expand sequences.
• separator – The string separator to use in flat keys.
• dt – The final output type for all levels of the nested dictionary.

Each key with the separator will become a nested dictionary key in the final dictionary.

Basic Configurations: Configuration

class pyshell.config.Configuration(*args, **kwargs)

Adds extra methods to dictionary for configuration

dn

Deep nesting attribute reader

dt

Deep storage type.

name = u'Configuration'

The name/type of this configuration.

hash

Return the HexDigest hash

filename

The filename which has been used to save/load this configuration most recently

update(other, deep=True)

Update the dictionary using merge().

Parameters:

• other (dict-like) – The other dictionary to be merged.
• deep (bool) – Whether to use deep merge (merge()) or shallow update.

merge(other)

Merge another configuration into this one (the master).

Parameters:other (dict-like) – The other dictionary to be merged.

See deepmerge().

>>> a = Configuration(**{'a':'b','c':'e'})
>>> a.merge({'c':'d'})
>>> a
{'a': 'b', 'c': 'd'}

imerge(other)

Inverse merge(), where other will be considered original, and this object will be canonical.

Parameters:other (dict-like) – The other dictionary to be merged.

See deepmerge().

>>> a = Configuration(**{'a':'b','c':'e'})
>>> a.imerge({'c':'d'})
>>> a
{'a': 'b', 'c': 'e'}

save(filename, silent=True)

Save this configuration as a YAML file. YAML files generally have the .yaml or .yml extension. If the filename ends in .dat, the configuration will be saved as a raw dictionary literal.

Parameters:

• filename (string) – The filename on which to save the configuration.
• silent (bool) – Unused.

load(filename, silent=True, fname=None)

Loads a configuration from a yaml file, and merges it into the master configuration.

Parameters:

• filename (string) – The filename to load from.
• silent (bool) – Silence IOErrors which might arise due to a non-existant configuration file. If this is the case, the failure to find a configuration file will be logged, will not raise an error.

Raises :

IOError if the file can’t be found.

Returns:

boolean, whether the file was loaded.

store

Dictionary representing this configuration. This property should be used if you wish to have a ‘true’ dictionary object. It is used internally to write this configuration to a YAML file.

renest(deep_store_type=None)

Re-nest this object. This method applies the dt deep-storage attribute to each nesting level in the configuration object.

Parameters:deep_store_type – mapping nesting type, will set dn.

This method does not return anything.

parse_literals(*literals, **kwargs)

Turn a list of literals into configuration items.

Parameters:

• literals – Any literals which are separated by the separator.
• sep – The separator to use, defaults to "=".

Keywords are parsed where foo=bar becomes self["foo"] = "bar". If bar can be parsed as a python literal (float, int, dict, list etc..), the literal value will be used in place of the string. For which literals will be parsed, see ast.literal_eval() from the Abstract-Syntax Tree features in python. There is great power in using this method with dotted configurations, as foo.bat=bar will get parsed to self["foo.bat"] = "bar". This is useful for parsing configuration command line options.

load_resource(module, filename, silent=True)

Load from a resource filename

configure(module='pyshell.config', defaultcfg=False, cfg=False, supercfg=None)

The configuration loads (starting with a blank configuration):

1. The list of supercfg ‘s. This list should contain tuples of (module,name) pairs.
2. The module configuration file named for defaultcfg
3. The cfg file from the user’s home folder ~/config.yml
4. The cfg file from the working directory.

If the fourth file is not found, and the user specified a new name for the configuration file (i.e. cfg != defaultcfg), then the user is warned that no configuration file could be found. This way the user is only warned about a missing configuration file if they requested a file specifically (and so intended to use a customized file).

Parameters:

• module (string) – The name of the module for searching for the default config.
• cfg (string) – The name of the requested configuration file.
• defaultcfg (string) – The name of the default configuration file which might exist in the module’s file.
• supercfg (list) – A list of configuration files to preload. The list should contian pairs of (module,name) as tuples.

classmethod create(module='pyshell.config', defaultcfg=False, cfg=False, supercfg=None)

Create a configuration from a series of YAML files.

See configure() for a detailed description of the resolution order of configuration files for this method.

classmethod fromfile(filename)

Create a configuration from a single YAML file.

classmethod fromresource(module, filename)

Create a configuration from a resource filename pair.

Parameters:

• module – The module containing the file.
• filename – The filename within that module.

classmethod make(base)

Make a configuration from the input object base.

Acceptable Inputs:

• An instance of this class.
• Any insatance of collections.Mapping
• A string filename for fromfile()
• A tuple of argumments to fromresource()
• A sequence of arguments to this method, which can be recursively added to this configuration.

Dotted Configurations: Configuration

class pyshell.config.DottedConfiguration(*args, **kwargs)

A configuration which can use dotted accessor methods.

Configuration variables can be accessed and set with dot-qualified names. E.g.:

>>> Config = DottedConfiguration( { "Data": { "Value": { "ResultA" : 10 }, }, })
>>> Config["Data"]["Value"]["ResultA"]
10
>>> Config["Data.Value.ResultA"]
10

By default, this will not work for doubly nested values:

>>> Config["Data"]["Value.ResultA"]
KeyError

However, this behavior can be changed by specifying a new default nesting structure:

>>> Config.dn = DottedConfiguration
>>> Config.merge(Config)
>>> Config["Data"]["Value.ResultA"]
10

separator = u'.'

The deep nesting separator character(s).

flatten(sequence=False)

Returns this dictionary, flattened so that all dotted names are at the root level.

clear() → None. Remove all items from D.

configure(module='pyshell.config', defaultcfg=False, cfg=False, supercfg=None)

The configuration loads (starting with a blank configuration):

1. The list of supercfg ‘s. This list should contain tuples of (module,name) pairs.
2. The module configuration file named for defaultcfg
3. The cfg file from the user’s home folder ~/config.yml
4. The cfg file from the working directory.

If the fourth file is not found, and the user specified a new name for the configuration file (i.e. cfg != defaultcfg), then the user is warned that no configuration file could be found. This way the user is only warned about a missing configuration file if they requested a file specifically (and so intended to use a customized file).

Parameters:

• module (string) – The name of the module for searching for the default config.
• cfg (string) – The name of the requested configuration file.
• defaultcfg (string) – The name of the default configuration file which might exist in the module’s file.
• supercfg (list) – A list of configuration files to preload. The list should contian pairs of (module,name) as tuples.

classmethod create(module='pyshell.config', defaultcfg=False, cfg=False, supercfg=None)

Create a configuration from a series of YAML files.

See configure() for a detailed description of the resolution order of configuration files for this method.

dn

Deep nesting attribute reader

dt

Deep storage type.

filename

The filename which has been used to save/load this configuration most recently

classmethod fromfile(filename)

Create a configuration from a single YAML file.

classmethod fromresource(module, filename)

Create a configuration from a resource filename pair.

Parameters:

• module – The module containing the file.
• filename – The filename within that module.

get(k[, d]) → D[k] if k in D, else d. d defaults to None.

hash

Return the HexDigest hash

imerge(other)

Inverse merge(), where other will be considered original, and this object will be canonical.

Parameters:other (dict-like) – The other dictionary to be merged.

See deepmerge().

>>> a = Configuration(**{'a':'b','c':'e'})
>>> a.imerge({'c':'d'})
>>> a
{'a': 'b', 'c': 'e'}

items() → list of D's (key, value) pairs, as 2-tuples

iteritems() → an iterator over the (key, value) items of D

iterkeys() → an iterator over the keys of D

itervalues() → an iterator over the values of D

keys() → list of D's keys

load(filename, silent=True, fname=None)

Loads a configuration from a yaml file, and merges it into the master configuration.

Parameters:

• filename (string) – The filename to load from.
• silent (bool) – Silence IOErrors which might arise due to a non-existant configuration file. If this is the case, the failure to find a configuration file will be logged, will not raise an error.

Raises :

IOError if the file can’t be found.

Returns:

boolean, whether the file was loaded.

load_resource(module, filename, silent=True)

Load from a resource filename

classmethod make(base)

Make a configuration from the input object base.

Acceptable Inputs:

• An instance of this class.
• Any insatance of collections.Mapping
• A string filename for fromfile()
• A tuple of argumments to fromresource()
• A sequence of arguments to this method, which can be recursively added to this configuration.

merge(other)

Merge another configuration into this one (the master).

Parameters:other (dict-like) – The other dictionary to be merged.

See deepmerge().

>>> a = Configuration(**{'a':'b','c':'e'})
>>> a.merge({'c':'d'})
>>> a
{'a': 'b', 'c': 'd'}

parse_literals(*literals, **kwargs)

Turn a list of literals into configuration items.

Parameters:

• literals – Any literals which are separated by the separator.
• sep – The separator to use, defaults to "=".

Keywords are parsed where foo=bar becomes self["foo"] = "bar". If bar can be parsed as a python literal (float, int, dict, list etc..), the literal value will be used in place of the string. For which literals will be parsed, see ast.literal_eval() from the Abstract-Syntax Tree features in python. There is great power in using this method with dotted configurations, as foo.bat=bar will get parsed to self["foo.bat"] = "bar". This is useful for parsing configuration command line options.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

renest(deep_store_type=None)

Re-nest this object. This method applies the dt deep-storage attribute to each nesting level in the configuration object.

Parameters:deep_store_type – mapping nesting type, will set dn.

This method does not return anything.

save(filename, silent=True)

Save this configuration as a YAML file. YAML files generally have the .yaml or .yml extension. If the filename ends in .dat, the configuration will be saved as a raw dictionary literal.

Parameters:

• filename (string) – The filename on which to save the configuration.
• silent (bool) – Unused.

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D

store

Dictionary representing this configuration. This property should be used if you wish to have a ‘true’ dictionary object. It is used internally to write this configuration to a YAML file.

update(other, deep=True)

Update the dictionary using merge().

Parameters:

• other (dict-like) – The other dictionary to be merged.
• deep (bool) – Whether to use deep merge (merge()) or shallow update.

values() → list of D's values

Structured Configurations: StructuredConfiguration

class pyshell.config.StructuredConfiguration(*args, **kwargs)

A structured configuration with some basic defaults.

This class does two things differently for configurations:

1. Configurations are stored in a “Configurations” variable set. They can then be loaded by configuration key instead of filename, using the setFile() method to set the filename, and then calling load() or meth:save with no arguments.

2. Configuration variables can be accessed and set with dot-qualified names. E.g.:

   >>> Config = StructuredConfigruation( { "Data": { "Value": { "ResultA" : 10 }, }, })
   >>> Config["Data"]["Value"]["ResultA"]
   10
   >>> Config["Data.Value.ResultA"]
   10
   

By default, this will not work for doubly nested values:

>>> Config["Data"]["Value.ResultA"]
KeyError

However, this behavior can be changed by specifying a new default nesting structure:

>>> Config.dn = DottedConfiguration

clear() → None. Remove all items from D.

configure(module='pyshell.config', defaultcfg=False, cfg=False, supercfg=None)

The configuration loads (starting with a blank configuration):

1. The list of supercfg ‘s. This list should contain tuples of (module,name) pairs.
2. The module configuration file named for defaultcfg
3. The cfg file from the user’s home folder ~/config.yml
4. The cfg file from the working directory.

If the fourth file is not found, and the user specified a new name for the configuration file (i.e. cfg != defaultcfg), then the user is warned that no configuration file could be found. This way the user is only warned about a missing configuration file if they requested a file specifically (and so intended to use a customized file).

Parameters:

• module (string) – The name of the module for searching for the default config.
• cfg (string) – The name of the requested configuration file.
• defaultcfg (string) – The name of the default configuration file which might exist in the module’s file.
• supercfg (list) – A list of configuration files to preload. The list should contian pairs of (module,name) as tuples.

classmethod create(module='pyshell.config', defaultcfg=False, cfg=False, supercfg=None)

Create a configuration from a series of YAML files.

See configure() for a detailed description of the resolution order of configuration files for this method.

dn

Deep nesting attribute reader

dt

Deep storage type.

filename

The filename which has been used to save/load this configuration most recently

flatten(sequence=False)

Returns this dictionary, flattened so that all dotted names are at the root level.

classmethod fromfile(filename)

Create a configuration from a single YAML file.

classmethod fromresource(module, filename)

Create a configuration from a resource filename pair.

Parameters:

• module – The module containing the file.
• filename – The filename within that module.

get(k[, d]) → D[k] if k in D, else d. d defaults to None.

hash

Return the HexDigest hash

imerge(other)

Inverse merge(), where other will be considered original, and this object will be canonical.

Parameters:other (dict-like) – The other dictionary to be merged.

See deepmerge().

>>> a = Configuration(**{'a':'b','c':'e'})
>>> a.imerge({'c':'d'})
>>> a
{'a': 'b', 'c': 'e'}

items() → list of D's (key, value) pairs, as 2-tuples

iteritems() → an iterator over the (key, value) items of D

iterkeys() → an iterator over the keys of D

itervalues() → an iterator over the values of D

keys() → list of D's keys

load_resource(module, filename, silent=True)

Load from a resource filename

classmethod make(base)

Make a configuration from the input object base.

Acceptable Inputs:

• An instance of this class.
• Any insatance of collections.Mapping
• A string filename for fromfile()
• A tuple of argumments to fromresource()
• A sequence of arguments to this method, which can be recursively added to this configuration.

merge(other)

Merge another configuration into this one (the master).

Parameters:other (dict-like) – The other dictionary to be merged.

See deepmerge().

>>> a = Configuration(**{'a':'b','c':'e'})
>>> a.merge({'c':'d'})
>>> a
{'a': 'b', 'c': 'd'}

parse_literals(*literals, **kwargs)

Turn a list of literals into configuration items.

Parameters:

• literals – Any literals which are separated by the separator.
• sep – The separator to use, defaults to "=".

Keywords are parsed where foo=bar becomes self["foo"] = "bar". If bar can be parsed as a python literal (float, int, dict, list etc..), the literal value will be used in place of the string. For which literals will be parsed, see ast.literal_eval() from the Abstract-Syntax Tree features in python. There is great power in using this method with dotted configurations, as foo.bat=bar will get parsed to self["foo.bat"] = "bar". This is useful for parsing configuration command line options.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

renest(deep_store_type=None)

Re-nest this object. This method applies the dt deep-storage attribute to each nesting level in the configuration object.

Parameters:deep_store_type – mapping nesting type, will set dn.

This method does not return anything.

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D

store

Dictionary representing this configuration. This property should be used if you wish to have a ‘true’ dictionary object. It is used internally to write this configuration to a YAML file.

update(other, deep=True)

Update the dictionary using merge().

Parameters:

• other (dict-like) – The other dictionary to be merged.
• deep (bool) – Whether to use deep merge (merge()) or shallow update.

values() → list of D's values

metadata

The metadata dictionary

files

The set of loaded filenames

set_file(filename=None)

Set the default/current configuration file for this configuration.

The configuration file set by this method will be used next time load() or save() is called with no filename.

Parameters:filename (string) – The filename to load from.

save(filename=None, silent=True)

Save the configuration to a YAML file. If filename is not provided, the configuration will use the file set by setFile().

Parameters:filename (string) – Destination filename.

Uses Configuration.save().

load(filename=None, silent=True, fname=None)

Load the configuration to a YAML file. If filename is not provided, the configuration will use the file set by setFile().

Parameters:

• filename (string) – Target filename.
• silent (bool) – Whether to raise an error if the target file cannot be found.

Uses Configuration.load().

Developer Notes

There are a few design decisions in pyshell.config that are worth explaining.

The configuration items work with an internal storage object. That storage object is a dictionary of dictionaries, in its purest form. Configuration classes are responsible for re-casting any returned value as the correct external configuration object when accessed. This should be done as lightly as possible (i.e. only on the first layer, not renested).

There is some tricky handling of dictionary keys which contain the default separator, ".". The following rules apply:

• Dictionaries inserted with dotted keys remain intact.
• Dotted keys inserted are interperted as heirarchical keys.
• Heirarchical keys will be retrieved only when the key can’t be interpreted as a dotted key.
• To retrieve a heirarchical key which is hidden by a dotted key, retrieve the key levels as separate items, without the "."

When writing to YAML files, there are two callback functions which can be used to encapsulate metadata, or multiple YAML documents in a single document. By default, the included YAML reader reads documents separated by ---. Only the last document is the configuration interpreted by the Configuration object. The other documents are passed to _load_yaml_callback(), which accepts many arguments as individual YAML documents. To save multiple YAML documents as output, the extra YAML documents should be returned as a list by _save_yaml_callback(). If you don’t want to save any extra documents, _save_yaml_callback() may return an empty list (as it does by default). These callback functions could be used to silently change the underlying configuration object before serialization to YAML. This behavior has not been tested.