5.3. yapydata.datatree.datatree¶

The module ‘datatree.datatree’ implements the actual access to structured in-memory data. The data is implemented by a subset of Python data types as a canonical in-memory structure comaptible to the standard package json. The specific modules for the supported syntaxes extend the required methods.

5.3.1. Module¶

The YapyData.datatree module provides the core class for the handling of simple data classes for the most common data representation languages. The provided core class DataTree provided the Python data types, which comprise the most standard data types.

The derived classes support for the most common data description languages and add therefore the specific constraints and extensions. The package provides these features for low-level libraries of the software stack, therefore depends whenever possible on standard libraries only. It supports low-level read-only access to files and in-memory data. The read data could be modified in-memory only for example in order to superpose higher priority data read from the call options of the command line .

The internal representation of the DDLs is exclusively compatible to the standard json package in accordance to RFC-7159. The supported DDLs of the read files are:

INI
JSON
XML
YAML

A similar package for higher application layer levels is available by multiconf, which provides sophisticated features such as cross-conversion and mixed-mode applications by modularization, and enhanced processing plugins for various DDLs.

The validation and preparation including cross-conversion is supported by multiconf, while the specifics of the DL is supported by the language module such as jsondata and xmldata.

Sources: yapydata/datatree/datatree.py

5.3.2. Functions¶

5.3.2.1. grow_branch¶

datatree.grow_branch(**kargs)¶

Creates a new branch including the assigned value to the last node. The node types are defined by the types of the subpath entries.

Supports a single linear branch only, no sub-branching.

The created path is validated for permitted types. The derived types such as JSON have to support their own branch method. Thus provided as a static method.

Parameters

subpath – Variable list/tuple of path keys and indexes.
kargs –

value:
Value to be assigned to the final node.

default := None

Returns

A created branch.

Raises

YapyDataDataTreeOidError –
pass-through –

5.3.2.2. readout_data¶

yapydata.datatree.datatree.readout_data(xval, **kargs)[source]¶

For API call-compliance with other syntaxes. Returns here the input tree only.

Parameters: xval – The input tree from the DataTree.
Returns: The returns here the input xval.
Raises: pass-through –

5.3.3. DataTree¶

class yapydata.datatree.datatree.DataTree(data=None)[source]¶

Provides JSON based read-only configuration of capabilities.

The access to structured data trees offers various methods to access paths of nested node attributes. This comprises the creation as well as the readout.

The following equivalent creation methods are supported, where ‘treenode’ could be either the root node, or any subordinated branch:

treenode['subnode0']['subnode1']['subnode7'] = value  # dynamic items

value = treenode.create(                              # dynamic items by 'create()'
            'subnode0', 'subnode1', 'subnode7',
        )

value = treenode.subnode0.subnode1.subnode7           # static attribute addressing style

The following equivalent readout methods are supported, where ‘treenode’ could be either the root node, or any subordinated branch:

value = treenode['subnode0']['subnode1']['subnode7']  # dynamic items
value = treenode('subnode0', 'subnode1', 'subnode7')  # dynamic items by '__call__'
value = treenode.subnode0.subnode1.subnode7           # static attribute addressing style

5.3.3.1. init¶

DataTree.__init__(data=None)[source]¶

Parameters

data –

Configuration data in accordance to the selected data language syntax. The default is the Python syntax including the appropriate data types. This may impose additional constraints by derived classes e.g. in case of persistent data such as JSON and XML - see other classes within this module.

The default Python DL implementation supports in-memory access only, while persistence will be available for example by pickling.

The initial data defines the permitted type of the first item within the subpath of the spanned data tree. The default value acts here as a placeholder for an empty structure, which could be defined by following extension operations arbitrarily.

The basic constraint introduced here is that intermediate nodes require support of subscription. This is due to the addressing concepts implemented by DataTree. Thus even though a set could technically be an intermediate node, it could not be indexed, thus could not be addressed by the standard member functions. Resulting ‘set’ and ‘frozenset’ are supported by DataTree as end-nodes only, same as atomic data types.

Anyhow, the <top-node> is by default permitted to be an end-node. Thus the context defines the applicability dynamically.

The consistency of the data tree including the valid intermediate nodes is verified by access, so basically within the responsibility of the caller.

Returns

None / initialized object

Raises

YapyDataDataTreeError –
pass-through –

5.3.3.2. call¶

DataTree.__call__(*subpath, **kargs)[source]¶

Readout the value of a node, or an attribute. The name binding of the path is provided as a tuple of path items.

Parameters

subpath –

The list of keys constituting a branch of a data tree. The subpath is treated as a branch of one of the nodes of a provided searchpath - which is by default the top node. The supported values are:

subpath := <list-of-node-ids>
<list-of-node-ids> := <node-id> [',' <list-of-node-ids>]
node-id := (
      str            # string:  dict
    | int            # integer: lists, tuple, dict
)

The value of the node within data:

nodeid := (
      <single-nodeid>
    | <list-of-nodeids>
    | <tuple-of-nodeids>
)
single-nodeid := <nodeid>
list-of-nodeids := '[' <nodeidlists> ']'
tuple-of-nodeids := '(' <nodeidlists> ')'
nodeidlists := <nodeid> [',' <nodeidlists>]
nodeid := (
      ItemKey
    | ListIndex
)
ItemKey := "valid dict-key"
ListIndex := "valid list-index"

The derived syntax classes may impose specific constraints. Thus it is recommended to use integers and strings only for maximum compatibility, and the ease of using mixed syntaxes:

ItemKey :=    str  # string:  dict
ListIndex :=  int  # integer: lists, tuple, dict

kargs –

searchpath:

Optional search path for the match of the provided address subpath. The provided subpath is applied to each node of the searchpath in accordance to the direction option. This provides the search and enumeration of side branches:

searchpath := <path-item-list>

path-item-list := <path-item> [, <path-item-list>]
path-item := (
      str  # item name
    | int  # item index
)

default := <top-node>

The search path entries has to be actually present by default. These could be either created by loading a complete tree structure, or by using the Capabilities.create() member. See also parameter ‘strict’.

direction:

The search direction of the subpath within the searchpath. In case of multiple superpositioned attributes the first traversed match.

The provided values are:

direction := (
      up   | 0  | False # search from right-to-left
    | down | 1  | True  # search from left-to-right
)

default:= up

match:

Sets the match criteria for the search operation. Interferes with direction:

match := (
      M_FIRST | 'first'   # use first matching node
    | M_LAST  | 'last'    # use last matching node
    | M_ALL   | 'all'     # use all - iterate all matches
)

default := M_FIRST

partial:

Enables the return of partial sub paths in case the requested path is not completely present.

partial := (
      True   # when not completely present, the longest
             # existing part is returned, the completeness
             # is provided by the result attribute <partial>
    | False  # when not completely present an exception
             # is raised
)

strict:

Controls the required consistency. This comprises:

the presence of the search path entries
the presence of the requested subpath within the set of search paths

pysyn:

Activates full Python syntax. This in particular enables all container types of intermediate nodes for arbitrary paths. Includes tuple, set, frozenset, etc.

pysyn := (
      True   # allows all python types as container nodes
    | False  # allows list and dict only as container nodes
)

default := False

Returns

In case of a match returns the tuple:

return := (<attr-value-path>, <attr-value>, <partial>)

attr-value-path := (
      "the list of keys of the top-down path"
    | "empty list when no item exists"        # see <complete>
)
attr-value := "value of the targeted node/attribute"
partial := (
      False   # the complete requested path
    | True    # the actually present part of the path
)

Else raises YapyDataDataTreeOidError.

Raises

YapyDataDataTreeOidError –
pass-through –

5.3.3.3. create¶

DataTree.create(*subpath, **kargs)[source]¶

Creates a subpath to a given node, default is from top. Reuses existing nodes, starts the creation at the first point of branch-out from the exiting tree.

In general no padding of pre-required entries is done. This e.g. requires in case of a list the start with the index 0, while in case of the dict arbitrary keys could be assigned.

Parameters

subpath –

The list of keys constituting a branch of a data tree. The subpath is treated as a branch of one of the nodes of a provided searchpath - which is by default the top node. The supported values are:

subpath := <list-of-node-ids>
<list-of-node-ids> := <node-id> [',' <list-of-node-ids>]
node-id := (
      <present-intermediate-node>
    | <new-node>
)
present-intermediate-node := (
      str             # string:  dict
    | int             # integer: lists, tuple, dict
    | True | False    # logic:   dict
    | None            # null:    dict
)
new-node := (
      str             # string:  dict
    | int             # integer: list
    | True | False    # logic:   dict
    | None            # null:    dict
)

Some Python data types are immutable, which could be subscripted read-only, e.g. strings. While others such as sets are iterable, but not subscriptable at all. Refer to the manual for a detailed list.

kargs –

hook:

Optional node as parent of the insertion point for the new sub path. The node must exist and be part of the targeted data structure. No additional checks are done:

hook := <memory-node>
memory-node := "node address"

default := <top-node>

The hook node could either be a border node of the existing tree, or any arbitrary node with a partial of complete part of the requested subpath. Existing nodes are reused.

strict:

If True requires all present nodes of the subpath to of the appropriate type, missing are created. When False present nodes of inappropriate type are simply replaced.

strict := (
      True   # nodes must be present
    | False  # missing are created
)
default := False

Nodes of type None are always treated as non-present placeholder, thus replaced in any case.

value:

Value of the created final node:

value := <value-of-node>

value-of-node := <valid-json-node-type>
valid-json-node-type := (
                      int | float
                    | str                  # unicode
                    | dict | list
                    | None | True | False  # equivalent: null|true|false
                    )

default := None

Returns

In case of success the in-memory nodes of the sub path:

return := (<attr-value-path>)

attr-value-path-tuple := (
      <in-memory nodes>
    | <non-subscriptable-node>
)
in-memory nodes := (
    "the list of in-memory nodes with support of subscription"
)
non-subscriptable-node := "any valid type"

else raises YapyDataDataTreeOidError.

The last node contains in case of an atomic type the value of the node, while the intermediate nodes represent the indexed containers.

Raises

YapyDataDataTreeOidError –
pass-through –

5.3.3.4. get¶

DataTree.get(*path)[source]¶

Gets the value of the path within the member ‘data’:

self.data[key]
self.data[key0][key1][key2]...

Parameters

key –

The value of the node within data:

key := (
      <single-key>
    | <list-of-keys>
    | <tuple-of-keys>
)
single-key := <key>
list-of-keys := '[' <keylists> ']'
tuple-of-keys := '(' <keylist> ')'
keylist := <key> [',' <keylist>]
key := (
      ItemKey
    | ListIndex
)
ItemKey := "valid dict-key"
ListIndex := "valid list-index"

Returns

The value of the addressed node/value.

Raises

pass-through –

5.3.3.5. isvalid_top¶

static DataTree.isvalid_top(value, **kargs)[source]¶

Validate compliance of top-node. To be provided by derived classes for specific syntaxes.

Parameters

value – Top node.
kargs – Specific syntax related dynamic parameters to be defined by derived classes.

Returns

None

Results:

YapyDataTreeError:: Raises exception when not valid.

5.3.3.6. join¶

DataTree.join(data, keyidx=None, hook=None)[source]¶

Superposes a JSON structure onto an existing. This is a fixed mode and strategy special case of the generic method superpose(). Implemented by recursion. The reduced parameter set provides better performance on large trees while the graph parameters still could be efficiently set by default values.

The superpositioning is supported by multiple strategies defined by the parameter mode. The provided algorithm of the strategy is join, where the input data is processed on the exisiting tree by modification and creation as required.

branches are resolved from top to the leafs
missing sub-branches are created
missing leafs are created
existing leafs are replaced

This implements a last-wins strategy, thus in particular supports incremental load of configurations by raising priority.

Parameters

data – Top node of the data tree to be superposed.

keyidx –

Key or index to be used at the insertion node. If not given the insertion node is:

into dict: update by the new node
into list: append the new node

default := None

hook –

The insertion node for the new data:

when  not given: use top.

default := None

Returns

Returns the merged data tree, raises an exception in case of failure.

Raises

YapyDataDataTreeError –
pass-through –

5.3.3.7. import_data¶

DataTree.import_data(fpname, key=None, node=None, **kargs)[source]¶

The core class *DataTree *does not support serialization.

For serialization use either a derived syntax class, or serialize it e..g. by pickling and use the in-memory data. For example by pickling:

def import_data(self, fpname, key=None, node=None, **kargs):
    if not os.path.isfile(fpname):
        if not os.path.isfile(fpname + '.pkl'):
            raise YapyDataTreeError(
                "Missing file: " + str(fpname))
        else:
            fpname = fpname + '.pkl'

    if key and node == None:
        raise YapyDataTreeError(
            "Given key(%s) requires a valid node." % (str(key)))

    datafile = os.path.abspath(fpname)
    with open(datafile, 'rb') as data_file:
        pval = pickle.load(data_file)

    if key:
        node[key] = pval
    else:
        self.data = pval

    return pval

See manuals for security issues of pickling.

5.3.4. Exceptions¶

exception yapydata.datatree.datatree.YapyDataDataTreeError[source]¶: Common access error.

s.. autoexception:: yapydata.datatree.datatree.YapyDataDataTreeOidError

5.3. yapydata.datatree.datatree¶

5.3.1. Module¶

5.3.2. Functions¶

5.3.2.1. grow_branch¶

5.3.2.2. readout_data¶

5.3.3. DataTree¶

5.3.3.1. __init__¶

5.3.3.2. __call__¶

5.3.3.3. create¶

5.3.3.4. get¶

5.3.3.5. isvalid_top¶

5.3.3.6. join¶

5.3.3.7. import_data¶

5.3.4. Exceptions¶

5.3.3.1. init¶

5.3.3.2. call¶