escore.core_ops.links package¶
Submodules¶
escore.core_ops.links.apply module¶
Project: Eskapade - A python-based package for data analysis.
Class: DsApply
Created: 2018-06-30
- Description:
Simple link to execute functions, to which datastore has been passed.
Helps in the development of links.
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.apply.
DsApply
(**kwargs)¶ Bases:
escore.core.element.Link
Simple link to execute functions to which datastore is passed.
-
__init__
(**kwargs)¶ Initialize an instance.
Parameters: - name (str) – name of link
- apply (list) – list of functions to execute at execute(), to which datastore is passed
-
execute
()¶ Execute the link.
Returns: status code of execution Return type: StatusCode
-
escore.core_ops.links.assert_in_ds module¶
Project: Eskapade - A python-based package for data analysis.
Class: AssertInDs
Created: 2016/11/08
- Description:
- Algorithm that asserts that items exists in the datastore
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.assert_in_ds.
AssertInDs
(**kwargs)¶ Bases:
escore.core.element.Link
Asserts that specified item(s) exists in the datastore.
-
__init__
(**kwargs)¶ Initialize link instance.
Store the configuration of link AssertInDs
Parameters: - name (str) – name of link
- keySet (lst) – list of keys to check
-
execute
()¶ Execute the link.
-
escore.core_ops.links.break_link module¶
Project: Eskapade - A python-based package for data analysis.
Class: Break
Created: 2017/02/26
- Description:
- Algorithm to send break signal to process manager and halt execution
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.break_link.
Break
(**kwargs)¶ Bases:
escore.core.element.Link
Halt execution.
Link sends failure signal and halts execution of process manager. Break the execution of the processManager at a specific location by simply adding this link at any location in a chain.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- send_break (bool) – if true, send StatusCode.BreakChain (skip execution of rest of chain). Default is false, then sends StatusCode.Failure (exit program).
-
execute
()¶ Execute the link.
-
escore.core_ops.links.ds_object_deleter module¶
Project: Eskapade - A python-based package for data analysis.
Class: DsObjectDeleter
Created: 2016/11/08
- Description:
- Algorithm to delete objects from the datastore.
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.ds_object_deleter.
DsObjectDeleter
(**kwargs)¶ Bases:
escore.core.element.Link
Delete objects from the datastore.
Delete objects from the DataStore by the key they are under, or keeps only the data by the specified keys.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- deletion_keys (list) – keys to clear. Overwrites clear_all to false.
- deletion_classes (list) – delete object(s) by class type.
- keep_only (lsst) – keys to keep. Overwrites clear_all to false.
- clear_all (bool) – clear all key-value pairs in the datastore. Default is true.
-
execute
()¶ Execute the link.
-
initialize
()¶ Initialize the link.
-
escore.core_ops.links.ds_to_ds module¶
Project: Eskapade - A python-based package for data analysis.
Class: DsToDs
Created: 2016/11/08
- Description:
- Algorithm to move, copy, or remove an object in the datastore.
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.ds_to_ds.
DsToDs
(**kwargs)¶ Bases:
escore.core.element.Link
Link to move, copy, or remove an object in the datastore.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- read_key (str) – key of data to read from data store
- store_key (str) – key of data to store in data store
- move (bool) – move read_key item to store_key. Default is true.
- copy (bool) – if True the read_key key, value pair will not be deleted. Default is false.
- remove (bool) – if True the item corresponding to read_key key will be deleted. Default is false.
-
execute
()¶ Execute the link.
-
initialize
()¶ Initialize the link.
-
escore.core_ops.links.event_looper module¶
Project: Eskapade - A python-based package for data analysis.
Class: EventLooper
Created: 2016/11/08
- Description:
- EventLooper algorithm processes input lines and reprints them, e.g. to use with map/reduce
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.event_looper.
EventLooper
(**kwargs)¶ Bases:
escore.core.element.Link
Event looper algorithm processes input lines and reprints or stores them.
Input lines are taken from sys.stdin, processed, and printed on screen.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- filename (str) – file name where the strings are located (txt or similar). Default is None. (optional)
- store_key (str) – key to collect in datastore. If set lines are collected. (optional)
- line_processor_set (list) – list of functions to apply to input lines. (optional)
- sort (bool) – if true, sort lines before storage (optional)
- unique (bool) – if true, keep only unique lines before storage (optional),
- skip_line_beginning_with (list) – skip line if it starts with any of the list. input is list of strings. Default is [‘#’] (optional)
-
execute
()¶ Process all incoming lines.
No output is printed except for lines that are passed on, such that the output lines can be picked up again by another parser.
-
finalize
()¶ Close open file if present.
-
initialize
()¶ Perform basic checks of configured attributes.
-
escore.core_ops.links.hello_world module¶
Project: Eskapade - A python-based package for data analysis.
Class: HelloWorld
Created: 2017/01/31
- Description:
- Algorithm to do print Hello {}!
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.hello_world.
HelloWorld
(**kwargs)¶ Bases:
escore.core.element.Link
Defines the content of link HelloWorld.
-
__init__
(**kwargs)¶ Store the configuration of link HelloWorld.
Parameters: - name (str) – name assigned to the link
- hello (str) – name to print in Hello World! Defaults to ‘World’
- repeat (int) – repeat print statement N times. Default is 1
-
execute
()¶ Execute the link.
-
escore.core_ops.links.import_data_store module¶
Project: Eskapade - A python-based package for data analysis.
Class: ImportDataStore
Created: 2018-03-17
- Description:
- Algorithm to import datastore from external pickle file
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.import_data_store.
ImportDataStore
(**kwargs)¶ Bases:
escore.core.element.Link
Link to import datastore from external pickle file.
Import can happen at initialize() or execute(). Default is initialize()
-
__init__
(**kwargs)¶ Initialize an instance of the datastore importer link.
Parameters: - name (str) – name of link
- path (str) – path of the datastore pickle file to import
- update (bool) – if true update the existing datastore, don’t replace it. Default is false.
- import_at_initialize (bool) – if false, perform datastore import at execute. Default is true, at initialize.
-
execute
()¶ Execute the link.
Returns: status code of execution Return type: StatusCode
-
import_and_update_datastore
()¶ Import and update the datastore
-
initialize
()¶ Initialize the link.
Returns: status code of execution Return type: StatusCode
-
escore.core_ops.links.ipython_embed module¶
Project: Eskapade - A python-based package for data analysis.
Class: IPythonEmbed
Created: 2017/02/26
- Description:
- Link that starts up a python console during execution for debugging.
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.ipython_embed.
IPythonEmbed
(**kwargs)¶ Bases:
escore.core.element.Link
Link to start up a python console.
Start up a python console by simply adding this link at any location in a chain. Note: not an ipython console, but regular python console.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: name (str) – name of link
-
execute
()¶ Execute the link.
-
escore.core_ops.links.line_printer module¶
Project: Eskapade - A python-based package for data analysis.
Class: LinePrinter
Created: 2017/02/21
- Description:
- Simple algorithm to pick up lines and reprint them.
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.line_printer.
LinePrinter
(**kwargs)¶ Bases:
escore.core.element.Link
LinePrinter picks up lines from the datastore and prints them.
-
__init__
(**kwargs)¶ Set up the configuration of link LinePrinter.
Parameters: - name (str) – name of link
- read_key (str) – key of input data to read from data store
-
execute
()¶ Execute the link.
No output is printed except for lines that are passed on, such that the output lines can be picked up again by another parser.
-
initialize
()¶ Initialize the link.
-
escore.core_ops.links.print_ds module¶
Project: Eskapade - A python-based package for data analysis.
Class: PrintDs
Created: 2016/11/08
- Description:
- Algorithm to print the content of the datastore.
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.print_ds.
PrintDs
(**kwargs)¶ Bases:
escore.core.element.Link
Print the content of the datastore.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- keys (list) – keys of items to print explicitly.
-
execute
()¶ Execute the link.
Print overview of the datastore in current state.
-
escore.core_ops.links.repeat_chain module¶
Project: Eskapade - A python-based package for data analysis.
Class: RepeatChain
Created: 2016/11/08
- Description:
- Algorithm that sends “repeat this chain” signal to processManager, until ready.
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.repeat_chain.
RepeatChain
(**kwargs)¶ Bases:
escore.core.element.Link
Algorithm that sends signal to processManager to repeat the current chain.
-
__init__
(**kwargs)¶ Link that sends signal to processManager to repeat the current chain.
Sents a RepeatChain deenums.StatusCode signal.
Parameters: - name (str) – name of link
- listen_to (list) – repeat this chain if given key is present in ConfigObject and set to true. E.g. this key is set by readtods link when looping over files.
- maxcount (int) – repeat this chain until max count has been reacher. Default is -1 (off).
-
execute
()¶ Execute the link.
-
initialize
()¶ Initialize the link.
-
escore.core_ops.links.skip_chain_if_empty module¶
Project: Eskapade - A python-based package for data analysis.
Class: SkipChainIfEmpty
Created: 2016/11/08
- Description:
- Algorithm to skip to the next Chain if input dataset is empty
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.skip_chain_if_empty.
SkipChainIfEmpty
(**kwargs)¶ Bases:
escore.core.element.Link
Sends a SkipChain enums.StatusCode signal when an appointed dataset is empty.
This signal causes that the Processs Manager to step immediately to the next Chain. Input collections can be either mongo collections or dataframes in the datastore.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- collection_set (list) – datastore keys holding the datasets to be checked. If any of these is empty, the chain is skipped.
- skip_missing (bool) – skip the chain if the dataframe is not present in the datastore. Default is True.
- skip_zero_len (bool) – skip the chain if the object is found in the datastore but has zero length. Default is True.
- check_at_initialize (bool) – perform dataset empty is check at initialize. Default is true.
-
check_collection_set
()¶ Check existence of collection in the datastore, and check if they are empty.
All collections need to be both present and not empty for the chain to be continued.
-
execute
()¶ Execute the link.
Skip to the next Chain if any of the input dataset is empty.
-
initialize
()¶ Initialize the link.
-
escore.core_ops.links.to_ds_dict module¶
Project: Eskapade - A python-based package for data analysis.
Class: ToDsDict
Created: 2016/11/08
- Description:
- Algorithm to store one object in the DataStore dict during run time.
- Authors:
- KPMG Advanced Analytics & Big Data team, Amstelveen, The Netherlands
Redistribution and use in source and binary forms, with or without modification, are permitted according to the terms listed in the file LICENSE.
-
class
escore.core_ops.links.to_ds_dict.
ToDsDict
(**kwargs)¶ Bases:
escore.core.element.Link
Stores one object in the DataStore dict during run time.
-
__init__
(**kwargs)¶ Link to store one external object in the DataStore dict during run time.
Parameters: - name (str) – name of link
- store_key (str) – key of object to store in data store
- obj – object to store
- force (bool) – overwrite if already present in datastore. default is false. (optional)
- at_initialize (bool) – store at initialize of link. Default is false.
- at_execute (bool) – store at execute of link. Default is true.
- copydict (bool) – if true and obj is a dict, copy all key value pairs into datastore. Default is false.
-
do_storage
(ds)¶ Perform storage in datastore.
Function makes a distinction been dicts and any other object.
-
execute
()¶ Execute the link.
-
initialize
()¶ Initialize the link.
-
Module contents¶
-
class
escore.core_ops.links.
AssertInDs
(**kwargs)¶ Bases:
escore.core.element.Link
Asserts that specified item(s) exists in the datastore.
-
__init__
(**kwargs)¶ Initialize link instance.
Store the configuration of link AssertInDs
Parameters: - name (str) – name of link
- keySet (lst) – list of keys to check
-
execute
()¶ Execute the link.
-
-
class
escore.core_ops.links.
Break
(**kwargs)¶ Bases:
escore.core.element.Link
Halt execution.
Link sends failure signal and halts execution of process manager. Break the execution of the processManager at a specific location by simply adding this link at any location in a chain.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- send_break (bool) – if true, send StatusCode.BreakChain (skip execution of rest of chain). Default is false, then sends StatusCode.Failure (exit program).
-
execute
()¶ Execute the link.
-
-
class
escore.core_ops.links.
DsObjectDeleter
(**kwargs)¶ Bases:
escore.core.element.Link
Delete objects from the datastore.
Delete objects from the DataStore by the key they are under, or keeps only the data by the specified keys.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- deletion_keys (list) – keys to clear. Overwrites clear_all to false.
- deletion_classes (list) – delete object(s) by class type.
- keep_only (lsst) – keys to keep. Overwrites clear_all to false.
- clear_all (bool) – clear all key-value pairs in the datastore. Default is true.
-
execute
()¶ Execute the link.
-
initialize
()¶ Initialize the link.
-
-
class
escore.core_ops.links.
DsToDs
(**kwargs)¶ Bases:
escore.core.element.Link
Link to move, copy, or remove an object in the datastore.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- read_key (str) – key of data to read from data store
- store_key (str) – key of data to store in data store
- move (bool) – move read_key item to store_key. Default is true.
- copy (bool) – if True the read_key key, value pair will not be deleted. Default is false.
- remove (bool) – if True the item corresponding to read_key key will be deleted. Default is false.
-
execute
()¶ Execute the link.
-
initialize
()¶ Initialize the link.
-
-
class
escore.core_ops.links.
EventLooper
(**kwargs)¶ Bases:
escore.core.element.Link
Event looper algorithm processes input lines and reprints or stores them.
Input lines are taken from sys.stdin, processed, and printed on screen.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- filename (str) – file name where the strings are located (txt or similar). Default is None. (optional)
- store_key (str) – key to collect in datastore. If set lines are collected. (optional)
- line_processor_set (list) – list of functions to apply to input lines. (optional)
- sort (bool) – if true, sort lines before storage (optional)
- unique (bool) – if true, keep only unique lines before storage (optional),
- skip_line_beginning_with (list) – skip line if it starts with any of the list. input is list of strings. Default is [‘#’] (optional)
-
execute
()¶ Process all incoming lines.
No output is printed except for lines that are passed on, such that the output lines can be picked up again by another parser.
-
finalize
()¶ Close open file if present.
-
initialize
()¶ Perform basic checks of configured attributes.
-
-
class
escore.core_ops.links.
HelloWorld
(**kwargs)¶ Bases:
escore.core.element.Link
Defines the content of link HelloWorld.
-
__init__
(**kwargs)¶ Store the configuration of link HelloWorld.
Parameters: - name (str) – name assigned to the link
- hello (str) – name to print in Hello World! Defaults to ‘World’
- repeat (int) – repeat print statement N times. Default is 1
-
execute
()¶ Execute the link.
-
-
class
escore.core_ops.links.
IPythonEmbed
(**kwargs)¶ Bases:
escore.core.element.Link
Link to start up a python console.
Start up a python console by simply adding this link at any location in a chain. Note: not an ipython console, but regular python console.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: name (str) – name of link
-
execute
()¶ Execute the link.
-
-
class
escore.core_ops.links.
LinePrinter
(**kwargs)¶ Bases:
escore.core.element.Link
LinePrinter picks up lines from the datastore and prints them.
-
__init__
(**kwargs)¶ Set up the configuration of link LinePrinter.
Parameters: - name (str) – name of link
- read_key (str) – key of input data to read from data store
-
execute
()¶ Execute the link.
No output is printed except for lines that are passed on, such that the output lines can be picked up again by another parser.
-
initialize
()¶ Initialize the link.
-
-
class
escore.core_ops.links.
PrintDs
(**kwargs)¶ Bases:
escore.core.element.Link
Print the content of the datastore.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- keys (list) – keys of items to print explicitly.
-
execute
()¶ Execute the link.
Print overview of the datastore in current state.
-
-
class
escore.core_ops.links.
RepeatChain
(**kwargs)¶ Bases:
escore.core.element.Link
Algorithm that sends signal to processManager to repeat the current chain.
-
__init__
(**kwargs)¶ Link that sends signal to processManager to repeat the current chain.
Sents a RepeatChain deenums.StatusCode signal.
Parameters: - name (str) – name of link
- listen_to (list) – repeat this chain if given key is present in ConfigObject and set to true. E.g. this key is set by readtods link when looping over files.
- maxcount (int) – repeat this chain until max count has been reacher. Default is -1 (off).
-
execute
()¶ Execute the link.
-
initialize
()¶ Initialize the link.
-
-
class
escore.core_ops.links.
SkipChainIfEmpty
(**kwargs)¶ Bases:
escore.core.element.Link
Sends a SkipChain enums.StatusCode signal when an appointed dataset is empty.
This signal causes that the Processs Manager to step immediately to the next Chain. Input collections can be either mongo collections or dataframes in the datastore.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- collection_set (list) – datastore keys holding the datasets to be checked. If any of these is empty, the chain is skipped.
- skip_missing (bool) – skip the chain if the dataframe is not present in the datastore. Default is True.
- skip_zero_len (bool) – skip the chain if the object is found in the datastore but has zero length. Default is True.
- check_at_initialize (bool) – perform dataset empty is check at initialize. Default is true.
-
check_collection_set
()¶ Check existence of collection in the datastore, and check if they are empty.
All collections need to be both present and not empty for the chain to be continued.
-
execute
()¶ Execute the link.
Skip to the next Chain if any of the input dataset is empty.
-
initialize
()¶ Initialize the link.
-
-
class
escore.core_ops.links.
ToDsDict
(**kwargs)¶ Bases:
escore.core.element.Link
Stores one object in the DataStore dict during run time.
-
__init__
(**kwargs)¶ Link to store one external object in the DataStore dict during run time.
Parameters: - name (str) – name of link
- store_key (str) – key of object to store in data store
- obj – object to store
- force (bool) – overwrite if already present in datastore. default is false. (optional)
- at_initialize (bool) – store at initialize of link. Default is false.
- at_execute (bool) – store at execute of link. Default is true.
- copydict (bool) – if true and obj is a dict, copy all key value pairs into datastore. Default is false.
-
do_storage
(ds)¶ Perform storage in datastore.
Function makes a distinction been dicts and any other object.
-
execute
()¶ Execute the link.
-
initialize
()¶ Initialize the link.
-
-
class
escore.core_ops.links.
DsApply
(**kwargs)¶ Bases:
escore.core.element.Link
Simple link to execute functions to which datastore is passed.
-
__init__
(**kwargs)¶ Initialize an instance.
Parameters: - name (str) – name of link
- apply (list) – list of functions to execute at execute(), to which datastore is passed
-
execute
()¶ Execute the link.
Returns: status code of execution Return type: StatusCode
-
-
class
escore.core_ops.links.
ImportDataStore
(**kwargs)¶ Bases:
escore.core.element.Link
Link to import datastore from external pickle file.
Import can happen at initialize() or execute(). Default is initialize()
-
__init__
(**kwargs)¶ Initialize an instance of the datastore importer link.
Parameters: - name (str) – name of link
- path (str) – path of the datastore pickle file to import
- update (bool) – if true update the existing datastore, don’t replace it. Default is false.
- import_at_initialize (bool) – if false, perform datastore import at execute. Default is true, at initialize.
-
execute
()¶ Execute the link.
Returns: status code of execution Return type: StatusCode
-
import_and_update_datastore
()¶ Import and update the datastore
-
initialize
()¶ Initialize the link.
Returns: status code of execution Return type: StatusCode
-
-
class
escore.core_ops.links.
ForkExample
(**kwargs)¶ Bases:
escore.core.element.Link
Defines the content of link.
-
__init__
(**kwargs)¶ Initialize an instance.
Parameters: - name (str) – name of link
- store_key (str) – key of object to store in data store
-
execute
()¶ Execute the link.
Returns: status code of execution Return type: StatusCode
-
finalize
()¶ Finalized the link.
Returns: status code of finalize Return type: StatusCode
-
initialize
()¶ Initialize the link.
Returns: status code of initialization Return type: StatusCode
-
-
class
escore.core_ops.links.
ForkDataCollector
(**kwargs)¶ Bases:
escore.core.element.Link
Defines the content of link.
-
__init__
(**kwargs)¶ Initialize an instance.
Parameters: - name (str) – name of link
- keys (list) – functions to apply (list of dicts) - ‘key_ds’ (string): input key in datastore - ‘key_fs’ (string, optional): output key in forkstore - ‘func’: function to apply, optional - ‘append’: if key_ds points to a list, append each item to list in forkstore. Default is True. - ‘args’ (tuple, optional): args for ‘func’ - ‘kwargs’ (dict, optional): kwargs for ‘func’
-
execute
()¶ Execute the link.
Returns: status code of execution Return type: StatusCode
-
finalize
()¶ Finalize the link.
Returns: status code of finalization Return type: StatusCode
-
initialize
()¶ Initialize the link.
Returns: status code of initialization Return type: StatusCode
-
-
class
escore.core_ops.links.
ApplyFunc
(*args, **kwargs)¶ Bases:
escore.core.element.Link
Algorithm that applies a function to an object in the datastore.
-
__init__
(*args, **kwargs)¶ Initialize an instance.
Parameters: - name (str) – name of link
- read_key (str) – key of input data to read from data store
- default – The default value of the key in case not found.
- assert_type – if set, check object for given type or tuple of types. If fails, raise TypeError.
- assert_len (bool) – if true, check that object has length greater than 0. If fails, raise TypeError or AssertionError.
- assert_in (bool) – assert that key is known, default is true.
- func – function to execute
- args – all args are passed pass to function as args.
- kwargs – all other key word arguments are passed on to the function as kwargs.
- store_key (str) – key of output data to store in data store
-
execute
()¶ Execute the link.
Returns: status code of execution Return type: StatusCode
-
initialize
()¶ Initialize the link.
Returns: status code of initialization Return type: StatusCode
-
-
class
escore.core_ops.links.
SkipChainIfPresent
(**kwargs)¶ Bases:
escore.core.element.Link
Sends a SkipChain enums.StatusCode signal when a requested dataset is present.
This signal causes that the Processs Manager to step immediately to the next Chain.
-
__init__
(**kwargs)¶ Initialize link instance.
Parameters: - name (str) – name of link
- collection_set (list) – datastore keys holding the datasets to be checked. If all of these are present, the chain is skipped.
- check_at_initialize (bool) – if false, perform dataset present check at execute. Default is true.
- assert_len (bool) – assert that each collection has a length greatet than zero. Default is true.
- assert_type (tuple) – types to assert for each object. Default is ().
-
check_collection_set
()¶ Check existence of collections in the datastore, and check that they are all present.
Collections need to be both present and not empty.
- For pandas dataframes the additional option ‘skip_chain_when_key_not_in_ds’ exists. Meaning, skip the chain as well if the dataframe is not present in the datastore.
-
execute
()¶ Execute the link.
Skip to the next Chain if all of the input collections are present.
-
initialize
()¶ Initialize the link.
-