Parameters versioning
Automatic parameters versioning
Parameters versioning is automatic when the MlflowNodeHook
is added to the hook list of the ProjectContext
. In kedro-mlflow==0.7.6
, the mlflow.yml
configuration file has a parameter called flatten_dict_params
which enables to log as distinct parameters the (key, value) pairs of a ```Dict`` parameter.
You do not need any additional configuration to benefit from parameters versioning.
How does MlflowNodeHook
operates under the hood?
The medium post which introduces hooks explains in detail the differents execution steps Kedro
executes when the user calls the kedro run
command.
The MlflowNodeHook
registers the parameters before each node (entry point number 3 on above picture) by calling mlflow.log_parameter(param_name, param_value)
on each parameters of the node.
Frequently Asked Questions
Will parameters be recorded if the pipeline fails during execution?
The parameters are registered node by node (and not in a single batch at the beginning of the execution). If the pipeline fails in the middle of its execution, the parameters of the nodes who have been run will be recorded, but not the parameters of non executed nodes.
How are parameters detected by the plugin?
The hook detects parameters through their prefix params:
or the value parameters
. These are the reserved keywords used by Kedro to define parameters in the pipeline.py
file(s).
How can I register a parameter if I use a TemplatedConfigLoader
?
If you use a TemplatedConfigLoader
to enable dynamic parameters contruction at runtime or dependency between configuration files, and if we assume your src/<project-name>/run.py
file looks like:
from kedro.config import TemplatedConfigLoader # new import
from datetime import date
class ProjectContext(KedroContext):
def _create_config_loader(self, conf_paths: Iterable[str]) -> TemplatedConfigLoader:
return TemplatedConfigLoader(
conf_paths,
globals_pattern="*globals.yml", # read the globals dictionary from project config
globals_dict={ # extra keys to add to the globals dictionary, take precedence over globals_pattern
execution_date: date.today()
},
)
Then you need to add this entry in your conf/<env>/parameters
to ensure that the parameter will be properly recorded:
execution_date: ${execution_date}