Workflow

The workflow component is responsible for converting a Gordo config into an Argo workflow which then runs the various components in order to build and serve the ML models.

Normalized Config

class gordo.workflow.config_elements.normalized_config.NormalizedConfig(config: dict, project_name: str, gordo_version: str | None = None, model_builder_env: dict | None = None, back_compatibles: dict[tuple[Optional[str], str], tuple[Optional[str], str]] | None = None, default_data_provider: str | None = None, json_path: str | None = None)

Bases: object

Handles the conversion of a single Machine representation in config format and updates it with any features which are ‘left out’ inside of globals key or the default config globals held here.

DEFAULT_CONFIG_GLOBALS: GlobalsConfig = {'evaluation': {'cv_mode': 'full_build', 'metrics': ['explained_variance_score', 'r2_score', 'mean_squared_error', 'mean_absolute_error'], 'scoring_scaler': 'sklearn.preprocessing.MinMaxScaler'}, 'runtime': {'builder': {'remote_logging': {'enable': False}, 'resources': {'limits': {'cpu': 1001, 'memory': 31200}, 'requests': {'cpu': 1001, 'memory': 3900}}}, 'client': {'max_instances': 30, 'resources': {'limits': {'cpu': 2000, 'memory': 4000}, 'requests': {'cpu': 100, 'memory': 3500}}}, 'influx': {'enable': True}, 'prometheus_metrics_server': {'resources': {'limits': {'cpu': 200, 'memory': 1000}, 'requests': {'cpu': 100, 'memory': 200}}}, 'reporters': [], 'server': {'resources': {'limits': {'cpu': 2000, 'memory': 6000}, 'requests': {'cpu': 1000, 'memory': 3000}}}}}

SPLITED_DOCKER_IMAGES: GlobalsConfig = {'runtime': {'builder': {'image': 'gordo-model-builder'}, 'client': {'image': 'gordo-client'}, 'deployer': {'image': 'gordo-deploy'}, 'prometheus_metrics_server': {'image': 'gordo-model-server'}, 'server': {'image': 'gordo-model-server'}}}

UNIFIED_DOCKER_IMAGES: GlobalsConfig = {'runtime': {'builder': {'image': 'gordo-base'}, 'client': {'image': 'gordo-base'}, 'deployer': {'image': 'gordo-base'}, 'prometheus_metrics_server': {'image': 'gordo-base'}, 'server': {'image': 'gordo-base'}}}

UNIFYING_GORDO_VERSION: str = '1.2.0'

classmethod get_default_globals(gordo_version: str) → GlobalsConfig

classmethod prepare_patched_globals(patched_globals: GlobalsConfig) → GlobalsConfig

static prepare_runtime(runtime: dict) → dict

Workflow Generator

Workflow loading/processing functionality to help the CLI ‘workflow’ sub-command.

gordo.workflow.workflow_generator.workflow_generator.default_image_pull_policy(gordo_version: Version) → str

gordo.workflow.workflow_generator.workflow_generator.get_dict_from_yaml(config_file: str | StringIO) → dict

Read a config file or file like object of YAML into a dict

gordo.workflow.workflow_generator.workflow_generator.load_workflow_template(workflow_template: str) → Template

Loads the Jinja2 Template from a specified path

Parameters:

workflow_template – Path to a workflow template

Return type:

Loaded but non-rendered jinja2 template for the workflow

gordo.workflow.workflow_generator.workflow_generator.yaml_filter(data: Any) → str

Schemas

class gordo.workflow.config_elements.schemas.BuilderPodRuntime(*, image: str, resources: ResourceRequirements | None = None, metadata: ObjectMeta | None = None, env: List[EnvVar] | None = None, volumeMounts: List[VolumeMount] | None = None, remote_logging: RemoteLogging)

Bases: PodRuntime

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

remote_logging: RemoteLogging

class gordo.workflow.config_elements.schemas.CSIVolumeSource(*, driver: str, readOnly: bool = False, fsType: str | None = None, volumeAttributes: Dict[str, Any] | None = None)

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

driver: str

fsType: str | None

readOnly: bool

volumeAttributes: Dict[str, Any] | None

class gordo.workflow.config_elements.schemas.Capabilities(*, add: List[str] | None = None, drop: List[str] | None = None)

Bases: Model

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

add: List[str] | None

drop: List[str] | None

class gordo.workflow.config_elements.schemas.ConfigMapKeySelector(*, key: str, value: str)

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

key: str

value: str

class gordo.workflow.config_elements.schemas.EnvVar(*, name: str, value: str | None = None, valueFrom: EnvVarSource | None = None)

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

name: str

value: str | None

valueFrom: EnvVarSource | None

class gordo.workflow.config_elements.schemas.EnvVarSource(*, configMapKeyRef: ConfigMapKeySelector | None = None, secretKeyRef: SecretKeySelector | None = None)

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

configMapKeyRef: ConfigMapKeySelector | None

secretKeyRef: SecretKeySelector | None

class gordo.workflow.config_elements.schemas.Model

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

class Config

Bases: object

extra = 'forbid'

class gordo.workflow.config_elements.schemas.ObjectMeta(*, labels: Dict[str, Any] | None = None)

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

labels: Dict[str, Any] | None

class gordo.workflow.config_elements.schemas.PodRuntime(*, image: str, resources: ResourceRequirements | None = None, metadata: ObjectMeta | None = None, env: List[EnvVar] | None = None, volumeMounts: List[VolumeMount] | None = None)

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

env: List[EnvVar] | None

image: str

metadata: ObjectMeta | None

resources: ResourceRequirements | None

volumeMounts: List[VolumeMount] | None

class gordo.workflow.config_elements.schemas.PodSecurityContext(*, fsGroup: int | None = None, fsGroupChangePolicy: str | None = None, runAsGroup: int | None = None, runAsNonRoot: bool | None = None, runAsUser: int | None = None, seLinuxOptions: SELinuxOptions | None = None, seccompProfile: SeccompProfile | None = None, supplementalGroups: List[int] | None = None, sysctls: List[Sysctl] | None = None, windowsOptions: WindowsSecurityContextOptions | None = None)

Bases: Model

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

fsGroup: int | None

fsGroupChangePolicy: str | None

runAsGroup: int | None

runAsNonRoot: bool | None

runAsUser: int | None

seLinuxOptions: SELinuxOptions | None

seccompProfile: SeccompProfile | None

supplementalGroups: List[int] | None

sysctls: List[Sysctl] | None

windowsOptions: WindowsSecurityContextOptions | None

class gordo.workflow.config_elements.schemas.RemoteLogging(*, enable: bool = False)

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

enable: bool

class gordo.workflow.config_elements.schemas.ResourceRequirements(*, requests: Dict[str, Any] | None = None, limits: Dict[str, Any] | None = None)

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

limits: Dict[str, Any] | None

requests: Dict[str, Any] | None

class gordo.workflow.config_elements.schemas.SELinuxOptions(*, level: str | None = None, role: str | None = None, type: str | None = None, user: str | None = None)

Bases: Model

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

level: str | None

role: str | None

type: str | None

user: str | None

class gordo.workflow.config_elements.schemas.SeccompProfile(*, localhostProfile: str | None = None, type: str | None = None)

Bases: Model

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

localhostProfile: str | None

type: str | None

class gordo.workflow.config_elements.schemas.SecretKeySelector(*, key: str, name: str)

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

key: str

name: str

class gordo.workflow.config_elements.schemas.SecurityContext(*, allowPrivilegeEscalation: bool | None = None, capabilities: Capabilities | None = None, privileged: bool | None = None, procMount: str | None = None, readOnlyRootFilesystem: bool | None = None, runAsNonRoot: bool | None = None, runAsGroup: bool | None = None, runAsUser: int | None = None, seLinuxOptions: SELinuxOptions | None = None, seccompProfile: SeccompProfile | None = None, windowsOptions: WindowsSecurityContextOptions | None = None)

Bases: Model

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

allowPrivilegeEscalation: bool | None

capabilities: Capabilities | None

privileged: bool | None

procMount: str | None

readOnlyRootFilesystem: bool | None

runAsGroup: bool | None

runAsNonRoot: bool | None

runAsUser: int | None

seLinuxOptions: SELinuxOptions | None

seccompProfile: SeccompProfile | None

windowsOptions: WindowsSecurityContextOptions | None

class gordo.workflow.config_elements.schemas.Sysctl(*, name: str, value: str)

Bases: Model

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

name: str

value: str

class gordo.workflow.config_elements.schemas.Volume(*, name: str, csi: CSIVolumeSource | None = None)

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

csi: CSIVolumeSource | None

name: str

class gordo.workflow.config_elements.schemas.VolumeMount(*, name: str, readOnly: bool | None = None, mountPath: str)

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

mountPath: str

name: str

readOnly: bool | None

class gordo.workflow.config_elements.schemas.WindowsSecurityContextOptions(*, gmsaCredentialSpec: str | None = None, gmsaCredentialSpecName: str | None = None, runAsUserName: str | None = None)

Bases: Model

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

gmsaCredentialSpec: str | None

gmsaCredentialSpecName: str | None

runAsUserName: str | None

Helpers

exception gordo.workflow.workflow_generator.helpers.ArgoVersionError

Bases: Exception

gordo.workflow.workflow_generator.helpers.determine_argo_version(argo_binary: str = 'argo') → str

Check installed Argo CLI version.

Raises:

ArgoVersionError – If something went wrong.

Return type:

Version of installed argo version.

gordo.workflow.workflow_generator.helpers.parse_argo_version(argo_version: str) → Version | None

Try to parse Argo version.

Parameters:

argo_version –

Return type:

None if failed to parse.

gordo.workflow.workflow_generator.helpers.patch_dict(original_dict: dict, patch_dictionary: dict) → dict

Patches a dict with another. Patching means that any path defines in the patch is either added (if it does not exist), or replaces the existing value (if it exists). Nothing is removed from the original dict, only added/replaced.

Parameters:

• original_dict – Base dictionary which will get paths added/changed

• patch_dictionary – Dictionary which will be overlaid on top of original_dict

Examples

>>> patch_dict({"highKey":{"lowkey1":1, "lowkey2":2}}, {"highKey":{"lowkey1":10}})
{'highKey': {'lowkey1': 10, 'lowkey2': 2}}
>>> patch_dict({"highKey":{"lowkey1":1, "lowkey2":2}}, {"highKey":{"lowkey3":3}})
{'highKey': {'lowkey1': 1, 'lowkey2': 2, 'lowkey3': 3}}
>>> patch_dict({"highKey":{"lowkey1":1, "lowkey2":2}}, {"highKey2":4})
{'highKey': {'lowkey1': 1, 'lowkey2': 2}, 'highKey2': 4}

Returns:

• A new dictionary which is the result of overlaying patch_dictionary on top of

• original_dict