Location>code7788 >text

ArgoWorkflow Tutorial (I) - DevOps Another Option? Cloud Native CICD First Experience

Popularity：142 ℃/2024-08-06 14:02:22

argo-workflow-logol

This article documents how to use ArgoWorkflow to build pipelines, as well as conceptual models such as Workflow and Template in ArgoWorkflow.

This paper analyzes the following issues:

  • 1) How to create a pipeline
  • 2) Workflow, Template, template own reference relationship
  • 3) Parameter passing issues between Workflow and Template
  • 4) ArgoWorkflow Pipeline Best Practices

1. Conceptual model

The next step is how to build our pipeline. The concepts related to pipelines in ArgoWorkflow are as follows:

  • Workflow: pipeline, a real running instance of a pipeline, similar to pipelinerun in Tekton.
  • WorkflowTemplate: pipeline template, you can create a pipeline based on the template, similar to the pipeline in Tekton.
  • ClusterWorkflowTemplate: cluster-level pipeline template, the relationship with WorkflowTemplate is similar to Role and ClusterRole in K8s.
  • templates: Workflow or Template's smallest component unit, the pipeline consists of multiple templates.

WorkflowTemplate and ClusterWorkflowTemplate are collectively referred to as Templates for now.

The relationship between Workflow, Template (uppercase), and template (lowercase) is as follows:

workflow-template-model

The relationship between the three is complicated, and officials have also mentioned that the naming of this area is confusing due to some historical problems.

Personal Understanding:

  • template (lowercase): the basic building block of a Template, understood as a step in a pipeline.
  • Template (uppercase): a complete pipeline, typically consisting of multiple templates (lowercase)
  • Workflow: the real running pipeline instance, generally created directly by the Template, similar to the pipeline run records, each record is a Workflow

Once the basic concepts are sorted out, the next step is to look at the specific uses.

In ArgoWorkflow, Workflow is a pipeline running instance, and every time a Workflow object is created, it will trigger the pipeline to run once.

Here's a simple Workflow example:

apiVersion: /v1alpha1
kind: Workflow
metadata:
  generateName: steps-
spec:
  entrypoint: hello           # We reference our first "template" here
  templates:
  - name: hello               # The first "template" in this Workflow, it is referenced by "entrypoint"
    steps:                    # The type of this "template" is "steps"
    - - name: hello
        template: whalesay    # We reference our second "template" here
        arguments:
          parameters: [{name: message, value: "hello"}]

  - name: whalesay             # The second "template" in this Workflow, it is referenced by "hello"
    inputs:
      parameters:
      - name: message
    container:                # The type of this "template" is "container"
      image: docker/whalesay
      command: [cowsay]
      args: ["{{}}"]

The core content of the entire Workflow object is divided into the following three parts:

  • entrypoint: A pipeline entry, similar to the main method in code, which generally references one of thetemplate invocators Ready to go.
  • templates: A list of templates where all the steps in the pipeline and the order of precedence between them are defined.
  • parameters: The parameters used in the pipeline, including global parameters in the arguments block and local parameters in the inputs block.

entrypoint

The entrypoint must be specified in the Workflow, and the entrypoint is the starting point for the execution of the task, similar to the main method in a program.

templates

See the official documentation for details:#template-types

Templates are categorized into various types, which are classified according to the type oftemplate definitions(template definition) andtemplate invocators(template caller).

  • template definitions: This type template is used to define what specific steps are to be performed, as in the whalesay template in the example.
    • embodycontainer, script, resource, suspend and other types
  • template invocators: This type template is used to combine othertemplate definitions The hello template in the example is of this type.
    • An entrypoint is generally a template of this type
    • embodydag cap (a poem)steps The hello template in the example is of type steps.

Spoiler alert: template is a bit convoluted, and it would be nice if theTemplate definition, template caller Splitting into two different objects is clearer.

Once you understand the template classification, it's clearer to go back and look at the previous Workflow examples:

apiVersion: /v1alpha1
kind: Workflow
metadata:
  generateName: steps-
spec:
  entrypoint: hello           # We reference our first "template" here
  templates:
  - name: hello               # The first "template" in this Workflow, it is referenced by "entrypoint"
    steps:                    # The type of this "template" is "steps"
    - - name: hello
        template: whalesay    # We reference our second "template" here
        arguments:
          parameters: [{name: message, value: "hello"}]

  - name: whalesay             # The second "template" in this Workflow, it is referenced by "hello"
    inputs:
      parameters:
      - name: message
    container:                # The type of this "template" is "container"
      image: docker/whalesay
      command: [cowsay]
      args: ["{{}}"]
  • 1) First of all the whalesay template is acontainer template of typetemplate definitions
  • 2) Secondly hello is asteps template of typetemplate invocators
    • The steps field in this caller defines a step named hello, which references the whalesay template
  • 3) Entrypoint specifies hello, thetemplate invocators

Summary:entrypoint, template definitions, and template invocators are the three components of a Workflow The necessary components of the

parameters

Workflow supports two types of parameters:

  • arguments: global arguments, available to all templates under Workflow.
  • inputs: local parameters, only available for the current template.

arguments Global arguments

is used to define global parameters that are available in all Templates under the current Workflow, and can be used with the{{.$name}} syntax to quote.

For example, the following example specifies a parameter named message and assigns it the value hello world.

  arguments:
    parameters:
      - name: message
        value: hello world

inputs Local parameters

Templates can use thetemplates[*].inputs field to specify thelocal parameterThis part of the parameter can only be used by the current Template. The parameters can be changed with the{{.$name}} syntax to refer to the parameters.

The following example declares that template requires a parameter called message, but does not provide a value.

  templates:
    - name: whalesay-template
      inputs:
        parameters:
          - name: message
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["{{}}"]

parameter passing

Global parameter values in Workflow are specified by the user, while values in local parameters are generally overridden by global parameters.

apiVersion: /v1alpha1
kind: Workflow
metadata:
  generateName: steps-
spec:
  entrypoint: hello           # We reference our first "template" here
  templates:
  - name: hello               # The first "template" in this Workflow, it is referenced by "entrypoint"
    steps:                    # The type of this "template" is "steps"
    - - name: hello
        template: whalesay    # We reference our second "template" here
        arguments:
          parameters: [{name: message, value: "hello"}]

  - name: whalesay             # The second "template" in this Workflow, it is referenced by "hello"
    inputs:
      parameters:
      - name: message
    container:                # The type of this "template" is "container"
      image: docker/whalesay
      command: [cowsay]
      args: ["{{}}"]

In the above example, template whalesay defines a local parameter named message, but does not assign a value to it, and defines a parameter named message in the template, so eventually the local parameter message will be overridden by the global parameter message.

The above example can be seen as such:

  templates:
  - name: hello               # The first "template" in this Workflow, it is referenced by "entrypoint"
    steps:                    # The type of this "template" is "steps"
    - - name: hello
        template: whalesay    # We reference our second "template" here
        arguments:
          parameters: [{name: message, value: "hello"}]

  - name: whalesay             # The second "template" in this Workflow, it is referenced by "hello"
    inputs:
      parameters:
      - name: message
        value: "{{}}"
    container:                # The type of this "template" is "container"
      image: docker/whalesay
      command: [cowsay]
      args: ["{{}}"]

That is: template references local parameters, local parameters come from global parameters, and the user configures global parameters.

As you can specify from the previous description, we can create Workflow objects directly to run the pipeline, but there are some problems with this approach:

  • 1) If there are a lot of templates, the Workflow object will be very large, which makes it more troublesome to modify it.
  • 2) templates can not be shared, different Workflow need to write the same template

Therefore, it is generally recommended to save the template to WorkflowTemplate, and only reference the Template in Workflow and provide parameters.

The workflow templates in ArgoWorkflow are categorized according to their scope asWorkflowTemplate cap (a poem)ClusterWorkflowTemplate Two kinds.

  • WorkflowTemplate: Namespace scope, can only be referenced in the same namespace
  • ClusterWorkflowTemplate: cluster scope, any namespace can be referenced

WorkflowTemplate

Here is a simple example of a WorkflowTemplate:

apiVersion: /v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-submittable
  namespace: default
spec:
  entrypoint: whalesay-template
  arguments:
    parameters:
      - name: message
        value: tpl-argument-default
  templates:
    - name: whalesay-template
      inputs:
        parameters:
          - name: message
            value: tpl-input-default
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["{{}}"]

You can see that the parameters of WorkflowTemplate and Workflow are exactly the same, just replace the kind in yaml from Workflow to WorkflowTemplate, so we won't go into details here.

workflowMetadata

The workflowMetadata is a field unique to Template and is mainly used for theStoring metadata，Subsequent Templates created by this Template Workflow All automatically carry this information with them。

With this information you can track which Template the Workflow was created from.

It is used like the following, with a label specified in the workflowMetadata

apiVersion: /v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-submittable
spec:
  workflowMetadata:
    labels:
      example-label: example-value

Workflow objects created by this Template will then carry this label:

apiVersion: /v1alpha1
kind: Workflow
metadata:
  annotations:
    /pod-name-format: v2
  creationTimestamp: "2023-10-27T06:26:13Z"
  generateName: workflow-template-hello-world-
  generation: 2
  labels:
    example-label: example-value
  name: workflow-template-hello-world-5w7ss
  namespace: default

ClusterWorkflowTemplate

Similar to WorkflowTemplate, it can be understood as the relationship between Role and ClusterRole in k8s.

and WorkflowTemplate are the same for all parameters, except that the yaml Replace kind with ClusterWorkflowTemplate.

wrap-up

It is a good idea to have all three of the following sections configured in the Template:

  • entrypoint: Recommended, although you can specify it in the Workflow, but then every time you create a Workflow you need to see what templates are in the Template first, which is troublesome, so it's best to fill in the Template directly.
  • templates: It is necessary, the role of Template is to write templates.
  • arguments: It's a good idea to fill in the default value so that Workflow can use it without specifying any parameters.

Once the Template is populated with these three core elements, Workflow is very simple to use.

4. TemplateRef

Once you have created a Template, you can use TemplateRef to directly reference the corresponding template in Workflow, so that the Workflow object will be cleaner.

workflowTemplateRef

This can be done byworkflowTemplateReffield directly references the WorkflowTemplate.

Note 📢: hereneed Workflow and WorkflowTemplate in the same namespace.。

It's like this:

apiVersion: /v1alpha1
kind: Workflow
metadata:
  generateName: workflow-template-hello-world-
spec:
  arguments:
    parameters:
      - name: message
        value: "from workflow"
  workflowTemplateRef:
    name: workflow-template-submittable

workflowTemplateRef Specify the name of the Template you want to reference, this sentence is equivalent to moving all the contents under the spec field of the corresponding Template, including entrypoint, template and so on.

Generally, Workflow only needs to override arguments with the argument field, but of course you can leave it unspecified. If you don't specify an argument in Workflow, the default value provided in the Template will be used.

A minimalist Workflow looks like this:

apiVersion: /v1alpha1
kind: Workflow
metadata:
  generateName: workflow-template-hello-world-
spec:
  workflowTemplateRef:
    name: workflow-template-submittable

only ifworkflowTemplateRef field, no argument is provided.

Contrasted with the previous Workflow, there is even less content, because most of it is written in the WorkflowTemplate, and the Workflow usually just needs to specify the parameters.

Parameter issues:

  • Global parameters can be used directly in the WorkflowTemplate, they just need to be defined in the Workflow by then.
  • Local parameters must be defined and assigned values in the WorkflowTemplate.

See the official documentation for details:#working-with-parameters

It seems that it is better to use global parameters in WorkflowTemplate, because the values of local parameters are defined directly in WorkflowTemplate and cannot be overridden in Workflow.

clusterWorkflowTemplateRef

Similar to workflowTemplateRef, just add theclusterScope: true Configuration is sufficient.

The default is false, i.e. WorkflowTemplate

It's like this:

apiVersion: /v1alpha1
kind: Workflow
metadata:
  generateName: workflow-template-hello-world-
spec:
  entrypoint: whalesay
  templates:
  - name: whalesay
    steps:                              # You should only reference external "templates" in a "steps" or "dag" "template".
      - - name: call-whalesay-template
          templateRef:                  # You can reference a "template" from another "WorkflowTemplate or ClusterWorkflowTemplate" using this field
            name: cluster-workflow-template-whalesay-template   # This is the name of the "WorkflowTemplate or ClusterWorkflowTemplate" CRD that contains the "template" you want
            template: whalesay-template # This is the name of the "template" you want to reference
            clusterScope: true          # This field indicates this templateRef is pointing ClusterWorkflowTemplate
          arguments:                    # You can pass in arguments as normal
            parameters:
            - name: message
              value: "hello world"

Core components:

  workflowTemplateRef:
    name: cluster-workflow-template-submittable
    clusterScope: true

When specified as cluster-scoped, ArgoWorkflow will go searching for ClusterWorkflowTemplate objects, otherwise it will search for WorkflowTemplate in the current namespace.

templateRef

In addition to using workflowTemplateRef / clusterWorkflowTemplateRef to reference an entire WorkflowTemplate, you can also use thetemplateRef parameter to reference a template in the WorkflowTemplate.

Note 📢: according to the official documentation, it's best not to use thesteps cap (a poem)dag Use templateRef in addition to templates of type templateRef.

apiVersion: /v1alpha1
kind: Workflow
metadata:
  generateName: workflow-template-hello-world-
spec:
  entrypoint: whalesay
  templates:
  - name: whalesay
    steps:                              # You should only reference external "templates" in a "steps" or "dag" "template".
      - - name: call-whalesay-template
          templateRef:                  # You can reference a "template" from another "WorkflowTemplate" using this field
            name: workflow-template-1   # This is the name of the "WorkflowTemplate" CRD that contains the "template" you want
            template: whalesay-template # This is the name of the "template" you want to reference
          arguments:                    # You can pass in arguments as normal
            parameters:
            - name: message
              value: "hello world"

5. Parameter prioritization tests

Since there are multiple places where parameters can be set, and at the same time there are differences such as global and local parameters, it is not very clear about the specific logic of the parameters, so we tested the specific parameter passing situation.

There are a total of three places to set the parameters:

  • 1) WorkflowTemplate arguments global parameters
  • 2) WorkflowTemplate inputs local parameters
  • 3) Workflow arguments global parameters

Note 📢: With workflowTemplateRef references, there is no way to specify input values in Workflow.

Tests were done for different scenarios and the final conclusions are as follows for:

Workflow hit the nail on the head argument > argument in Template > Template input

Whenever a higher priority parameter is supplied, the lower priority overrides the

  • Argument > Input i.e., global parameters have higher priority than local parameters.
  • Workflow > Template i.e. pipeline parameters can override the default values in the template

6. Summary

Workflow in ArgoWorkflow includes the following concepts:

  • Workflow
  • Template：WorkflowTemplate、ClusterWorkflowTemplate
  • TemplateRef：workflowTemplateRef、clusterWorkflowTemplateRef
  • Template

The relationship is shown in the figure

workflow-template-model

A Workflow / WorkflowTemplate contains three parts of parameters:

  • entrypoint: entry point, required, value is the name of the Template.
    • can be derived from Workflow, i.e. explicitly written inside Workflow.
    • It can also be derived from the WorkflowTemplate, i.e., the field is not written in Workflow, but in the WorkflowTemplate.
    • It's best to write it directly in the WorkflowTemplate.
  • arguments: arguments, again, can be specified by both Workflow and WorkflowTemplate, and arguments in Workflow have higher priority than WorkflowTemplate.
  • templates: templates, this section contains the actions that Workflow will perform.
    • This part is recommended to be defined in the WorkflowTemplate, and the WorkflowTemplate can be directly referenced in Workflow.

Best Practices:

  • 1) Use Template to manage the pipeline, and define the entrypoint and templates and arguments in WorkflowTemplate.
    • Template specifies entrypoints, templates, arguments, etc.
  • 2) Use TemplateRef in Workflow to directly reference an existing Template and specify arguments to override.

Parameter Priority:The argument > argument in Template > Template input

[ArgoWorkflow Series]Continuously updated, search the public number [Explore Cloud Native]Subscribe to read more articles.

<Previous Next>