Skip to main content

Concurrency control

Concurrency control prevents multiple workflow instances from running simultaneously, protecting shared resources and preventing conflicts.

Schema​

See Schema Reference for the complete #Concurrency definition.

Fields​

group (required)​

Concurrency group identifier. Workflows with the same group cannot run concurrently.

Type: string (non-empty)

Supports expressions: Yes

Example:

concurrency:
  group: production-deploy

With expressions:

concurrency:
  group: train-${{ .inputs.model_type }}

cancel_in_progress (optional)​

Controls behavior when a new workflow arrives while another is running in the same group.

Type: bool

Default: false

Values:

  • true: Cancel running workflow, start new one immediately.
  • false: Queue new workflow (max 1 running + 1 pending, newer replaces pending).

Example:

concurrency:
  group: model-training
  cancel_in_progress: true

Behavior​

Queue Mode (cancel_in_progress: false)​

  • Maximum: 1 running + 1 pending workflow.
  • New arrivals replace pending workflow (keeps newest).
  • When running completes, pending starts.

Queue timeout: A run does not wait in the queue indefinitely. A run that stays in STATE_QUEUED longer than the orchestrator's maximum queued duration (configurable via dispatch.maxQueuedDuration, default 3h; 0 disables the limit) is force-cancelled. Like a concurrency cancellation, this is recorded as a failed run (STATE_FAILED) with failure_reason = RUN_FAILURE_REASON_CANCELED.

Cancel Mode (cancel_in_progress: true)​

  • New workflow cancels running workflow.
  • No queue — newest always runs.

Examples​

Model Deployment​

Ensure only one model deployment at a time:

id: deploy-model

concurrency:
  group: production-deploy

jobs:
  deploy:
    steps:
      - run: python deploy_model.py --env production

Per-Model Training​

Allow concurrent training of different models, but serialize same-model training:

id: train-model

inputs:
  model_type:
    type: string
    required: true

concurrency:
  group: train-${{ .inputs.model_type }}

jobs:
  train:
    steps:
      - run: python train.py --model ${{ .inputs.model_type }}

Experiment with Cancel​

For experiments where only the latest hyperparameters matter:

id: hyperparameter-tuning

inputs:
  experiment_id:
    type: string

concurrency:
  group: experiment-${{ .inputs.experiment_id }}
  cancel_in_progress: true

jobs:
  tune:
    steps:
      - run: python tune.py --experiment ${{ .inputs.experiment_id }}

Interaction with Other Features​

cancel_on_failure: There is no distinct cancelled run state. A concurrency cancellation is recorded as a failed run (state STATE_FAILED) with failure_reason = RUN_FAILURE_REASON_CANCELED. The failure_reason is what distinguishes a cancellation from an ordinary failure, and cancellations do not trigger cancel_on_failure in other workflows.

Expressions: Concurrency groups support .inputs and .env expressions. .secrets references are rejected at validation time because group values are persisted and queried, so they must not carry secret material.

Feedback

Related content

Workflow structure
Failure handling
Expressions