Skip to main content

Matrix jobs

Matrix jobs enable parallel execution of a job with different parameter combinations. You define variables with multiple values, and the system creates a separate job instance for each combination using the Cartesian product.

No include / exclude filtering: A matrix always expands to the full Cartesian product of its variables. There is no include or exclude filter — a key named include or exclude under matrix is treated as just another matrix variable (and its values become part of the combinations).

Schema​

See Schema Reference for the complete #Matrix definition.

Defining Matrices​

Basic Matrix​

A matrix with a single variable creates multiple parallel job instances:

jobs:
  train-model:
    name: Train with Different Algorithms
    matrix:
      algorithm: [xgboost, lightgbm, random_forest, neural_net]
    runner: gpu-large
    timeout: "2h"
    steps:
      - name: Train model
        env:
          ALGORITHM: ${{ .matrix.algorithm }}
        run: |
          python train.py --algorithm $ALGORITHM --data ./data/train.csv

This creates 4 job instances, one for each algorithm.

Multi-Variable Matrix​

Multiple variables create a Cartesian product of all combinations:

jobs:
  hyperparameter-search:
    name: Hyperparameter Tuning
    matrix:
      learning_rate: ["0.001", "0.01", "0.1"]
      batch_size: ["32", "64", "128"]
    runner: gpu-large
    timeout: "3h"
    steps:
      - name: Train with hyperparameters
        env:
          LR: ${{ .matrix.learning_rate }}
          BATCH: ${{ .matrix.batch_size }}
        run: |
          echo "Training with learning_rate=$LR and batch_size=$BATCH"
          python train.py --lr $LR --batch-size $BATCH --epochs 100

      - name: Upload model
        upload:
          path: models/model.pkl
          destination: drive://models/lr-${{ .matrix.learning_rate }}-batch-${{ .matrix.batch_size }}/

This creates 9 job instances (3 learning rates × 3 batch sizes). Each instance gets a generated job ID of the form <job>-<value1>-<value2> and a display name of the form <name> (<value1>, <value2>). The matrix values are ordered by variable name (sorted alphabetically — here batch_size before learning_rate), and the variable names themselves are not part of the ID or display name:

Job IDDisplay name
hyperparameter-search-32-0.001Hyperparameter Tuning (32, 0.001)
hyperparameter-search-32-0.01Hyperparameter Tuning (32, 0.01)
hyperparameter-search-64-0.001Hyperparameter Tuning (64, 0.001)
......

(The display name uses the job's name if set, otherwise the job ID. When the job has no matrix, the ID and display name are just the job ID / name with no suffix.)

Matrix with Workflow Calls​

Matrices work with reusable workflows, passing matrix variables as inputs:

jobs:
  evaluate-models:
    name: Evaluate on Multiple Datasets
    matrix:
      model: [xgboost, lightgbm, random_forest]
      dataset: [train, validation, test]
    workflow:
      name: model-evaluation
      inputs:
        model_type: ${{ .matrix.model }}
        dataset_name: ${{ .matrix.dataset }}
        metrics: "accuracy,f1,auc"

This creates 9 job instances (3 models × 3 datasets), each calling the model-evaluation workflow with different parameters.

Matrix Expressions​

Accessing Matrix Variables​

Matrix variables are accessed using the ${{ .matrix.variable_name }} expression syntax, consistent with other workflow expressions.

Format: ${{ .matrix.<variable_name> }}

Example:

jobs:
  process:
    matrix:
      region: [us-east, eu-west, ap-south]
      data_type: [transactions, events]
    steps:
      - name: Download data
        download:
          source: drive://data/${{ .matrix.region }}/${{ .matrix.data_type }}/
          path: ./data/

      - name: Process data
        env:
          REGION: ${{ .matrix.region }}
          TYPE: ${{ .matrix.data_type }}
        run: python process.py --region $REGION --type $TYPE

Job Expansion and Execution​

Parallel Execution​

All matrix job instances run in parallel by default. There is no automatic sequencing or max-parallel limiter—all combinations execute concurrently as soon as the job's dependencies are satisfied.

Dependency Handling​

When a job depends on a matrix job, it waits for ALL matrix instances to complete successfully.

Example:

jobs:
  train:
    matrix:
      model: [xgboost, lightgbm, random_forest]
    steps:
      - run: python train.py --model ${{ .matrix.model }}

  evaluate:
    depends_on: [train]  # Waits for all 3 training instances
    steps:
      - run: python evaluate_all.py --models-dir ./models/

Examples​

Example 1: Hyperparameter Optimization​

id: hyperparameter-optimization
name: Train Models with Different Hyperparameters

inputs:
  dataset_bucket:
    type: string
    required: true
    description: "Training dataset bucket UUID"

env:
  SCRIPTS_REPO: "https://github.com/h2oai/ml-training"

jobs:
  train-models:
    name: Train with Hyperparameters
    matrix:
      algorithm: [xgboost, lightgbm, random_forest]
      max_depth: ["5", "10", "15"]
    runner: gpu-large
    timeout: "3h"

    steps:
      - name: Download training data
        download:
          source: drive://${{ .inputs.dataset_bucket }}/train.csv
          path: ./data/train.csv

      - name: Clone training scripts
        run: git clone --depth 1 $SCRIPTS_REPO scripts

      - name: Train model
        env:
          ALGORITHM: ${{ .matrix.algorithm }}
          MAX_DEPTH: ${{ .matrix.max_depth }}
        run: |
          echo "Training $ALGORITHM with max_depth=$MAX_DEPTH"
          python scripts/train.py \
            --algorithm $ALGORITHM \
            --max-depth $MAX_DEPTH \
            --data ./data/train.csv \
            --output ./models/

      - name: Upload trained model
        upload:
          path: ./models/
          destination: drive://models/${{ .matrix.algorithm }}-depth${{ .matrix.max_depth }}/

      - name: Upload metrics
        upload:
          path: metrics.json
          destination: drive://metrics/${{ .matrix.algorithm }}-depth${{ .matrix.max_depth }}.json

This creates 9 parallel training runs (3 algorithms × 3 max_depth values).

Feedback

Related content

Job structure
Expressions
Matrix data processing