Custom Recipe Lifecycle and Troubleshooting

This page describes what Driverless AI does with a custom recipe between upload and use, and how to diagnose a recipe that fails to upload or does not appear in an experiment. For instructions on adding recipes, see Adding Custom Recipes. For guidance on writing recipe code, see the How to Write a Recipe guide.

Recipe Upload Sequence

Adding a recipe starts a background task. The Recipes page reports the progress of that task through the following stages:

Staging. Driverless AI extracts ZIP uploads, then writes the file to the contrib/<recipe type> directory under the Driverless AI data directory. The contrib_relative_directory setting controls the base directory name.
Static code analysis. When you turn on custom_recipe_security_analysis_enabled, Driverless AI scans the source before importing it and rejects any file that is not Python source. This is a backend configuration option, off by default, so it does not appear in the generated config.toml template or in Expert Settings. For the full set of recipe security options, see Custom Recipe Security.
Global package installation. Driverless AI reads _global_modules_needed_by_name from the file and installs those packages. It does this in a first pass, before importing the rest of the module, so your recipe can import those packages at module level. Set this parameter on a single line: line breaks in the value cause a syntax error during upload.
Module import. Driverless AI imports the file as a Python module. A syntax error, a failed import, or a module-level sys.exit() fails the upload at this point, and Driverless AI deletes the staged file.
Class discovery and per-class packages. Driverless AI collects the recipe classes in the module and installs each class’s _modules_needed_by_name packages. It skips any class whose is_enabled() returns False, and any class whose packages fail to install.
Versioning. Driverless AI deactivates any existing recipe that shares a class name or _display_name with the incoming one. See Custom Recipe Management for the versioning rules.
Acceptance testing. Driverless AI tests each new class before accepting it. See Acceptance Testing.
Activation. Recipes that pass become selectable in the include lists under Expert Settings > Training.

Driverless AI runs this check again on recipes it already holds: at server startup, or at user login if the server uses per-user directories. It removes any recipe that no longer passes, which protects previews and experiments from recipes that have fallen out of date with the current API. With many stored recipes, this makes startup considerably slower. Set contrib_reload_and_recheck_server_start to false to skip it.

Acceptance Testing

Acceptance testing runs a recipe against small, randomly generated data to catch problems at upload time rather than partway through an experiment. It runs in two stages, both enabled by default.

Basic acceptance tests (enable_basic_acceptance_tests) confirm that Driverless AI can pickle the recipe class and its blueprint. This stage also rejects a recipe that imports xgboost, lightgbm, torch, transformers, cudf, or tensorflow at module level. Import those packages inside your methods instead.

Acceptance tests (enable_acceptance_tests) exercise the recipe itself. What runs depends on the recipe type:

Recipe type	Test data	Checks
Transformer	200 random rows (`num_rows_acceptance_test_custom_transformer`), with columns generated to match the declared `col_type`	`get_default_properties()` returns a consistent `min_cols`, `max_cols`, and `col_type`; every key returned by `get_parameter_choices()` appears as an argument of `__init__`; fit and transform succeed across sampled parameter combinations
Model	100 random rows (`num_rows_acceptance_test_custom_model`) and 10 numeric columns, plus random text and categorical columns when the recipe sets `_can_handle_text` or `_can_handle_non_numeric`	Fit and predict succeed for each declared problem type, including binary labels of `0`/`1`, `-1`/`1`, and strings; the fitted model pickles with its full state; the model accepts a validation set through `eval_set` and `sample_weight_eval_set`; `mutate_params` succeeds across accuracy, time tolerance, and interpretability values
Scorer	Small fixed arrays of actual and predicted values for each declared problem type	The scorer pickles; `_description` is a string; the score stays consistent under row weights, so that a weight of 2 matches a duplicated row and a weight of 0 matches a dropped row
Data	None	Data recipes are not acceptance-tested

A model, transformer, or scorer recipe must declare at least one of _regression, _binary, _multiclass, or _unsupervised. If it declares none, acceptance testing fails with an error listing those four attributes.

Driverless AI skips acceptance testing, rather than failing it, when a recipe sets _must_use_gpu and the machine has no visible GPUs.

Timeout

Each recipe gets acceptance_test_timeout minutes, 20 by default. Driverless AI rejects a recipe that runs longer. The timeout excludes the time spent installing packages. This setting also appears in Expert Settings > Training as Timeout in minutes for testing acceptance of each recipe. To set a different limit for one recipe, define the acceptance_test_timeout() static method on the recipe class and return the number of minutes.

Opting Out

Return False from the do_acceptance_test() static method on a recipe class to exempt that recipe. Do this when the recipe needs specific data and cannot run against randomly generated input.

To turn acceptance testing off across the server, set enable_acceptance_tests or enable_basic_acceptance_tests to false. Driverless AI then accepts recipes without checking them, so any problem surfaces during the experiment preview or the experiment itself.

Why an Uploaded Recipe Does Not Appear

A recipe that uploaded successfully can still be absent from an experiment. First rule out the built-in selection hierarchy, which applies to custom and built-in components alike: see Why do my selected algorithms not show up in the Experiment Preview?. The following causes are specific to custom recipes:

A newer version replaced the recipe. Uploading a recipe that shares a class name or _display_name with an existing one deactivates the older version, and only active recipes appear in the include lists. Select Include inactive recipes on the Recipes page to see deactivated versions.
is_enabled() returns False. Driverless AI ignores the class entirely, including during acceptance testing.
can_use() returns False. The recipe is not used by default for the current settings and data shape. If every recipe in an include list returns False, Driverless AI falls back to the include list and ignores can_use(), so a recipe that returns False can still appear when nothing else qualifies.
enabled_setting() returns "auto". Automatic model and transformer selection can drop the recipe, and Driverless AI only adds it back when no other candidate remains. The custom model and transformer templates return "on", which keeps the recipe in the running whenever it applies.
The recipe requires a GPU that is not available. Driverless AI filters out a recipe with _must_use_gpu set when no GPUs are visible, including when CUDA_VISIBLE_DEVICES is empty or -1.
The name in the include list is wrong. Driverless AI matches include lists such as included_models against recipe display names. By default it logs an unrecognized name as a warning to the server log and then ignores it, which looks the same as the recipe never loading. Set raise_on_invalid_included_list to true to fail with an error that lists the invalid entries alongside the valid names.
The experiment pins recipe versions. The recipe_activation setting records which recipe versions an experiment may use, so that retrain and refit runs stay comparable with the parent experiment. When that list has entries, Driverless AI skips recipes outside it even when they are active and newer. An experiment created with NEW WITH SAME SETTINGS inherits the parent’s list.

Where to Look When Something Fails

Location	What it holds
Recipes page progress messages	The stage the upload reached and the message it failed with. Start here: this is visible to any user.
dai.log	Recipe load and acceptance messages from the server, naming the recipes it accepted and giving the stack trace for each one it rejected. Covers uploads and the startup re-check. Requires administrator access.
`<recipe file>.info` and `<recipe file>_<ClassName>.err`	Driverless AI writes these next to the recipe in its `contrib/<recipe type>` directory. The `.info` file lists classes that failed, and the matching `.err` file holds the stack trace. Driverless AI skips a class named in `.info` on later loads without retrying it, so edit and re-upload the recipe rather than restarting the server. Requires administrator access.
`contrib` directory in the experiment folder	The recipe code an experiment actually used, written per recipe type. Use this to confirm which version of a recipe a completed experiment ran with.
Experiment log	Recipe behavior during the experiment itself, including failures that acceptance testing did not catch. Download it from the experiment page.

Python and MOJO Scoring Support by Recipe Type

The Python Scoring Pipeline supports custom recipes. MOJO support depends on the recipe type and, for transformers and models, on the recipe itself.

Recipe type	MOJO support	Notes
Transformer	Opt-in	`_mojo` defaults to `False`. Set it to `True` and define `to_mojo()`. See the transformer template.
Model	Opt-in	`_mojo` defaults to `False`. Set it to `True` and define `write_to_mojo()`. Note the difference from transformers, which use `to_mojo()`. See the model template.
Scorer	Not applicable	Scorers rank models during an experiment; they do not produce predictions. `CustomScorer` has no `_mojo` attribute and no MOJO writer method.
Data	No	The MOJO scoring package does not support data recipes. Apply the data recipe before making MOJO predictions. See Modifying Datasets With Recipes.

Complex transformer and model recipes often need help from H2O.ai before a MOJO is available. See the notes in Custom Recipe Management.

Developing a Recipe with MOJO Support

While developing a recipe’s MOJO support, set Make MOJO scoring pipeline (make_mojo_scoring_pipeline) to On in Training > Deployment. The default of Auto can skip building a MOJO, which hides problems in the recipe’s MOJO support until later.

When Driverless AI builds a MOJO at the end of an experiment, it scores a small sample of data through the MOJO and compares the predictions against the Python pipeline. The mojo_acceptance_test_mojo_types setting controls which runtimes this comparison covers, C++ and Java by default. The comparison tolerances (mojo_acceptance_test_rtol and mojo_acceptance_test_atol) default to 0.0, which derives them from the pipeline’s floating-point precision. A mismatch means the recipe’s MOJO operations do not reproduce its Python scoring: fix the recipe rather than loosening the tolerances.

A recipe declares which runtimes its MOJO supports with the _mojo_cpp and _mojo_java attributes, both True by default. Set one to False if the recipe’s MOJO operations only work in the other runtime.