Reinforcement learning

Overview

Reinforcement learning (RL) is a training approach that fine-tunes a model by generating its own responses and scoring them with a reward, rather than training against reference answers. In Enterprise LLM Studio, it's available as the Causal LM with Reinforcement Learning problem type. This makes RL useful when you do not have labeled outputs but can define a way to judge whether a response is good.

During RL training, the model performs a full rollout for each input: it generates a completely new response, that response is scored by a reward function, and the model is updated to make higher-scoring responses more likely. Because the model is not compared against a fixed target answer, your training data does not need an output column.

Enterprise LLM Studio implements RL using GRPO (Group Relative Policy Optimization), with an LLM-as-a-Judge reward that scores each response on a scale of 1–10.

When to use RL

Reinforcement learning is the right choice when you do not have labeled outputs but you can score a response with a reward function (for example, an LLM-as-a-Judge rubric). If you already have high-quality labeled outputs, standard supervised fine-tuning (Causal LM) is faster and more efficient — see Supervised fine-tuning vs. reinforcement learning.

How reinforcement learning works

Standard supervised fine-tuning computes a loss on every token of a known target answer, so each sample carries a lot of learning signal in a single forward pass. RL is different:

• For each prompt, the model generates one or more complete responses (rollouts).
• Each response is scored by the reward function, producing a single reward number for the whole response.
• The model is updated to increase the likelihood of higher-reward responses, while a KL-divergence penalty keeps it from drifting too far from the original model.

Because RL generates full responses token by token instead of doing a single forward pass, it is considerably slower than supervised fine-tuning, and the reward signal (one number per response) is noisier than per-token supervision. Expect RL runs to take longer and to benefit from more data and more epochs. This is inherent to the method, not a configuration problem — no special tuning is required beyond allowing enough time and samples.

Supervised fine-tuning vs. reinforcement learning

	Supervised fine-tuning (Causal LM)	Reinforcement learning
Requires labeled outputs	Yes	No
Learning signal	Per-token loss against the target answer	Single reward score per response
Speed	Fast (single forward pass per sample)	Slow (full rollout / generation per sample)
Typical epochs	1	2–3 (oversampling helps compensate for the weaker signal)
Best when	You have high-quality labels	You have no labels but can score responses

For most projects, teams use one approach or the other. If you do have labels, plain supervised fine-tuning is more efficient and is usually the better first choice.

Combining supervised fine-tuning and RL

A common advanced workflow is to fine-tune a model with supervised learning first, then apply reinforcement learning on top of the result. This is how many production LLMs are trained: a base model is fine-tuned using supervised learning on labeled data to get the style and behavior right, and RL is applied as a final step to optimize against a reward you can validate.

Starting from a supervised-fine-tuned model gives RL a much better starting point, especially for harder tasks where a model that begins with no task knowledge would train for a long time without reaching good scores.

Continuing training from a fine-tuned model

First-class support for continuing RL training directly from a previous experiment's checkpoint is on the roadmap. If you want to run a supervised-fine-tuning → RL pipeline today, contact your H2O.ai representative for the current recommended workflow.

Prepare your data

RL uses the Causal LM with Reinforcement Learning problem type (text_rl_causal_language_modeling).

Your dataset needs:

• Input column (required): the prompt or instruction for the model.
• System prompt column (optional): a system prompt that is prepended to each input to steer the model toward the desired behavior.

An output column is not required and is ignored during RL training, because the model generates and scores its own responses rather than comparing against a reference answer. If your dataset includes an output column, it has no effect on RL.

Use a system prompt for anything non-trivial

A system prompt column makes it much easier to steer the model toward the right output — for example, You are an expert summarizer who writes concise, factual summaries. Simple tasks such as summarization may train well without one, but for more complex tasks a system prompt (or a supervised fine-tuning step beforehand) is strongly recommended so the model understands the task and your reward function produces a meaningful signal.

YAML key: dataset.system_column | Default: null

Changing the problem type of an uploaded dataset

If you uploaded a dataset as Causal LM and want to use it for reinforcement learning, you can change its problem type after upload. This applies to your own uploaded datasets; demo datasets are already configured correctly.

You can do this in two ways:

From the dataset's detail page:

1. Open the dataset

   Open the dataset.

2. Edit the problem type

   Click the edit (pencil) icon next to Problem Type.

   

3. Select the RL problem type

   Select Causal LM (Reinforcement Learning) from the dropdown.

   The change saves automatically.

From the Datasets list:

1. Switch the table into edit mode

   Click the checkbox icon at the top-left of the table to switch it into edit mode.

   

2. Set the problem type on the row

   In the Problem Type column, choose Causal LM (Reinforcement Learning) for the dataset's row.

   

Configure an RL experiment

To create an RL experiment:

1. Start a new experiment

   Go to the Experiments page and click New Experiment.

2. Choose an RL train dataset

   Choose a dataset with the Causal LM (Reinforcement Learning) problem type as your Train Dataset.

   The experiment's problem type is automatically set to match.

The settings below are unique to, or behave differently for, RL experiments.

Before you start: the LLM judge model

RL scoring uses whichever model is set as the LLM-as-a-Judge Model on the Settings page — this is a global setting, not something you pick per experiment. Your administrator restricts the list of available judge models. Default: gpt-5-mini.

Metrics

For RL, LLM-as-a-Judge is the only available metric. Reference-based metrics such as BLEU, perplexity, and accuracy are not available, because there are no labeled targets to compare against.

YAML key: prediction.metrics | Value for RL: ["LLM-as-a-Judge"]

Reward (judge) prompt template

The reward is defined by the LLM-as-a-Judge prompt template. This template tells the judge what to evaluate and instructs it to return a score from 1 (worst) to 10 (best). You can edit it freely to match your use case; the default template scores responses on helpfulness, relevance, accuracy, clarity, and completeness.

The template supports these placeholders:

• {PROMPT} — the input (and system prompt, when present) sent to the model being trained.
• {PREDICTED_TEXT} — the response generated by the model being trained.

The response is wrapped with explicit start and end markers in the template. Generated responses can be messy (for example, containing many trailing newlines), and clear boundaries help the judge evaluate the response itself rather than mistaking formatting artifacts for a quality problem.

YAML key: training.llm_judge_prompt_template | Default: built-in standalone (label-free) evaluation template

The default RL template is:

[Instruction]
Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant to the user instruction displayed below. Your evaluation should consider:
- Helpfulness: Does the response address the instruction effectively?
- Relevance: Is the response on-topic and appropriate for the given instruction?
- Accuracy: Is the information in the response correct and free of hallucinations?
- Clarity: Is the response well-organized, coherent, and easy to understand?
- Completeness: Does the response adequately cover the key aspects of the instruction?

Begin your evaluation by providing a short explanation. Be as objective as possible. After providing your explanation, you must rate the response on a scale of 1 (worst) to 10 (best).

[User Instruction]
{PROMPT}

[The Start of Assistant's Response to Evaluate]
{PREDICTED_TEXT}
[The End of Assistant's Response to Evaluate]

RL-specific training parameters

These parameters are only used for reinforcement learning and can be set in the Advanced Configuration.

Number of generations

The number of completions generated per prompt during GRPO training. Higher values give a better estimate of the reward signal but use more GPU memory (VRAM).

YAML key: training.num_generations | Default: 8 | Minimum: 2

GRPO beta

The KL-divergence penalty coefficient. It controls how far the trained policy is allowed to diverge from the reference (original) model. Higher values are more conservative and keep the model closer to its starting point.

YAML key: training.grpo_beta | Default: 0.04

Epochs

For RL, it is reasonable to train for more than one epoch (for example, 2–3), because the signal from a single sample is lower than in supervised training. This is different from supervised fine-tuning, where a single epoch is usually recommended.

YAML key: training.epochs | Default: 1

Evaluate before training

When enabled, the model is scored on the validation set before training begins, giving you a baseline (epoch-0) score to compare against. This is especially useful for RL, where the starting model already produces reasonable responses and a baseline makes it clear how much training improved the reward.

YAML key: training.evaluate_before_training | Default: auto (true for RL)

Get a baseline score

RL runs this baseline evaluation by default. Combined with Evaluation epochs — a separate setting controlling how often the model is scored during the run (YAML key: training.evaluation_epochs, default 0.5, meaning twice per epoch) — you get scores at 0%, 50%, and 100% of the run. Set Evaluate before training to false if you want to skip the baseline and start training immediately.

Read the training metrics

RL experiments log the usual training and validation curves, plus two RL-specific signals:

• Reward — the LLM-as-a-Judge score for each batch during training. You want this to go up over the run.
• KL divergence — how much the model has diverged from the original model. You want to keep this low, so the model retains its existing knowledge while adjusting mainly style and behavior. GRPO governs this with the grpo_beta penalty.

You'll also see a Validation LLM-as-a-Judge curve on the same chart — this is the judge score computed on the validation set at each evaluation point, distinct from the per-batch training Reward above. It's the same score discussed in Interpret the results.

The training loss reflects both signals: it broadly moves opposite to the reward (higher reward tends to mean lower loss), with the KL-divergence penalty acting as a regularizer so the model does not drift too far.

Each training step is slow because it includes a full rollout, so RL runs progress more slowly than supervised runs.

Validation loss without labels

RL does not compute a meaningful validation loss because there are no reference answers/targets; the UI currently reports a constant validation loss (0.0). Use the validation LLM-as-a-Judge score as the primary indicator of validation performance.

Interpret the results

Because RL has no ground-truth labels, evaluating quality relies on your reward function — the LLM-as-a-Judge score is usually the closest thing to an automated quality measure. Reviewing every response by hand does not scale, so a well-designed judge template matters.

On a completed experiment you can review the validation results, including:

• The overall validation LLM-as-a-Judge score (on the 1–10 scale).
• Per-example rows showing the input, the generated completion, the score it received, and the judge's explanation.

Reviewing the highest- and lowest-scoring examples, along with the judge's explanations, is the best way to sanity-check whether the reward is capturing what you actually care about. Results can be downloaded for offline review.

Deploy an RL model

Deployment via the standard Deploy flow (see Deployments) does not currently support RL-trained experiments — only the Causal LM and Classification LM problem types can be deployed this way.

To use an RL-trained model, download the model weights from the completed experiment and serve them on your own inference server (for example, vLLM).

AutoML

AutoML supports the reinforcement learning problem type, which is useful when you want to compare different backbones or hyperparameters. Because RL has a single relevant metric (LLM-as-a-Judge), AutoML optimizes for that by running multiple experiments to improve the reward.

---

Feedback

• Submit and view feedback for this page
• Send feedback about H2O Enterprise LLM Studio to cloud-feedback@h2o.ai