Reinforcement Learning for Calibrated Decisions (RLCD) is central to decision models, yet remains surprisingly mysterious, with few resources explaining how it actually works. Understanding it is key to using decision models effectively, especially when setting automation thresholds and escalating uncertain decisions. This post explains how RLCD works, how it differs from ordinary confidence scoring, and how to adapt calibration to your own use case.

What is RLCD?

RLCD trains models to return structured decisions with probabilities that reflect how often those decisions are correct. This allows software to automate high-confidence decisions and escalate uncertain ones. Essentially, RLCD rewards the model for both choosing correctly and expressing honest uncertainty about its prediction.

Imagine a model deciding whether a support ticket is urgent. Instead of only answering “yes,” it answers “yes, with 80% probability.” An 80% prediction means that, across many comparable cases assigned roughly 80% confidence, about 80% should match the defined outcome. A classifier’s confidence only shows how strongly it prefers a label, while an RLCD-trained decision model outputs calibrated probabilities, so a prediction made with 80% confidence is right about 80% of the time. This offers a distinct advantage compared to a text classifier that produces answers with confidence scores, which reflect how strongly the model itself prefers a certain label, because it allows users to set meaningful thresholds for automating high-confidence decisions and escalating uncertain ones.

How is RLCD related to decision models?

A decision model is a system that, given a state and one or more user-defined questions, returns bounded, typed decisions and probabilities. RLCD is a training method intended to make those probabilities reflect real correctness rates rather than merely the model’s label preference.

However, just because a model has been adapted to provide bounded outputs and probability scores doesn’t imply that those probability scores have undergone calibration. Likewise, just because a model has undergone RLCD doesn’t necessarily imply that it is calibrated to your own subjective preferences. For instance, the definition of an “urgent” classification for a support ticket will vary significantly depending on factors such as a company’s SLA commitments, escalation policies, operational risk tolerance, and the business impact of a delayed response. We’ll talk more about how you can test and recalibrate those probabilities for your own operational definition through local validation later in this piece.

How does RLCD work?

RLCD is a post-training technique performed on decision models that return calibrated probabilities over a bounded set of answers. The process rewards the model for returning correct decisions as well as honest confidence in those decisions. For the purposes of this blog, we break down RLCD into eight separate steps.

1. Define the outcome

First, specify exactly what the model should predict. For a support ticket, “urgent” might mean that the ticket will breach its SLA unless someone responds within an hour. The target comes from whatever definition you supply, whether that be an observed event, an answer key, a business rule, or a human label.

2. Collect labeled examples

Build a dataset containing the state, question, allowed answers, and known outcome. Historical records can already contain outcomes that occurred after each original decision, so training does not pause and wait for new events. The state should contain only information that would have been available when the prediction was made.

3. Generate a probability distribution

Give the model the state and question, then record its complete distribution over the allowed answers:

The full distribution matters because RLCD evaluates how probability was allocated across every possible answer, not only whether the highest-scoring label was correct.

4. Score the prediction

Compare the distribution with the known outcome using a proper scoring rule such as the Brier score or log score. The Brier score is the squared gap between the predicted probability and the outcome:

Here, p is the predicted probability and y is 1 if the outcome occurred and 0 if it didn’t. For a ticket predicted 80% urgent:

Being confidently wrong costs 16 times more than being confidently right.

5. Update the policy

Use the proper score as the reward or loss that updates the model’s weights. The model is rewarded for probabilities that match outcomes and penalized when its confidence exceeds what the evidence supports. Repeating this process gradually sharpens justified predictions and softens unjustified ones.

6. Repeat across many examples

Continue predicting, scoring, and updating across the dataset. Calibration is a statistical property: it cannot be established from one prediction. Instead, among many cases assigned approximately 80% probability, roughly 80% should eventually match the defined outcome.

7. Evaluate on unseen data

Test the trained model on examples that were not used for optimization. Measure accuracy, probability calibration, and ranking separately: a model can choose the correct label without producing trustworthy probabilities, or be calibrated while doing little to distinguish positive from negative cases.

8. Validate locally before deployment

Finally, test the model against representative outcomes from the actual deployment environment. If its probability levels or action thresholds do not transfer, fit a local calibrator and choose thresholds using held-out local data. Recalibration can correct the probability scale, but it cannot repair poor ranking or a badly defined target.

Calibrating decision models for your domain

One of the many selling points of decision models is their ability to produce calibrated probabilities, as opposed to non-calibrated confidence scores, allowing users to create workflows that escalate uncertain responses to a human reviewer. Decision models, however, are not universally calibrated across domains and use cases. Probabilities calibrated on one dataset may not transfer to another organization, policy, or outcome definition. To ensure that those probabilities are meaningful for your specific use case, you have two options: local recalibration or performing your own RLCD.

The first step for both of these methods is to define what the probability given by your decision model should represent. Clearly define what event the probability predicts, when that event must occur, and how you will verify whether it occurred. For example, if one possible label for a support ticket is “urgent,” give it a precise definition, such as “the likelihood this ticket breaches its SLA within one hour.”

Once you’ve clearly defined what probabilities should represent according to your use case, you’re ready to choose between pursuing local calibration and performing RLCD from scratch.

Local recalibration

Local recalibration adjusts an existing model’s probability scores to match outcomes in your domain without retraining the model itself. It is typically the fastest and least expensive option, which also makes it a great first option.

Choose local recalibration when:

• The model predicts the correct decisions but its probabilities are over- or underconfident

• You are unable to access or modify the model’s underlying weights

• You have a representative set of local examples with known outcomes

• Your use case is narrow and its target definition is relatively stable

• Incorrect confidence is manageable through human review or other safeguards

It’s important to note that recalibration cannot fix poor predictions, missing context, or invalid targets.

Performing RLCD from scratch

Training with RLCD adapts the model itself to produce probability distributions that better reflect observed correctness. This requires open weights, reliable outcome data, greater compute, and substantially more engineering.

Consider RLCD when:

• Calibrated probabilities are a core product capability rather than an optional signal

• Confidently wrong decisions carry significant operational or financial costs

• You control the model weights and training pipeline

• You have enough reliable outcome data to train and evaluate calibration

RLCD does not eliminate the need for local validation. An RLCD-trained model may still require recalibration for a specific organization or deployment environment.

Conclusion

RLCD is the post-training technique that enables decision models to return probabilities that correspond to how often their decisions are correct in practice. It trains decision models to report probability distributions and rewards them according to how well those probabilities predict observed outcomes, allowing uncertain predictions to be routed to human review. To ensure that outcome probabilities produced by decision models are accurate for your use cases, you can either perform local calibration or post-train a decision model with RLCD.

To learn more about decision models, including our own GLiDE and GLiNER2.5-Decide, check out this blog post. And to try our decision models on your own data, check out our inference API.