I Took Laya's "Calibrated RL" Apart. It's Cross-Entropy With Extra Steps.

Laya's README reports a raw ECE of 0.466 on the typed-decisions benchmark before temperature scaling. The method it's named for - Reinforcement Learning for Calibrated Decisions - is supposed to be what makes those probabilities mean something. So I went looking for the RL term's contribution.

I found a suspect with a very solid alibi: it was doing nothing.

Atomic answer: Laya's RL term is an evolution-strategies estimate of the gradient of a noise-smoothed proper score. As the noise scale goes to zero it converges to the cross-entropy gradient the model already computes. At practical noise levels it makes the model over-sharp at inference, and across 30 training runs it never improved a single accuracy number over plain cross-entropy.

TL;DR

• Laya's RL reward is a score-function / evolution-strategies estimator of ∇ E_ε[R(softmax(z+ε))]. For the log score, that gradient → t − softmax(z) as σ → 0 -which is exactly the CE gradient.

• The noise-smoothed objective's optimum satisfies E_ε[softmax(z*+ε)] = t. Inference reports the noise-free softmax(z*), which is provably sharper than the target (I prove it for binary questions).

• On the real benchmark, fitted temperature rises monotonically with the training noise scale: 1.14 → 2.42 as σ goes 0.25 → 4. Accuracy does not move.

• Across CE-only, RL+CE, RL-only, a σ sweep, a reward-weight sweep, and a reward-composition sweep, CE-only is at least as good on every metric I track. The only thing that raised accuracy was doubling the input token budget.

This is the short version. The deep version - the estimator derivation, the proof, and the per-run tables - is in the companion post: Inside RLCD: The Estimator, the Proof, and the 30 Runs That Changed My Mind.

First, what Jev and Laya actually are

TypeSafe's Jev is a "System One" model: you hand it unstructured state, it returns a probability distribution over a closed, typed set of options - a choice among named options, an ordinal score, or a binary noul (yes/no) probability. No text generation. The headline claim is calibration: an answer given with 0.8 confidence should be right about 80% of the time.

The method credited for that, RLCD, has no published paper, no reward function, no dataset, and no calibration figure. That's the whole reason this project exists - I wrote about Jev from the outside here: Jev, Explained: The AI That Refuses to Write a Word.

Laya is the only open reproduction of the same interface. It publishes code, checkpoints, and an evaluation on LocalLLaMA/typed-decisions: 400 test cases, 2,000 decisions, four workflows. Its fine-tuned checkpoint reports 0.766 accuracy, Brier 0.062, ECE 0.213. Against Jev's published 0.727. That's the target I reproduced and then took apart.

The method, in one paragraph

Laya encodes each question as its own row - [CLS] <type> question: <instr> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] <state> [SEP] - runs a ModernBERT-large encoder, a small transformer head, and reads one logit out of each option's [MASK] position. Training adds noise to the logits, scores the noisy distributions with a proper scoring rule (log score + spherical + RPS), and applies a REINFORCE-style update next to a soft cross-entropy term on the teacher's probability distribution.

Here's the full training step:

z  ← model(x)                       # one logit per option
for g in 1..G:                       # G = 4 noise samples
    ε_g ~ N(0, σ²), zero-mean over options
    q_g = softmax(z + ε_g)
    r_g = R(q_g, t)                  # proper score vs the teacher target t
adv = (r − mean(r)) / std(r)
L_rl = −mean(adv · log N(z+ε | z, σ²))    # score-function estimator
L_ce = −Σ t · log softmax(z)              # soft cross-entropy
loss = w_rl · L_rl + w_ce · L_ce

σ anneals from 0.4 to 0.1 over four epochs. After training, one temperature per question type is fitted on a held-out slice.

The finding: the RL term is a smoothed cross-entropy gradient

Write J_σ(z) = E_ε[R(softmax(z+ε), t)]. The score-function identity gives

∇_z J_σ(z) = (1/σ²) · E[ R(softmax(z+ε), t) · ε ]

which Laya estimates with four samples and a group-mean baseline. For the log score, ∇R = t − softmax(z). So as σ → 0, the RL estimator's expected value is t − softmax(z) — the exact negative CE gradient. Laya is paying for a Monte Carlo estimate of a gradient that the CE term next to it computes exactly.

That alone would be a curiosity. The problem is the noise: it doesn't vanish, it biases the optimum.

The over-confidence is real, and it grows with the noise

The smoothed objective's stationary point satisfies E_ε[softmax(z*+ε)] = t. Averaging a softmax over logit noise flattens it. So the model's noise-averaged prediction matches the target, while the noise-free prediction that inference actually reports is sharper than the target. Over-confident by construction.

I checked this two ways. The toy first: minimise the smoothed log score for t = [0.7, 0.2, 0.1] and look at both predictions.

The averaged column tracks the target. The noise-free column is the one inference ships, and at σ=2 it puts 0.905 on a class the teacher gave 0.70.

Then the real thing. Train with fixed noise and fit the temperature:

Fitted temperature and NLL climb monotonically with σ. Accuracy is flat. The temperature is the model's own tell that its raw outputs are too sharp - and it rises exactly as the theory predicts.

30 runs later: the loss doesn't matter, the input budget does

I ran the full ablation. Reproduced Laya: 0.773 accuracy, Brier 0.054. CE-only: 0.782, Brier 0.052. RL-only: 0.769. Then a reward-weight sweep and a reward-composition sweep. Nothing beat CE-only.

The only thing that moved accuracy was matching the token budget the checkpoint card documents (1024 context / 256 option tokens), which my port had under-provisioned at 512/192. That's the option-token starvation Laya warns about: many options share a fixed budget, so a large option set loses tokens per option.

What this means if you're building typed-decision models

• If your targets come from a data engine you control, cross-entropy on those distributions is already a proper-score optimum. You don't need RL for it.

• Logit-noise smoothing is not a free regulariser. It has a direction, and the direction is over-confidence. If you use it, temperature scaling is undoing your training, not just cleaning up after it.

• Report the fitted temperature. It's a cheap, honest signal that an upstream stage distorted the distribution.

• Check your option-token budget before blaming the loss. It was the only accuracy lever I found.

How to reproduce

Everything is public. The code, configs, and result files are in LakoreAI/sev. Each run is a single A100/A5000 fine-tune of Laya's own checkpoint.

If you build decision models for a living, I'd like to hear whether the fitted temperature signal matches what you see in production. It's the cheapest diagnostic in this whole post.

Follow me: LinkedIn | GitHub