--- title: "I Took Laya's \"Calibrated RL\" Apart. It's Cross-Entropy With Extra Steps." author: "Lê Đức Minh" published: "2026-09-26T14:39:10.209Z" updated: "2026-09-26T14:39:10.209Z" url: "https://zyvop.com/i-took-laya-s-calibrated-rl-apart-it-s-cross-entropy-with-extra-steps-vtl6r" canonical: "https://zyvop.com/i-took-laya-s-calibrated-rl-apart-it-s-cross-entropy-with-extra-steps-vtl6r" tags: ["AI","typed decisions","RLCD","calibration","TypeSafe","Laya","Jev","AI agents"] --- # I Took Laya's "Calibrated RL" Apart. It's Cross-Entropy With Extra Steps. ![Cover image]() Laya's README reports a raw ECE of 0.466 on the typed-decisions benchmark before temperature scaling. The method it's named for - Reinforcement Learning for Calibrated Decisions - is supposed to be what makes those probabilities mean something. So I went looking for the RL term's contribution. I found a suspect with a very solid alibi: it was doing nothing. **Atomic answer:** Laya's RL term is an evolution-strategies estimate of the gradient of a _noise-smoothed_ proper score. As the noise scale goes to zero it converges to the cross-entropy gradient the model already computes. At practical noise levels it makes the model over-sharp at inference, and across 30 training runs it never improved a single accuracy number over plain cross-entropy. > ## TL;DR > > - Laya's RL reward is a score-function / evolution-strategies estimator of `∇ E_ε[R(softmax(z+ε))]`. For the log score, that gradient `→ t − softmax(z)` as `σ → 0` -which is exactly the CE gradient. > > - The noise-smoothed objective's optimum satisfies `E_ε[softmax(z*+ε)] = t`. Inference reports the _noise-free_ `softmax(z*)`, which is provably sharper than the target (I prove it for binary questions). > > - On the real benchmark, fitted temperature rises monotonically with the training noise scale: `1.14 → 2.42` as `σ` goes `0.25 → 4`. Accuracy does not move. > > - Across CE-only, RL+CE, RL-only, a σ sweep, a reward-weight sweep, and a reward-composition sweep, **CE-only is at least as good on every metric I track.** The only thing that raised accuracy was doubling the input token budget. > This is the short version. The deep version - the estimator derivation, the proof, and the per-run tables - is in the companion post: [Inside RLCD: The Estimator, the Proof, and the 30 Runs That Changed My Mind.]() ## First, what Jev and Laya actually are TypeSafe's Jev is a "System One" model: you hand it unstructured state, it returns a probability distribution over a closed, typed set of options - a `choice` among named options, an ordinal `score`, or a binary `noul` (yes/no) probability. No text generation. The headline claim is calibration: an answer given with 0.8 confidence should be right about 80% of the time. The method credited for that, RLCD, has no published paper, no reward function, no dataset, and no calibration figure. That's the whole reason this project exists - I wrote about Jev from the outside here: [Jev, Explained: The AI That Refuses to Write a Word](). Laya is the only open reproduction of the same interface. It publishes code, checkpoints, and an evaluation on `LocalLLaMA/typed-decisions`: 400 test cases, 2,000 decisions, four workflows. Its fine-tuned checkpoint reports 0.766 accuracy, Brier 0.062, ECE 0.213. Against Jev's published 0.727. That's the target I reproduced and then took apart. ## The method, in one paragraph Laya encodes each question as its own row - `[CLS] question: [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] [SEP]` - runs a ModernBERT-large encoder, a small transformer head, and reads one logit out of each option's `[MASK]` position. Training adds noise to the logits, scores the noisy distributions with a proper scoring rule (log score + spherical + RPS), and applies a REINFORCE-style update next to a soft cross-entropy term on the teacher's probability distribution. Here's the full training step: ```javascript z ← model(x) # one logit per option for g in 1..G: # G = 4 noise samples ε_g ~ N(0, σ²), zero-mean over options q_g = softmax(z + ε_g) r_g = R(q_g, t) # proper score vs the teacher target t adv = (r − mean(r)) / std(r) L_rl = −mean(adv · log N(z+ε | z, σ²)) # score-function estimator L_ce = −Σ t · log softmax(z) # soft cross-entropy loss = w_rl · L_rl + w_ce · L_ce ``` `σ` anneals from 0.4 to 0.1 over four epochs. After training, one temperature per question type is fitted on a held-out slice. ## The finding: the RL term is a smoothed cross-entropy gradient Write `J_σ(z) = E_ε[R(softmax(z+ε), t)]`. The score-function identity gives ``` ∇_z J_σ(z) = (1/σ²) · E[ R(softmax(z+ε), t) · ε ] ``` which Laya estimates with four samples and a group-mean baseline. For the log score, `∇R = t − softmax(z)`. So as `σ → 0`, the RL estimator's expected value is `t − softmax(z)` — the exact negative CE gradient. Laya is paying for a Monte Carlo estimate of a gradient that the CE term next to it computes exactly. That alone would be a curiosity. The problem is the noise: it doesn't vanish, it biases the optimum. ## The over-confidence is real, and it grows with the noise The smoothed objective's stationary point satisfies `E_ε[softmax(z*+ε)] = t`. Averaging a softmax over logit noise flattens it. So the model's _noise-averaged_ prediction matches the target, while the _noise-free_ prediction that inference actually reports is sharper than the target. Over-confident by construction. I checked this two ways. The toy first: minimise the smoothed log score for `t = [0.7, 0.2, 0.1]` and look at both predictions. | σ | noise-free `softmax(z*)` | noise-averaged `E[softmax(z*+ε)]` | target | | --- | --- | --- | --- | | 0.5 | \[0.723, 0.187, 0.090\] | \[0.698, 0.202, 0.100\] | \[0.7, 0.2, 0.1\] | | 1.0 | \[0.780, 0.154, 0.066\] | \[0.696, 0.205, 0.099\] | \[0.7, 0.2, 0.1\] | | 2.0 | \[0.905, 0.071, 0.024\] | \[0.706, 0.196, 0.098\] | \[0.7, 0.2, 0.1\] | The averaged column tracks the target. The noise-free column is the one inference ships, and at σ=2 it puts 0.905 on a class the teacher gave 0.70. Then the real thing. Train with fixed noise and fit the temperature: | σ | fitted T (noul) | NLL vs targets | accuracy | | --- | --- | --- | --- | | 0.25 | 1.14 | 0.866 | 0.777 | | 0.5 | 1.16 | 0.871 | 0.772 | | 1 | 1.38 | 0.890 | 0.764 | | 2 | 1.72 | 0.928 | 0.769 | | 3 | 2.08 | 0.956 | 0.775 | | 4 | 2.42 | 0.987 | 0.774 | Fitted temperature and NLL climb monotonically with σ. Accuracy is flat. The temperature is the model's own tell that its raw outputs are too sharp - and it rises exactly as the theory predicts. ## 30 runs later: the loss doesn't matter, the input budget does I ran the full ablation. Reproduced Laya: 0.773 accuracy, Brier 0.054. CE-only: 0.782, Brier 0.052. RL-only: 0.769. Then a reward-weight sweep and a reward-composition sweep. Nothing beat CE-only. | configuration | accuracy | Brier | NLL | | --- | --- | --- | --- | | CE-only | 0.782 | 0.052 | 0.861 | | RL+CE (Laya) | 0.773 | 0.054 | 0.866 | | RL-only | 0.769 | 0.054 | 0.866 | | **CE-only, 1024/256 tokens** | **0.789** | **0.0495** | **0.858** | The only thing that moved accuracy was matching the token budget the checkpoint card documents (1024 context / 256 option tokens), which my port had under-provisioned at 512/192. That's the option-token starvation Laya warns about: many options share a fixed budget, so a large option set loses tokens per option. ## What this means if you're building typed-decision models - If your targets come from a data engine you control, cross-entropy on those distributions is already a proper-score optimum. You don't need RL for it. - Logit-noise smoothing is not a free regulariser. It has a direction, and the direction is over-confidence. If you use it, temperature scaling is undoing your training, not just cleaning up after it. - Report the fitted temperature. It's a cheap, honest signal that an upstream stage distorted the distribution. - Check your option-token budget before blaming the loss. It was the only accuracy lever I found. ## How to reproduce Everything is public. The code, configs, and result files are in [LakoreAI/sev.]() Each run is a single A100/A5000 fine-tune of Laya's own checkpoint. If you build decision models for a living, I'd like to hear whether the fitted temperature signal matches what you see in production. It's the cheapest diagnostic in this whole post. Follow me: [**LinkedIn**]() | [**GitHub**]()