{"schemaVersion":"1.0","type":"Article","types":["Article"],"slug":"i-took-laya-s-calibrated-rl-apart-it-s-cross-entropy-with-extra-steps-vtl6r","url":"https://zyvop.com/i-took-laya-s-calibrated-rl-apart-it-s-cross-entropy-with-extra-steps-vtl6r","title":"I Took Laya's \"Calibrated RL\" Apart. It's Cross-Entropy With Extra Steps.","subtitle":null,"tldr":"Laya's README reports a raw ECE of 0.466 on the typed-decisions benchmark before temperature scaling.","keywords":["AI","typed decisions","RLCD","calibration","TypeSafe","Laya","Jev","AI agents","AI Research"],"entities":["Lê Đức Minh","AI Engineer","AI","typed decisions","RLCD","calibration","TypeSafe","Laya","Jev","AI agents","AI Research","ZyVOP"],"keyTakeaways":["Laya's RL reward is a score-function / evolution-strategies estimator of ∇ E_ε[R(softmax(z+ε))]. For the log score, that gradient → t − softmax(z) as σ → 0 -which is exactly the CE gradient.","The noise-smoothed objective's optimum satisfies E_ε[softmax(z*+ε)] = t. Inference reports the noise-free softmax(z*), which is provably sharper than the target (I prove it for binary questions).","On the real benchmark, fitted temperature rises monotonically with the training noise scale: 1.14 → 2.42 as σ goes 0.25 → 4. Accuracy does not move.","Across CE-only, RL+CE, RL-only, a σ sweep, a reward-weight sweep, and a reward-composition sweep, CE-only is at least as good on every metric I track. The only thing that raised accuracy was doubling the input token budget."],"headings":["TL;DR","First, what Jev and Laya actually are","The method, in one paragraph","The finding: the RL term is a smoothed cross-entropy gradient","The over-confidence is real, and it grows with the noise","30 runs later: the loss doesn't matter, the input budget does","What this means if you're building typed-decision models","How to reproduce"],"outboundLinks":["https://medium.com/@minhle_0210/jev-explained-the-ai-that-refuses-to-write-a-word-b1788b932ae0","https://github.com/LakoreAI/sev","https://www.linkedin.com/in/minhle007/","https://github.com/MinLee0210"],"contentText":"Laya's README reports a raw ECE of 0.466 on the typed-decisions benchmark before temperature scaling. The method it's named for - Reinforcement Learning for Calibrated Decisions - is supposed to be what makes those probabilities mean something. So I went looking for the RL term's contribution. I found a suspect with a very solid alibi: it was doing nothing. Atomic answer: Laya's RL term is an evolution-strategies estimate of the gradient of a noise-smoothed proper score. As the noise scale goes to zero it converges to the cross-entropy gradient the model already computes. At practical noise levels it makes the model over-sharp at inference, and across 30 training runs it never improved a single accuracy number over plain cross-entropy. TL;DR Laya's RL reward is a score-function / evolution-strategies estimator of ∇ E_ε[R(softmax(z+ε))]. For the log score, that gradient → t − softmax(z) as σ → 0 -which is exactly the CE gradient. The noise-smoothed objective's optimum satisfies E_ε[softmax(z*+ε)] = t. Inference reports the noise-free softmax(z*), which is provably sharper than the target (I prove it for binary questions). On the real benchmark, fitted temperature rises monotonically with the training noise scale: 1.14 → 2.42 as σ goes 0.25 → 4. Accuracy does not move. Across CE-only, RL+CE, RL-only, a σ sweep, a reward-weight sweep, and a reward-composition sweep, CE-only is at least as good on every metric I track. The only thing that raised accuracy was doubling the input token budget. This is the short version. The deep version - the estimator derivation, the proof, and the per-run tables - is in the companion post: Inside RLCD: The Estimator, the Proof, and the 30 Runs That Changed My Mind. First, what Jev and Laya actually are TypeSafe's Jev is a \"System One\" model: you hand it unstructured state, it returns a probability distribution over a closed, typed set of options - a choice among named options, an ordinal score, or a binary noul (yes/no) probability. No text generation. The headline claim is calibration: an answer given with 0.8 confidence should be right about 80% of the time. The method credited for that, RLCD, has no published paper, no reward function, no dataset, and no calibration figure. That's the whole reason this project exists - I wrote about Jev from the outside here: Jev, Explained: The AI That Refuses to Write a Word. Laya is the only open reproduction of the same interface. It publishes code, checkpoints, and an evaluation on LocalLLaMA/typed-decisions: 400 test cases, 2,000 decisions, four workflows. Its fine-tuned checkpoint reports 0.766 accuracy, Brier 0.062, ECE 0.213. Against Jev's published 0.727. That's the target I reproduced and then took apart. The method, in one paragraph Laya encodes each question as its own row - [CLS] <type> question: <instr> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] <state> [SEP] - runs a ModernBERT-large encoder, a small transformer head, and reads one logit out of each option's [MASK] position. Training adds noise to the logits, scores the noisy distributions with a proper scoring rule (log score + spherical + RPS), and applies a REINFORCE-style update next to a soft cross-entropy term on the teacher's probability distribution. Here's the full training step: z ← model(x) # one logit per option for g in 1..G: # G = 4 noise samples ε_g ~ N(0, σ²), zero-mean over options q_g = softmax(z + ε_g) r_g = R(q_g, t) # proper score vs the teacher target t adv = (r − mean(r)) / std(r) L_rl = −mean(adv · log N(z+ε | z, σ²)) # score-function estimator L_ce = −Σ t · log softmax(z) # soft cross-entropy loss = w_rl · L_rl + w_ce · L_ceσ anneals from 0.4 to 0.1 over four epochs. After training, one temperature per question type is fitted on a held-out slice. The finding: the RL term is a smoothed cross-entropy gradient Write J_σ(z) = E_ε[R(softmax(z+ε), t)]. The score-function identity gives ∇_z J_σ(z) = (1/σ²) · E[ R(softmax(z+ε), t) · ε ]which Laya estimates with four samples and a group-mean baseline. For the log score, ∇R = t − softmax(z). So as σ → 0, the RL estimator's expected value is t − softmax(z) — the exact negative CE gradient. Laya is paying for a Monte Carlo estimate of a gradient that the CE term next to it computes exactly. That alone would be a curiosity. The problem is the noise: it doesn't vanish, it biases the optimum. The over-confidence is real, and it grows with the noise The smoothed objective's stationary point satisfies E_ε[softmax(z*+ε)] = t. Averaging a softmax over logit noise flattens it. So the model's noise-averaged prediction matches the target, while the noise-free prediction that inference actually reports is sharper than the target. Over-confident by construction. I checked this two ways. The toy first: minimise the smoothed log score for t = [0.7, 0.2, 0.1] and look at both predictions. σ noise-free softmax(z*) noise-averaged E[softmax(z*+ε)] target 0.5 [0.723, 0.187, 0.090] [0.698, 0.202, 0.100] [0.7, 0.2, 0.1] 1.0 [0.780, 0.154, 0.066] [0.696, 0.205, 0.099] [0.7, 0.2, 0.1] 2.0 [0.905, 0.071, 0.024] [0.706, 0.196, 0.098] [0.7, 0.2, 0.1] The averaged column tracks the target. The noise-free column is the one inference ships, and at σ=2 it puts 0.905 on a class the teacher gave 0.70. Then the real thing. Train with fixed noise and fit the temperature: σ fitted T (noul) NLL vs targets accuracy 0.25 1.14 0.866 0.777 0.5 1.16 0.871 0.772 1 1.38 0.890 0.764 2 1.72 0.928 0.769 3 2.08 0.956 0.775 4 2.42 0.987 0.774 Fitted temperature and NLL climb monotonically with σ. Accuracy is flat. The temperature is the model's own tell that its raw outputs are too sharp - and it rises exactly as the theory predicts. 30 runs later: the loss doesn't matter, the input budget does I ran the full ablation. Reproduced Laya: 0.773 accuracy, Brier 0.054. CE-only: 0.782, Brier 0.052. RL-only: 0.769. Then a reward-weight sweep and a reward-composition sweep. Nothing beat CE-only. configuration accuracy Brier NLL CE-only 0.782 0.052 0.861 RL+CE (Laya) 0.773 0.054 0.866 RL-only 0.769 0.054 0.866 CE-only, 1024/256 tokens 0.789 0.0495 0.858 The only thing that moved accuracy was matching the token budget the checkpoint card documents (1024 context / 256 option tokens), which my port had under-provisioned at 512/192. That's the option-token starvation Laya warns about: many options share a fixed budget, so a large option set loses tokens per option. What this means if you're building typed-decision models If your targets come from a data engine you control, cross-entropy on those distributions is already a proper-score optimum. You don't need RL for it. Logit-noise smoothing is not a free regulariser. It has a direction, and the direction is over-confidence. If you use it, temperature scaling is undoing your training, not just cleaning up after it. Report the fitted temperature. It's a cheap, honest signal that an upstream stage distorted the distribution. Check your option-token budget before blaming the loss. It was the only accuracy lever I found. How to reproduce Everything is public. The code, configs, and result files are in LakoreAI/sev. Each run is a single A100/A5000 fine-tune of Laya's own checkpoint. If you build decision models for a living, I'd like to hear whether the fitted temperature signal matches what you see in production. It's the cheapest diagnostic in this whole post. Follow me: LinkedIn | GitHub","contentHash":"sha256:afe673c8564c8e7c87b6bc327310f89ce6985b4f6638d257e9ba59d8ca783a0c","authorName":"Lê Đức Minh","authorUrl":"https://zyvop.com/author/l445","authorSameAs":["https://minlee0210.github.io","https://github.com/MinLee0210"],"category":"AI Research","tags":["AI","typed decisions","RLCD","calibration","TypeSafe","Laya","Jev","AI agents"],"audience":"Software engineers and developers building applications with AI Research","tone":"Professional, ai engineer perspective","readingTimeMinutes":6,"wordCount":1268,"faqs":null,"primaryTopic":"AI Research","publishedAt":"2026-09-26T14:39:10.209Z","updatedAt":"2026-09-26T14:39:10.209Z","canonicalUrl":"https://zyvop.com/i-took-laya-s-calibrated-rl-apart-it-s-cross-entropy-with-extra-steps-vtl6r"}