TypeSafe's Jev returns calibrated probabilities over a closed set of options and credits "Reinforcement Learning for Calibrated Decisions" for it. No paper, no reward function, no dataset, no calibration figure. Laya is the only open reproduction of the same interface, so I reproduced it, then took its training loop apart.
Atomic answer: Laya's RLCD RL term is an evolution-strategies estimate of the gradient of a noise-smoothed proper scoring rule. As the noise vanishes it equals the cross-entropy gradient the model already computes. At real noise levels it makes inference over-confident, and across 30 runs it never beat plain cross-entropy on any accuracy or proper-score number. The only lever that raised accuracy was the input token budget.
This is the short introduction to the paper. The full report, code, and result files are linked at the bottom.
💡 TL;DR & Key Takeaways:
TL;DR
Reproducing Laya’s RLCD shows that its “RL” term is just a noise‑smoothed estimate of the cross‑entropy gradient, which makes inference over‑confident, and that any accuracy gains come from increasing the option‑token budget rather than from the RL loss itself.Key points
The RL gradient converges to the cross‑entropy gradient as noise vanishes, so it adds no new information beyond CE.
Logit‑noise smoothing causes the trained model to output sharper predictions than the target distribution, leading to over‑confidence at inference time.
Across multiple ablations, the only factor that improves proper‑score metrics is the input token budget; the RL loss and its hyperparameters have negligible impact.
What the paper is about
Two models. Jev, from TypeSafe, is the closed model: unstructured state in, typed probabilistic decisions out - a choice among named options, an ordinal score, or a binary noul. It is marketed on calibration: a 0.8 answer should be right about 80% of the time. Laya is the open reproduction: same interface, published code, checkpoints, and a benchmark (LocalLLaMA/typed-decisions, 400 cases, 2,000 decisions).
Laya's loop adds noise to the logits, scores the noisy distributions with a proper scoring rule, and applies a REINFORCE-style update next to a soft cross-entropy term. That is the "RL" in RLCD. I reproduced it, verified it tensor-for-tensor against the released weights, and then asked one question: what does that RL term actually contribute?
Three things the paper shows
1. The RL term is a noisy estimate of the cross-entropy gradient. Write J_σ(z) = E_ε[R(softmax(z+ε), t)]. The score-function identity plus Stein's lemma turns the RL gradient into the CE gradient averaged over logit noise. For the log score, as σ → 0 it converges to t − softmax(z) - which the CE term next to it already gives exactly.
2. The smoothing makes inference over-confident, provably. The smoothed optimum satisfies E_ε[softmax(z*+ε)] = t, so the noise-averaged prediction matches the teacher. Inference runs at ε=0 and reports the noise-free softmax(z*), which averaging has flattened, so it is sharper than the target. I verified it on trained models: at σ=2, the noise-averaged prediction lands within −0.006 of the target while the noise-free one is over-sharp by +0.108. The fitted temperature rises monotonically from 1.14 to 2.42 as the noise scale goes 0.25 → 4.
3. The loss is not the lever; the input budget is. Across CE-only, RL+CE, RL-only, a σ sweep, a reward-weight sweep and a reward-composition sweep, CE-only is at least as good on every proper score. The only change that raised accuracy was matching the checkpoint's documented 1024-token context and 256-token option budget.
model | accuracy | Brier |
|---|---|---|
CE-only (512/192) | 0.782 | 0.052 |
RL+CE (Laya recipe) | 0.773 | 0.054 |
CE-only, 1024/256 | 0.789 | 0.0495 |
Laya | 0.766 | 0.062 |
The result I did not expect
The budget effect is small on typed decisions. On a high-cardinality task it is not. I recast four standard classification sets into typed choice questions and scored base Laya and my model zero-shot at two option-token budgets:
task | options | Laya 256 | Laya 512 | Sev 256 | Sev 512 |
|---|---|---|---|---|---|
AG News | 4 | 94.5 | 94.5 | 94.3 | 94.3 |
Emotion | 6 | 59.8 | 59.8 | 60.2 | 60.2 |
SST-5 | 5 | 34.3 | 34.3 | 42.9 | 42.9 |
Banking77 | 77 | 15.8 | 31.2 | 15.8 | 32.0 |
AG News and Emotion reproduce Laya's own published numbers (95.0 and 59.5). The budget does nothing when every option already fits. At 77 options it roughly doubles accuracy, for both models. That is the option-token starvation Laya's card warns about, and it dwarfs every loss knob I tried.
Why you might care
If you build typed-decision models in production, three things generalise past this benchmark:
Cross-entropy against good target distributions is already a proper-score optimum. You do not need RL for it.
Logit-noise smoothing is not a free regulariser; it has a direction, and the direction is over-confidence. Fitting a temperature afterwards is undoing the training, not cleaning up after it.
Check your option-token budget before you touch the loss. It was the only thing that moved a number.
Read it, run it
Paper (IEEE format, PDF + LaTeX): https://github.com/LakoreAI/sev
Code, configs, every per-run result file: same repo
Best checkpoint (weights + fitted temperatures): https://huggingface.co/LakoreAI/laya-typed-decisions-ce-1024
All ablation checkpoints and metrics: https://huggingface.co/minhleduc/rlcd-e2-checkpoints
This builds on the framing I wrote earlier in Jev, Explained: The AI That Refuses to Write a Word.
If you have fitted a temperature on your own classifier and it came out above 1, I would like to know whether the cause was the loss, the data, or the input budget.
Comments (0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!