Ultramarine 1 Flash: A Model Built for Office Work, at 13 Cents a Task

Most AI agents get trained on code. That makes sense, because code is easy to check. Run the tests and you know if it worked.

A diligence memo has no test suite. Neither does a board deck or a financial model. Whether the work is good comes down to judgment, and judgment is hard to turn into a reward.

Paper Instruments went after that gap with Ultramarine 1 Flash, a model trained for finance, law, consulting and operations work. I read the product page and the full technical report. Here's what it is, what the numbers say, and the parts I found most interesting.

The short version

• It's GLM-5.3 Flash, an open base model, plus a lot of reinforcement learning on professional deliverables like Excel models, decks and Word memos.

• On Paper Instruments' own knowledge-work benchmark it scores 59.3 at $0.13 per task. Claude Fable 5 scores 57.8 at $8.19.

• It beats its base model on all six knowledge-work benchmarks, by 4.1 points on average, while writing 6.5% fewer tokens.

• It is first on three of the six, loses to Claude Fable 5 on two, and is slightly weaker than its base model on coding.

• It's available through Paper Instruments' API. The weights are listed as "coming soon".

What Ultramarine 1 Flash actually is

Take GLM-5.3 Flash, an open model from Z.ai, and train it with large-scale reinforcement learning on realistic professional tasks: build a model in Excel, update a deck, edit a Word document in place, or work inside a live business system like an ERP. That's Ultramarine 1 Flash.

API pricing matches the base model's: $0.15 per million input tokens, $0.03 for cached input and $0.50 for output. So you get a better model at the same price per token, and it writes fewer tokens to get there: 30.5M output tokens across the six benchmarks, against 32.7M for the base model. The report calls that a 6.5% drop. (The totals are rounded, so dividing them gives a slightly different percentage.)

A note on "open": Paper Instruments calls this an open model, but the product page currently says the weights are coming soon. A GitHub repo called feather-train is linked.

The benchmark results

The report tests Ultramarine on six knowledge-work benchmarks and two coding benchmarks. Every model runs at the highest reasoning setting it supports.

On coding it scores 83.2 on Terminal-Bench 2.1 and 61.5 on DeepSWE 1.1. Both are a little below the base model's 84.3 and 63.4.

Ultramarine is first on JobBench, PI Knowledge Work and FrontierFinance, and second on AutomationBench, just behind DeepSeek V4.1 Flash. It beats Claude Fable 5 on four of the six and loses to it on DiligenceBench and APEX-Agents, where the gap to the leader is large (57.0 against 64.3).

DiligenceBench is Paper Instruments' own benchmark, and the model doesn't win it. I read that as a good sign for the report's honesty. The coding dip fits a model trained for office work.

Worth knowing: The comparison scores come from different places. Some were run by Paper Instruments in the same harness, and others are leaderboard or developer-reported numbers. The report's evaluation notes say which is which.

The cost story

On PI Knowledge Work, Ultramarine scores 59.3 at $0.13 per task. The page says that's about 60 times cheaper than Fable.

Each dot is one model at one reasoning setting, using the data in Paper Instruments' report. The dashed line marks runs that no other run beats on both cost and score.

The main models in numbers, all at max reasoning:

A few honest points:

• It isn't the top scorer. In the report's chart data, seven runs score higher than Ultramarine, and every one costs at least $1.22 per task. The claim is about value. It trails Claude Opus 5.5 by 2.6 points and GPT-6 Sol by about 2.

• A close rival exists. GPT-6 Luna scores 58.3 at $0.11 per task.

• It costs a bit more than its base here. $0.13 against $0.10 per task. The 6.5% token saving is measured across all six benchmarks, not just this one.

• The value claim holds up in the data. Of the 39 other runs, spread across eighteen models and their reasoning settings, 27 cost more and score lower.

How much to trust this benchmark: PI Knowledge Work is Paper Instruments' internal suite, graded by GPT-5.6 Terra against each task's rubric. As a sanity check, it correlates strongly (r = 0.81) with the Artificial Analysis Intelligence Index across the 24 runs both measure. That's reasonable, but it's still the makers' own yardstick.

The real story: training on work with no answer key

Verifiable tasks are easy to reward: the code passes or it doesn't. Professional work has no such check. No regex tells you whether a recommendation is sound. Here's how Paper Instruments handled it.

A rubric is a fence, not a script

Each deliverable is graded against a rubric written for that task. Think of each criterion as a fence around the acceptable answers. Too tight, and it's copied from one golden answer and rejects other valid ones. Too loose ("gives a useful update for the board"), and plausible but wrong answers slip through. Just right states the real condition for credit and names the facts, dates and thresholds needed to check it.

Each criterion tests one idea. An "and" is allowed when things must hold together, and an "or" when there are truly equivalent ways to pass. Independent achievements get separate criteria, so one miss doesn't wipe out credit for another.

In the report's board-note example, a good answer separates what the board can approve now from what still needs lawyers to clear. Three differently worded answers pass. Answers that give only the approval, only the legal caveat, or only the valuation multiples don't.

Why synthetic data?

The team looked at three sources of training tasks.

The report's example of the first problem: a user asks for the "top 10" medical technology companies and may have meant top 10 by revenue. A reward built from that request would punish good answers that ranked by market cap. So they went fully synthetic and put the effort into quality control.

88 environments, four domains

The curriculum covers four domains, 19 practice areas and 88 environment types.

Each practice area samples across many axes, such as deliverable format, stakeholders, conflicts between sources and the stage of the deal. The nominal configuration space is roughly 10⁷ to 10¹⁰ combinations per practice area, which is how they fight mode collapse, where synthetic data turns repetitive.

An environment is a workspace the agent lives in. It holds realistic documents (models, memos, filings, correspondence) built from real SEC filings, public web material, research snapshots and synthetic documents. It can also include live systems the agent can read and change, like CRM, ERP, warehouse or ticketing tools.

Each task takes real compute: several dozen independent agent loops and multiple rounds of review, with filters for realism, difficulty, answer leaks and reward hacking. Grading is simple. The agent's Excel, PowerPoint or Word file is parsed and rendered as text, and a plain LLM grader scores it criterion by criterion, using Paper Instruments' open-source rubric package.

flowchart LR
    A[Pick a practice area<br/>and sample taxonomy axes] --> B[Build the environment<br/>files and live systems]
    B --> C[Generate task and rubric<br/>dozens of agent loops, repeated review]
    C --> D[Filters<br/>realism, difficulty, leaks, reward hacking]
    D --> E[Agent does the work<br/>in a sandbox]
    E --> F[Deliverable<br/>xlsx, pptx or docx]
    F --> G[LLM grader checks<br/>each rubric criterion]
    G --> H[Reward feeds the RL update]

One task, up close

The report includes an edited example that shows what "professional judgment" means in a rubric. In Project Aurora, a board meets at 4:30 p.m. on April 17 while investor education is underway. The agent must update a Word pre-read in place, changing only the forecast-and-disclosure section and one director decision callout.

The catch: management approved a new forecast (v7) on April 16, but the projection table in the circulating proxy draft is still the older v5. The agent has to notice, use v7 for board and valuation work, and flag that the proxy still needs to be conformed.

The rubric rewards getting each v7 figure right (+10 each), computing the new multiples (+18 each) and separating what the board can approve now from what still needs counsel clearance (+18). It penalizes stale v5 numbers presented as current (−25 each), silently picking a version (−25), rewriting sections outside the assignment (−25) and leaving internal notes in a board-facing document (−10).

The biggest penalty, −50, goes to telling management they can use the new figures in investor education without counsel clearance. That isn't an arithmetic slip. It's a judgment error about who is allowed to say what, and when.

flowchart LR
    V7[Management forecast v7<br/>approved April 16] --> B[Board can approve now<br/>v7 as the sole forecast basis<br/>and the qualitative roadshow position]
    V7 --> C[Needs counsel clearance first<br/>exact external wording and<br/>next-turn projection tables]

Feather: the harness the model learned in

Training happens inside Feather, a Python agent harness Paper Instruments built from the conventions shared by popular coding harnesses such as Claude Code, Codex, OpenCode and OpenHands. The goal was a model that transfers to other harnesses. Their internal evaluations run in Feather, while the public benchmarks ran in others (OpenCode for JobBench, Claude Code for Terminal-Bench, Mercor's Loop agent for APEX-Agents), so those results partly test that goal.

The tool set is small: read, edit, write, grep and glob for files, bash with explicit background tasks, and a to-do tool whose list survives context compaction. One choice is worth copying: edits need a fresh read. Feather records a fingerprint (a hash) of a file when the model reads it, and rejects an edit if the file has changed since, so a background process can't have its changes overwritten.

A system prompt of about 50 tokens

This is my favorite detail. During training the system prompt is three sentences, around 50 tokens: you're an agent in a Linux sandbox, don't leak secrets, put deliverables where the task says. The production prompts the team surveyed run from roughly 1,800 to 4,400 tokens.

Tool descriptions follow the same idea: what a tool does and guarantees, not when or how to use it. The reasoning is that detailed instructions can lift test-time performance while boxing in exploration during training, so production-style instructions only come in during the final stages.

Two more details:

• No vision. The model wasn't trained with images. If it tries to read an image file, the tool simply replies that vision isn't available.

• Safer Office editing. Agents often corrupt Word, Excel and PowerPoint files when they drop into raw XML. Ultramarine trained on Paper Instruments' own forks of the usual libraries (paper-docx, paper-pptx, paper-xlsx), which validate supported changes and refuse unsafe ones.

The plumbing and the recipe

Each rollout runs in its own isolated E2B sandbox. The agent loop runs outside it, so nothing the model does inside can touch its own harness, credentials or grader. Packing a 50-file upload into one tarball cut it from 52 seconds to 2.2 seconds, a 96% cut.

Failures follow one rule: only count what the model caused. A bad tool call goes back to the model and the rollout continues. If the environment breaks, such as a lost sandbox, the attempt is thrown out and restarted, not scored as a zero.

The RL algorithm is a modified CISPO with asynchronous updates, trained with LoRA rank 64, batch size 64 × 8, a 131,072-token context and up to 250 agent turns. A length penalty cuts reward for excess output, too many turns and malformed or nonexistent tool calls. That probably explains the 6.5% token saving, though the report doesn't say so directly.

My take

What's convincing. The training design is careful, and the report is open about trade-offs. Gains over the base model show up on all six knowledge-work benchmarks, and the price-to-score ratio is strong.

What I'd want to check.

• Two of the benchmarks (PI Knowledge Work and DiligenceBench) belong to Paper Instruments, and some comparison scores come from other sources.

• Several benchmarks are graded by LLM judges, not human experts. On Paper Instruments' own suite the judge sees a text rendering of each file, so I'd treat claims about layout and visual polish as unproven until you look at real slides yourself.

• The model was trained without vision, so documents that depend on reading images or charts may be a weak spot.

• Several pricier models still score higher, coding is a step down from the base model, and the weights aren't released yet.

How I'd test it. Don't trust any leaderboard, including this one. Pick 20 real tasks from your own work, write a rubric for each, and run Ultramarine next to whatever you use today. Paper Instruments has open-sourced the rubric package it uses for grading, so you don't have to start from nothing.

It looks most interesting for high-volume drafting and document work where cost per task matters. If your work leans on coding or on reading images, test those first.

Links

• Product page: paperinstruments.com/ultramarine

• Technical report: Ultramarine 1 Flash: Scaling Non-Verifiable RL

• API: platform.paperinstruments.com

• Training harness repo: github.com/paper-instruments/feather-train

• Rubric package: github.com/paper-instruments/rubric

• Office libraries: Paper Office

All scores, prices and details in this post come from the Ultramarine product page and its technical report (published October 14, 2026). The cost chart is redrawn from the report's data.