A 4B Open Model Matches GPT-5.6 Sol on Retrieval at 100× Lower Cost

What if one of the biggest problems with AI agents isn't that their models aren't powerful enough—but that we're using expensive general-purpose models for jobs that smaller models could learn to do?

That's the idea behind a new experiment from Neon and Castform.

According to results published by Neon, a 4-billion-parameter open model post-trained with Castform achieved retrieval accuracy comparable to GPT-5.6 Sol on their evaluated search workload while costing roughly 100× less.

That's an attention-grabbing result.

But the more interesting story isn't "small model beats big model."

It's how the model was trained to become specialized at searching.

Instead of asking a frontier model to solve every part of an agent workflow, Castform uses reinforcement-learning post-training to teach a smaller model how to search, retrieve evidence, cite relevant sources, and produce the correct answer.

That approach hints at a different future for AI infrastructure.

Let's unpack it.

RAG Is Evolving Into Agentic Search

The first generation of Retrieval-Augmented Generation was relatively straightforward.

A user asks something, the system converts that question into an embedding, retrieves similar documents, places them into the model's context window, and generates an answer.

Conceptually:

flowchart TD
    Question["User Question"] --> Embed["Embedding"]
    Embed --> Search["Vector Search"]
    Search --> Docs["Relevant Documents"]
    Docs --> LLM["LLM"]
    LLM --> Answer["Answer"]

This architecture remains useful, but modern AI agents increasingly need something more flexible.

Instead of searching once, an agent can search repeatedly.

Consider:

"Which database configuration caused our API latency increase last week, and was it fixed?"

Answering that might require the agent to:

1. Search incident reports.

2. Identify the affected service.

3. Search deployment history.

4. Find the configuration change.

5. Search subsequent deployments.

6. Determine whether the problem was resolved.

The architecture starts looking more like this:

flowchart TD
    Question["User Question"] --> Agent["AI Agent"]
    Agent --> Search["Search"]
    Search --> Results["Read Results"]
    Results --> ContextCheck{"Enough Context?"}
    
    ContextCheck -- "No" --> Agent
    ContextCheck -- "Yes" --> Answer["Final Answer"]

This is agentic retrieval.

It's considerably more powerful than a single vector-search request.

It also introduces a serious problem.

Every iteration costs money.

Frontier Models Become Expensive Inside Search Loops

One frontier-model request might be affordable.

But an agent may make several model calls while solving one user request.

Neon reports that in its setup, a typical multi-turn search using GPT-5.6 Sol takes more than 10 seconds and costs approximately $0.03 end-to-end.

That doesn't sound catastrophic until you scale it.

At:

1,000 requests     → ~$30
100,000 requests   → ~$3,000
1,000,000 requests → ~$30,000

Those figures are simple extrapolations from the reported per-request figure rather than Neon's own production estimates, but they illustrate the scaling problem.

If your agent performs retrieval millions of times, using a frontier model for every decision becomes expensive quickly.

Small open models offer another option.

They're dramatically cheaper to serve.

The problem?

Out of the box, they may not search nearly as effectively.

That's where Castform comes in.

Instead of Making the Model Bigger, Train It for Search

Castform takes a fundamentally different approach.

Rather than expecting a small model to compete with frontier models across reasoning, coding, mathematics, writing, and everything else, it uses reinforcement-learning post-training to specialize the model for a particular task.

In this experiment, that task is retrieval.

The basic RL setup requires three ingredients:

flowchart TD
    subgraph Setup["Setup Components"]
        T["Task"]
        E["Environment / Tools"]
        R["Reward Function"]
    end

    Setup --> Attempt["Model Attempts Task"]
    Attempt --> Measure["Measure Performance"]
    Measure --> Feedback["Reward / Feedback"]
    Feedback --> Improve["Improve Model"]
    Improve --> TryAgain["Try Again"]
    TryAgain -.-> Attempt

The model repeatedly attempts the task.

Successful behavior receives a better reward.

Poor behavior receives a worse reward.

Over many iterations, training pushes the model toward strategies that generate higher rewards.

For retrieval, this means teaching the model not simply to produce plausible text but to find the information required to produce the correct answer.

Your Database Becomes Training Data

There's another clever idea here.

Companies already possess enormous amounts of useful information:

• Internal documentation

• Support articles

• Wikis

• Product records

• Customer interactions

• Operational databases

Traditionally, turning that information into ML training data requires considerable cleaning and labeling.

Castform instead generates training tasks from the existing corpus.

Imagine an internal document contains:

Production database backups are retained for 30 days.
Enterprise customers can request retention up to 90 days.

A training example could be generated from it:

Question:
How long are production database backups retained?

Ground truth:
30 days by default, with retention up to 90 days
for enterprise customers when requested.

Now the system has something measurable.

Give the model access to search and ask it the question.

Did it find the correct document?

Did it retrieve the appropriate passage?

Did it cite the right source?

Did it produce the correct answer?

Those signals can become rewards.

The Search Stack Behind the Model

The database side is handled through Neon's Lakebase Search.

According to Neon, the Castform pipeline uses the database throughout the lifecycle:

The retrieval implementation combines two complementary approaches.

Keyword retrieval

BM25-style search is useful when exact terms matter.

Think:

PostgreSQL error 53300

Exact tokens carry enormous information.

Vector retrieval

Vector search handles semantic similarity.

For example:

Why won't Postgres accept more connections?

could retrieve documents discussing:

remaining connection slots are reserved

even though the wording differs.

Instead of choosing between them, the demonstrated search tool retrieves results from both approaches and combines their rankings using Reciprocal Rank Fusion (RRF).

Conceptually:

  flowchart TD
    Query["Query"] --> BM25["BM25 Search"]
    Query --> Vector["Vector Search"]
    BM25 --> RRF["RRF Merge"]
    Vector --> RRF
    RRF --> Docs["Ranked Documents"]

This gives the agent both lexical precision and semantic matching.

The Reward Function Is the Secret Sauce

Retrieving something isn't enough.

The agent needs to retrieve the right evidence.

Neon's example describes rewarding multiple aspects of a run:

flowchart TD
    Reward["Reward"] --> CR["Correct retrieval"]
    Reward --> CC["Correct citation"]
    Reward --> CA["Correct final answer"]

That's an important distinction.

Suppose a model guesses the correct answer without retrieving supporting evidence.

A conventional evaluation might mark it correct.

A retrieval-oriented reward can penalize it because the agent failed to ground its answer properly.

Conversely, retrieving an appropriate document but producing the wrong answer shouldn't receive full credit either.

The desired behavior is the combination:

Retrieve → Ground → Cite → Answer

So Did a 4B Model Really Beat GPT-5.6 Sol?

This requires careful wording.

Neon's reported result is that its Castform-post-trained 4B open model retrieved search results as accurately as GPT-5.6 Sol on the evaluated workload while costing around 100× less.

That does not mean:

4B model > GPT-5.6 Sol

in general.

It means something more interesting:

flowchart LR
    subgraph Scope["Specific Retrieval Task"]
        direction LR
        M1["Specialized 4B Model"] <-->|"≈ Comparable Performance"| M2["Frontier Model"]
    end

That's an important difference.

There's no evidence from this experiment that the 4B model matches GPT-5.6 Sol across unrelated capabilities such as general reasoning, mathematics, coding, or creative generation.

The reported advantage comes from specialization.

Why the 100× Cost Difference Matters

A 100× difference changes engineering decisions.

Imagine two approaches:

flowchart TD
    subgraph ArchA["Architecture A"]
        Task["Every task"] --> Model["Frontier Model"]
    end

versus:

flowchart LR
    subgraph ArchB["Architecture B"]
        direction LR
        Task["Task"] -->|"Search"| SSM1["Specialized Small Model"]
        Task -->|"Ranking"| SSM2["Specialized Small Model"]
        Task -->|"Hard Reasoning"| FM["Frontier Model"]
    end

Architecture B doesn't eliminate frontier models.

It uses them selectively.

That's potentially much more economical.

Instead of paying for maximum intelligence on every operation, developers can route simpler or specialized workloads toward cheaper models.

Training Infrastructure Matters Too

RL training generates unusual database workloads.

Thousands of parallel rollouts may repeatedly execute searches, creating bursts of activity followed by periods of relative inactivity.

Neon argues its autoscaling infrastructure fits this pattern because compute can scale with demand and decrease during idle periods.

The company also points toward database branching as an important capability for future stateful agents.

If an agent needs to modify data while learning, each training rollout ideally operates inside an isolated environment.

Otherwise:

flowchart TD
    A["Agent A changes database"] --> B["Agent B sees the change"] --> C["Training becomes contaminated"]

Database branches could instead provide isolated states:

flowchart TD
    DB[("Production Database")]
    DB --> RunA["Run A"]
    DB --> RunB["Run B"]
    DB --> RunC["Run C"]

Each agent can experiment without corrupting another rollout or touching production state.

The Bigger Lesson: Specialization May Beat Scale

The most interesting takeaway isn't the benchmark itself.

It's the architectural philosophy behind it.

For years, AI development largely followed:

Need better performance? Use a bigger model.

Castform suggests another strategy:

Need better performance? Train a smaller model specifically for your problem.

That's a fundamentally different approach.

The future AI stack might therefore contain many models:

flowchart TD
    App["AI Application"]
    Resp["Response"]

    App --> Ret["Retrieval Model"]
    App --> Code["Coding Model"]
    App --> Reas["Reasoning Model"]

    Ret --> Resp
    Code --> Resp
    Reas --> Resp

Rather than sending everything to one expensive frontier model, applications could orchestrate specialized models according to the task.

Final Thoughts

Neon and Castform's experiment demonstrates something developers should pay attention to.

The competition in AI isn't only about building increasingly powerful foundation models.

It's increasingly about extracting more capability from smaller models through better training, tools, data, and infrastructure.

A 4B model doesn't need to become GPT-5.6 Sol.

It needs to become exceptionally good at the task you actually need it to perform.

For retrieval, Neon reports that Castform's post-training approach managed exactly that—matching GPT-5.6 Sol's retrieval accuracy on its evaluated workload while operating at roughly 1/100th of the inference cost.

If results like these generalize beyond individual workloads, the next major optimization in AI applications may not be upgrading to a larger model.

It may be realizing that you never needed the larger model for every step in the first place.

Key Takeaways

• Agentic retrieval is replacing one-shot RAG for complex search workflows.

• Repeated frontier-model calls can make agentic search expensive and slow.

• Castform uses RL post-training to specialize open models for retrieval.

• Existing company data can be transformed into synthetic training tasks.

• The demonstrated pipeline combines BM25 and vector retrieval with RRF.

• Training rewards retrieval quality, citation quality, and answer correctness.

• Neon reports a 4B open model achieved comparable retrieval accuracy to GPT-5.6 Sol at roughly 100× lower inference cost on its evaluation.

• The result applies to the evaluated retrieval workload—not general model capability.

• Specialized models could reduce dependence on frontier models for routine agent operations.

Reference

Neon Engineering — How Castform + Neon Beats Frontier Models on Price and Efficiency, published August 5, 2026.