--- title: "NVIDIA Nemotron 3.5 Bolt E4B 4GB: Specs, Benchmarks, and How to Run It Locally" author: "ZyVOP" published: "2026-11-01T07:00:23.226Z" updated: "2026-11-01T07:00:23.226Z" url: "https://zyvop.com/nvidia-nemotron-3-5-bolt-e4b-4gb-specs-benchmarks-and-how-to-run-it-locally-k7adh" canonical: "https://zyvop.com/nvidia-nemotron-3-5-bolt-e4b-4gb-specs-benchmarks-and-how-to-run-it-locally-k7adh" tags: ["nvidia","small language models","Voice Agents","Edge AI","Nemotron 3.5 Bolt"] --- # NVIDIA Nemotron 3.5 Bolt E4B 4GB: Specs, Benchmarks, and How to Run It Locally An open 8B (4B effective) model post-trained from Gemma 4 E4B for local voice agents: specs, benchmarks, RTX and Jetson latency, and how to run it. ![Cover image]() NVIDIA released Nemotron 3.5 Bolt E4B 4GB on October 23, 2026. It is an open small language model post-trained from Google's Gemma 4 E4B, tuned to act as the conversational brain of a local voice assistant, game character, or robot that must answer fast and call tools correctly. It is the smallest of three [Nemotron 3.5 models announced together](), next to the 120B Super VL and the 74B-A9B Bolt. Figures below come from NVIDIA's [launch post](), [technical report](), and [model card]() unless another source is named. ## At a glance | Property | Value | Source | | --- | --- | --- | | Base model | Gemma 4 E4B-it from Google | Model card | | Size | 8B parameters, 4B effective | Model card | | Context | 64K tokens | Model card | | Post-training | Reinforcement learning only, no SFT stage | Technical report | | Builds | Q4\_K\_M GGUF, BF16, and two DFlash drafters | Launch post | | License | OpenMDW-1.1 (Gemma 4 itself is Apache 2.0) | Model card, Gemma 4 card | ## What the 4GB in the name means The 4GB is a memory target for running the model, not the size of the download. The [launch post]() says the model fits conversational agent intelligence into as little as about 4GB in its target workloads, and its hardware table shows peak VRAM between 3.1 and 5.0 GB. [Unsloth's guide]() lists the E4B Q4\_K\_M GGUF at 6.18GB to download. The gap comes from Gemma's design. Google's [Gemma 4 card]() defines E4B as 4.5B effective parameters, 8B including embeddings, because the per-layer embedding tables are large but only used for lookups. Measure peak VRAM on your own runtime before sizing hardware. ## Run it in five minutes The quickest route is Ollama, with one trap. The bare tag `nemotron-3.5-bolt` resolves to a 25GB model with a 256K context window, which, going by the [Ollama page's]() readme, is the larger 74B-A9B sibling and not E4B. Ask for the E4B tag, which Ollama lists at 7.2GB to 7.9GB on disk. ```bash ollama run nemotron-3.5-bolt:e4b ``` For the full NVIDIA setup, the [model card]() gives a llama.cpp server recipe with the speculative decoding drafter. Drop the four `--spec-*` flags to run without it. ```bash export MODEL_REPO=nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB export MODEL_FILE=Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf export DRAFTER_REPO=nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash export DRAFTER_FILE=Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf export MODEL_DIR="$PWD/bolt-models" hf download "$MODEL_REPO" "$MODEL_FILE" --local-dir "$MODEL_DIR" hf download "$DRAFTER_REPO" "$DRAFTER_FILE" --local-dir "$MODEL_DIR" docker run --rm --gpus all -p 8000:8000 -v "$MODEL_DIR:/models:ro" \ ghcr.io/ggml-org/llama.cpp:server-cuda13-b11459 \ -m "/models/$MODEL_FILE" \ --spec-draft-model "/models/$DRAFTER_FILE" \ --spec-type draft-dflash --spec-draft-n-max 7 --spec-draft-ngl 999 \ --alias nemotron-bolt --host 0.0.0.0 --port 8000 \ -c 65536 -ngl 999 --jinja ``` For vLLM with tool calling, the model card says to add `--enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4` to the serve command. ## Calling it from code The server speaks the OpenAI chat completions protocol. Bolt exposes two switches through chat template arguments: `enable_thinking` toggles reasoning, and `conversational_mode` adds a system instruction for concise, light-formatting replies that suit speech. NVIDIA recommends temperature 1.0 and top\_p 0.95 in both thinking modes, and the [model card]() says conversational mode combines with thinking on or off and with tool calling. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") def ask(messages, tools=None, thinking=False, spoken=True): extra = {"tools": tools} if tools else {} return client.chat.completions.create( model="nemotron-bolt", messages=messages, max_tokens=512, temperature=1.0, top_p=0.95, extra_body={ "skip_special_tokens": False, "chat_template_kwargs": { "enable_thinking": thinking, "conversational_mode": spoken, }, }, **extra, ) reply = ask([{"role": "user", "content": "What is the capital of Japan?"}]) print(reply.choices[0].message.content) ``` Reasoning is on by default, so set `enable_thinking` to false for the fastest replies. If you enable it, raise the sample's `max_tokens` of 512, since the model card's examples use 16000. To cap thinking instead, llama-server accepts `--reasoning on --reasoning-budget `, though the card warns that a tight budget can truncate reasoning mid-sentence. ## Asynchronous tool calls for voice agents Voice agents face a problem chat apps do not: a tool can take longer than the user will wait in silence. Bolt's chat template rejects stray tool messages, so the [model card]() describes a workaround. Acknowledge the call at once with a tool message whose status is pending, let the model keep talking, then deliver the result later as a user message prefixed with a marker. ```mermaid sequenceDiagram participant U as User participant A as App participant M as Bolt E4B participant T as Slow tool U->>A: Build me the weekly report A->>M: messages M-->>A: text plus tool_call build_report A->>T: start build_report A->>M: tool message with status pending M-->>A: On it, anything else meanwhile? T-->>A: result ready A->>M: user message starting TOOL UPDATE M-->>A: Your report is ready. ``` ## Latency on real hardware [NVIDIA measured]() the Q4\_K\_M build at batch size 1 on llama.cpp with CUDA, with a 3K-token prompt and 50 output tokens. The last column is my own estimate for a 40-token spoken reply, calculated as time to first token plus 40 divided by decode speed. It ignores thinking, speech synthesis, and network time. | Hardware | Peak VRAM (GB) | TTFT (ms) | Decode (tok/s) | Est. 40-token reply (s) | | --- | --- | --- | --- | --- | | RTX 5070 (12GB) | 3.5 | 548 | 133.4 | 0.85 | | RTX Spark (32GB) | 3.1 | 930 | 67.0 | 1.53 | | Jetson AGX Orin (32GB) | 4.2 | 3,687 | 24.0 | 5.35 | | Jetson Orin Nano (8GB) | 4.2 | 5,785 | 15.5 | 8.37 | Time to first token dominates on Jetson: a 3K-token prompt in 5,785 ms on the Orin Nano implies prompt processing of roughly 520 tokens per second. On that class of device, keep the system prompt and tool schemas small. ## DFlash speculative decoding The release ships a DFlash drafter that proposes seven tokens per step in one forward pass for the main model to verify. Per the [technical report](), it is a five-layer transformer with 360M parameters, not counting the token embeddings and LM head it reuses from the target. Verification preserves the target model's output distribution, so quality is unchanged. In the report's BF16 NIM tests with reasoning on and one user, decode speed rose from 124.7 to 341.2 tokens per second on an RTX PRO 6000 (2.74x) and from 20.1 to 71.6 on a DGX Spark (3.56x). Speculative decoding shortens decoding, not prefill, and time to first token moved by less than 100 ms in every setting, so expect no help with Jetson prompt latency. Against Gemma 4 E4B with its own MTP drafter, the report puts the local Q4\_K\_M gain at 1.21x on an RTX 5070 (272.6 versus 225.9 tokens per second) and 1.05x on RTX Spark, measured on the coding subset of SPEED-Bench. That 272.6 is a different workload from the 133.4 in the [launch post's latency table](), which does not say whether it used the drafter, so do not read one as a speedup of the other. ## How NVIDIA trained it Per the [technical report](), the recipe is multi-environment reinforcement learning from verifiable rewards and from human feedback, run through NeMo RL and NeMo Gym. The [model card]() names GRPO as the algorithm. The seven environments are tool calling, tool-call schemas, instruction following, instruction hierarchy, structured outputs, free-form formatting, and GenRM. The report adds that every sample was used in both modes at a 2 to 1 ratio of non-reasoning to reasoning, which suits latency-bound agents that run with thinking off. Two decisions are worth copying. Verbosity grew as RL progressed, so NVIDIA raised group-relative length penalties in the GenRM environment and disabled overlong filtering elsewhere. And the team skipped SFT: the report says SFT followed by RLVR reached a higher ceiling on multi-turn tool calling, but SFT on data from diverse teacher models regressed conversational quality and verbosity. NVIDIA then quantized the BF16 checkpoint to Q4\_K\_M with an importance matrix, which the report says was calibrated on about 1M tokens of RL rollouts. My unweighted mean over the report's Table 5 is 63.58 for that build against 61.01 for Q4\_K\_M without calibration, and my mean over Table 3 is 63.85 for BF16, so quantization costs about 0.3 points on that crude average. ## What the benchmarks say NVIDIA evaluated 21 benchmarks across five groups using its [NeMo Gym]() harness, against Gemma 4 E4B-it, Qwen3.5 4B, and its previous Nemotron 3 Nano 4B. The table shows 14 of the 21 BF16 rows, including all six losses, with the best score in bold. Runs used reasoning mode except RoleBench, EQ-Bench v3, and RULER. Two of the 21 benchmarks, Brevity Bench and When2Call-MT, are internal and not yet public. | Benchmark | Bolt E4B | Gemma 4 E4B | Qwen3.5 4B | Nano 4B | | --- | --- | --- | --- | --- | | IHEval (instruction hierarchy) | **84.79** | 67.04 | 60.57 | 51.31 | | MultiChallenge | **53.10** | 42.08 | 51.76 | 35.30 | | RoleBench | **67.05** | 49.95 | 53.66 | 12.02 | | IFBench | **63.79** | 42.84 | 58.91 | 60.10 | | When2Call-MT | **63.36** | 62.90 | 37.85 | 27.70 | | RelyToolBench | **63.16** | 61.60 | 43.35 | 41.20 | | GaRAGe | **42.61** | 34.73 | 24.66 | 25.18 | | LongMemEval | **72.10** | 70.95 | 66.85 | 25.20 | | Tau2-Bench | 55.10 | 38.50 | **75.06** | 29.20 | | BFCL-v3 | 68.51 | 66.39 | **70.95** | 61.10 | | ToolSandbox | 68.77 | 65.43 | **69.54** | 57.88 | | RULER (64K) | 86.99 | 86.55 | **95.15** | 91.51 | | LongBench v2 (48K) | 50.50 | 49.34 | **59.93** | 31.40 | | EQ-Bench v3 | 39.40 | **41.17** | 32.92 | 19.31 | Source: [technical report, Table 3](). Gemma is the starting checkpoint, so the gap between those two columns is what post-training added. The gain, computed from Table 3, is 17.75 points on IHEval, 20.95 on IFBench, 17.10 on RoleBench, and 16.60 on Tau2-Bench, but only 0.44 on RULER, and EQ-Bench v3 sits 1.77 points below base. NVIDIA reports wins on 15 of 21 benchmarks, and a recount of Table 3 gives the same 15. The losses are EQ-Bench v3 to Gemma, and BFCL-v3, ToolSandbox, Tau2-Bench, RULER, and LongBench v2 to Qwen3.5 4B. In the 4-bit comparison, which is the build you will actually run, Bolt's Q4\_K\_M leads the other three models' Q4\_K\_M builds on 11 of 21 rows, which matches the bolding on the [model card](). The [launch post]() also claims roughly 40% higher average scores than Nano 4B and roughly 10% higher than other peers. An unweighted mean of the 21 BF16 scores gives 63.85 for Bolt against 44.77 for Nano (+42.6%), 58.10 for Gemma (+9.9%), and 57.75 for Qwen (+10.6%). That mean mixes unlike metrics, so treat it as a sanity check, not a ranking. Outside confirmation is thin. Scale's small-model [MultiChallenge leaderboard]() lists Bolt in thinking mode first at 56.03 plus or minus 6.20, ahead of Gemma 4 E4B at 43.23 and Nano 4B at 38.67. It has Qwen3.5-4B at 25.13 where NVIDIA's report has 51.76, and the two use different judges, so compare the ordering, not the numbers. [Artificial Analysis]() showed no Intelligence Index score when checked on October 31, 2026. The launch post also claims up to about 80% fewer output tokens than peers in reasoning-enabled comparisons. The supporting chart is an image on a log axis, so verify the saving on your own prompts. ## Caveats the headline skips Post-training used English only. Per the [technical report](), translation held up, with average English-to-X xCOMET at 74.4 (0.2 above Gemma) and X-to-English unchanged at 83.4, but multilingual knowledge slipped slightly: MMLU-ProX averages 66.15 in reasoning mode, 1.52 below Gemma. Training was also text-only, which affected the other modalities unevenly. In reasoning mode, the report shows audio improving and vision degrading: VoiceBench rose to 80.30 (+2.10) on inputs limited to 30 seconds, while MMMU-Pro Vision fell to 42.02, 5.61 below Gemma, and the OmniDocBench 1.5 score, where lower is better, doubled from 0.21 to 0.42. Sources also disagree on a few specs. The GGUF model card lists text in and text out, while the NIM described in the report accepts text, image, and audio at 65,536 tokens. [Ollama lists a 128K window](), and [Artificial Analysis lists 131K]() and calls the model a mixture of experts, which conflicts with the per-layer-embedding design above. The [base Gemma 4 E4B supports 128K](). Use NVIDIA's documented 64K as your limit. ## Verdict Bolt E4B fits a narrow, valuable slot. It is strongest where an agent must hold a persona, obey the system prompt over a conflicting user message, avoid inventing tool results, and keep replies short. It is weaker as a long-dialog task completer, where Tau2-Bench favors Qwen3.5 4B by 20 points, and at retrieval across its full context, where RULER favors Qwen and Nano. The practical test is cheap. If your pipeline is speech in, speech out, with tool calls, on a 4GB to 8GB GPU or a Jetson, build a 50-turn evaluation from your own transcripts. Run it with thinking on and off and with and without conversational mode, and compare against Gemma 4 E4B-it and Qwen3.5 4B at the same quantization. Benchmark gains only matter if they show up in your application.