{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"nvidia-nemotron-3-5-bolt-e4b-4gb-specs-benchmarks-and-how-to-run-it-locally-k7adh","url":"https://zyvop.com/nvidia-nemotron-3-5-bolt-e4b-4gb-specs-benchmarks-and-how-to-run-it-locally-k7adh","title":"NVIDIA Nemotron 3.5 Bolt E4B 4GB: Specs, Benchmarks, and How to Run It Locally","subtitle":"An open 8B (4B effective) model post-trained from Gemma 4 E4B for local voice agents: specs, benchmarks, RTX and Jetson latency, and how to run it.","tldr":"NVIDIA’s Nemotron 3.5 Bolt E4B 4GB is an open small language model post-trained from Gemma 4 E4B with reinforcement learning only. This post covers its specs, what the 4GB really means, benchmark wins and losses against Gemma and Qwen, latency on RTX and Jetson, and how to run it locally.","keywords":["nvidia","small language models","Voice Agents","Edge AI","Nemotron 3.5 Bolt","AI Models"],"entities":["ZyVOP","nvidia","small language models","Voice Agents","Edge AI","Nemotron 3.5 Bolt","AI Models"],"keyTakeaways":["NVIDIA released Nemotron 3.5 Bolt E4B 4GB on October 23, 2026.","It is an open small language model post-trained from Google's Gemma 4 E4B, tuned to act as the conversational brain of a local voice assistant, game character, or robot that must answer fast and call tools correctly.","It is the smallest of three Nemotron 3.5 models announced together, next to the 120B Super VL and the 74B-A9B Bolt."],"headings":["At a glance","What the 4GB in the name means","Run it in five minutes","Calling it from code","Asynchronous tool calls for voice agents","Latency on real hardware","DFlash speculative decoding","How NVIDIA trained it","What the benchmarks say","Caveats the headline skips","Verdict"],"outboundLinks":["https://blogs.nvidia.com/blog/nemotron-3-5-super-vl/","https://research.nvidia.com/labs/nemotron/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB/","https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-Technical-Report.pdf","https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB","https://unsloth.ai/docs/models/nemotron-3.5-bolt","https://huggingface.co/google/gemma-4-E4B-it","https://ollama.com/library/nemotron-3.5-bolt","https://github.com/NVIDIA-NeMo/Gym","https://labs.scale.com/leaderboard/multichallenge?tab=small","https://artificialanalysis.ai/models/nemotron-3-5-bolt-e4b-4gb"],"contentText":"NVIDIA released Nemotron 3.5 Bolt E4B 4GB on October 23, 2026. It is an open small language model post-trained from Google's Gemma 4 E4B, tuned to act as the conversational brain of a local voice assistant, game character, or robot that must answer fast and call tools correctly. It is the smallest of three Nemotron 3.5 models announced together, next to the 120B Super VL and the 74B-A9B Bolt. Figures below come from NVIDIA's launch post, technical report, and model card unless another source is named. At a glance Property Value Source Base model Gemma 4 E4B-it from Google Model card Size 8B parameters, 4B effective Model card Context 64K tokens Model card Post-training Reinforcement learning only, no SFT stage Technical report Builds Q4_K_M GGUF, BF16, and two DFlash drafters Launch post License OpenMDW-1.1 (Gemma 4 itself is Apache 2.0) Model card, Gemma 4 card What the 4GB in the name means The 4GB is a memory target for running the model, not the size of the download. The launch post says the model fits conversational agent intelligence into as little as about 4GB in its target workloads, and its hardware table shows peak VRAM between 3.1 and 5.0 GB. Unsloth's guide lists the E4B Q4_K_M GGUF at 6.18GB to download. The gap comes from Gemma's design. Google's Gemma 4 card defines E4B as 4.5B effective parameters, 8B including embeddings, because the per-layer embedding tables are large but only used for lookups. Measure peak VRAM on your own runtime before sizing hardware. Run it in five minutes The quickest route is Ollama, with one trap. The bare tag nemotron-3.5-bolt resolves to a 25GB model with a 256K context window, which, going by the Ollama page's readme, is the larger 74B-A9B sibling and not E4B. Ask for the E4B tag, which Ollama lists at 7.2GB to 7.9GB on disk. ollama run nemotron-3.5-bolt:e4bFor the full NVIDIA setup, the model card gives a llama.cpp server recipe with the speculative decoding drafter. Drop the four --spec-* flags to run without it. export MODEL_REPO=nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB export MODEL_FILE=Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf export DRAFTER_REPO=nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash export DRAFTER_FILE=Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf export MODEL_DIR=\"$PWD/bolt-models\" hf download \"$MODEL_REPO\" \"$MODEL_FILE\" --local-dir \"$MODEL_DIR\" hf download \"$DRAFTER_REPO\" \"$DRAFTER_FILE\" --local-dir \"$MODEL_DIR\" docker run --rm --gpus all -p 8000:8000 -v \"$MODEL_DIR:/models:ro\" \\ ghcr.io/ggml-org/llama.cpp:server-cuda13-b11459 \\ -m \"/models/$MODEL_FILE\" \\ --spec-draft-model \"/models/$DRAFTER_FILE\" \\ --spec-type draft-dflash --spec-draft-n-max 7 --spec-draft-ngl 999 \\ --alias nemotron-bolt --host 0.0.0.0 --port 8000 \\ -c 65536 -ngl 999 --jinjaFor vLLM with tool calling, the model card says to add --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4 to the serve command. Calling it from code The server speaks the OpenAI chat completions protocol. Bolt exposes two switches through chat template arguments: enable_thinking toggles reasoning, and conversational_mode adds a system instruction for concise, light-formatting replies that suit speech. NVIDIA recommends temperature 1.0 and top_p 0.95 in both thinking modes, and the model card says conversational mode combines with thinking on or off and with tool calling. from openai import OpenAI client = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"EMPTY\") def ask(messages, tools=None, thinking=False, spoken=True): extra = {\"tools\": tools} if tools else {} return client.chat.completions.create( model=\"nemotron-bolt\", messages=messages, max_tokens=512, temperature=1.0, top_p=0.95, extra_body={ \"skip_special_tokens\": False, \"chat_template_kwargs\": { \"enable_thinking\": thinking, \"conversational_mode\": spoken, }, }, **extra, ) reply = ask([{\"role\": \"user\", \"content\": \"What is the capital of Japan?\"}]) print(reply.choices[0].message.content)Reasoning is on by default, so set enable_thinking to false for the fastest replies. If you enable it, raise the sample's max_tokens of 512, since the model card's examples use 16000. To cap thinking instead, llama-server accepts --reasoning on --reasoning-budget <tokens>, though the card warns that a tight budget can truncate reasoning mid-sentence. Asynchronous tool calls for voice agents Voice agents face a problem chat apps do not: a tool can take longer than the user will wait in silence. Bolt's chat template rejects stray tool messages, so the model card describes a workaround. Acknowledge the call at once with a tool message whose status is pending, let the model keep talking, then deliver the result later as a user message prefixed with a marker. sequenceDiagram participant U as User participant A as App participant M as Bolt E4B participant T as Slow tool U->>A: Build me the weekly report A->>M: messages M-->>A: text plus tool_call build_report A->>T: start build_report A->>M: tool message with status pending M-->>A: On it, anything else meanwhile? T-->>A: result ready A->>M: user message starting TOOL UPDATE M-->>A: Your report is ready.Latency on real hardware NVIDIA measured the Q4_K_M build at batch size 1 on llama.cpp with CUDA, with a 3K-token prompt and 50 output tokens. The last column is my own estimate for a 40-token spoken reply, calculated as time to first token plus 40 divided by decode speed. It ignores thinking, speech synthesis, and network time. Hardware Peak VRAM (GB) TTFT (ms) Decode (tok/s) Est. 40-token reply (s) RTX 5070 (12GB) 3.5 548 133.4 0.85 RTX Spark (32GB) 3.1 930 67.0 1.53 Jetson AGX Orin (32GB) 4.2 3,687 24.0 5.35 Jetson Orin Nano (8GB) 4.2 5,785 15.5 8.37 Time to first token dominates on Jetson: a 3K-token prompt in 5,785 ms on the Orin Nano implies prompt processing of roughly 520 tokens per second. On that class of device, keep the system prompt and tool schemas small. DFlash speculative decoding The release ships a DFlash drafter that proposes seven tokens per step in one forward pass for the main model to verify. Per the technical report, it is a five-layer transformer with 360M parameters, not counting the token embeddings and LM head it reuses from the target. Verification preserves the target model's output distribution, so quality is unchanged. In the report's BF16 NIM tests with reasoning on and one user, decode speed rose from 124.7 to 341.2 tokens per second on an RTX PRO 6000 (2.74x) and from 20.1 to 71.6 on a DGX Spark (3.56x). Speculative decoding shortens decoding, not prefill, and time to first token moved by less than 100 ms in every setting, so expect no help with Jetson prompt latency. Against Gemma 4 E4B with its own MTP drafter, the report puts the local Q4_K_M gain at 1.21x on an RTX 5070 (272.6 versus 225.9 tokens per second) and 1.05x on RTX Spark, measured on the coding subset of SPEED-Bench. That 272.6 is a different workload from the 133.4 in the launch post's latency table, which does not say whether it used the drafter, so do not read one as a speedup of the other. How NVIDIA trained it Per the technical report, the recipe is multi-environment reinforcement learning from verifiable rewards and from human feedback, run through NeMo RL and NeMo Gym. The model card names GRPO as the algorithm. The seven environments are tool calling, tool-call schemas, instruction following, instruction hierarchy, structured outputs, free-form formatting, and GenRM. The report adds that every sample was used in both modes at a 2 to 1 ratio of non-reasoning to reasoning, which suits latency-bound agents that run with thinking off. Two decisions are worth copying. Verbosity grew as RL progressed, so NVIDIA raised group-relative length penalties in the GenRM environment and disabled overlong filtering elsewhere. And the team skipped SFT: the report says SFT followed by RLVR reached a higher ceiling on multi-turn tool calling, but SFT on data from diverse teacher models regressed conversational quality and verbosity. NVIDIA then quantized the BF16 checkpoint to Q4_K_M with an importance matrix, which the report says was calibrated on about 1M tokens of RL rollouts. My unweighted mean over the report's Table 5 is 63.58 for that build against 61.01 for Q4_K_M without calibration, and my mean over Table 3 is 63.85 for BF16, so quantization costs about 0.3 points on that crude average. What the benchmarks say NVIDIA evaluated 21 benchmarks across five groups using its NeMo Gym harness, against Gemma 4 E4B-it, Qwen3.5 4B, and its previous Nemotron 3 Nano 4B. The table shows 14 of the 21 BF16 rows, including all six losses, with the best score in bold. Runs used reasoning mode except RoleBench, EQ-Bench v3, and RULER. Two of the 21 benchmarks, Brevity Bench and When2Call-MT, are internal and not yet public. Benchmark Bolt E4B Gemma 4 E4B Qwen3.5 4B Nano 4B IHEval (instruction hierarchy) 84.79 67.04 60.57 51.31 MultiChallenge 53.10 42.08 51.76 35.30 RoleBench 67.05 49.95 53.66 12.02 IFBench 63.79 42.84 58.91 60.10 When2Call-MT 63.36 62.90 37.85 27.70 RelyToolBench 63.16 61.60 43.35 41.20 GaRAGe 42.61 34.73 24.66 25.18 LongMemEval 72.10 70.95 66.85 25.20 Tau2-Bench 55.10 38.50 75.06 29.20 BFCL-v3 68.51 66.39 70.95 61.10 ToolSandbox 68.77 65.43 69.54 57.88 RULER (64K) 86.99 86.55 95.15 91.51 LongBench v2 (48K) 50.50 49.34 59.93 31.40 EQ-Bench v3 39.40 41.17 32.92 19.31 Source: technical report, Table 3. Gemma is the starting checkpoint, so the gap between those two columns is what post-training added. The gain, computed from Table 3, is 17.75 points on IHEval, 20.95 on IFBench, 17.10 on RoleBench, and 16.60 on Tau2-Bench, but only 0.44 on RULER, and EQ-Bench v3 sits 1.77 points below base. NVIDIA reports wins on 15 of 21 benchmarks, and a recount of Table 3 gives the same 15. The losses are EQ-Bench v3 to Gemma, and BFCL-v3, ToolSandbox, Tau2-Bench, RULER, and LongBench v2 to Qwen3.5 4B. In the 4-bit comparison, which is the build you will actually run, Bolt's Q4_K_M leads the other three models' Q4_K_M builds on 11 of 21 rows, which matches the bolding on the model card. The launch post also claims roughly 40% higher average scores than Nano 4B and roughly 10% higher than other peers. An unweighted mean of the 21 BF16 scores gives 63.85 for Bolt against 44.77 for Nano (+42.6%), 58.10 for Gemma (+9.9%), and 57.75 for Qwen (+10.6%). That mean mixes unlike metrics, so treat it as a sanity check, not a ranking. Outside confirmation is thin. Scale's small-model MultiChallenge leaderboard lists Bolt in thinking mode first at 56.03 plus or minus 6.20, ahead of Gemma 4 E4B at 43.23 and Nano 4B at 38.67. It has Qwen3.5-4B at 25.13 where NVIDIA's report has 51.76, and the two use different judges, so compare the ordering, not the numbers. Artificial Analysis showed no Intelligence Index score when checked on October 31, 2026. The launch post also claims up to about 80% fewer output tokens than peers in reasoning-enabled comparisons. The supporting chart is an image on a log axis, so verify the saving on your own prompts. Caveats the headline skips Post-training used English only. Per the technical report, translation held up, with average English-to-X xCOMET at 74.4 (0.2 above Gemma) and X-to-English unchanged at 83.4, but multilingual knowledge slipped slightly: MMLU-ProX averages 66.15 in reasoning mode, 1.52 below Gemma. Training was also text-only, which affected the other modalities unevenly. In reasoning mode, the report shows audio improving and vision degrading: VoiceBench rose to 80.30 (+2.10) on inputs limited to 30 seconds, while MMMU-Pro Vision fell to 42.02, 5.61 below Gemma, and the OmniDocBench 1.5 score, where lower is better, doubled from 0.21 to 0.42. Sources also disagree on a few specs. The GGUF model card lists text in and text out, while the NIM described in the report accepts text, image, and audio at 65,536 tokens. Ollama lists a 128K window, and Artificial Analysis lists 131K and calls the model a mixture of experts, which conflicts with the per-layer-embedding design above. The base Gemma 4 E4B supports 128K. Use NVIDIA's documented 64K as your limit. Verdict Bolt E4B fits a narrow, valuable slot. It is strongest where an agent must hold a persona, obey the system prompt over a conflicting user message, avoid inventing tool results, and keep replies short. It is weaker as a long-dialog task completer, where Tau2-Bench favors Qwen3.5 4B by 20 points, and at retrieval across its full context, where RULER favors Qwen and Nano. The practical test is cheap. If your pipeline is speech in, speech out, with tool calls, on a 4GB to 8GB GPU or a Jetson, build a 50-turn evaluation from your own transcripts. Run it with thinking on and off and with and without conversational mode, and compare against Gemma 4 E4B-it and Qwen3.5 4B at the same quantization. Benchmark gains only matter if they show up in your application.","contentHash":"sha256:0f58d0d569d2736e0c2ba4a0e552117368ba9bee1faff5030244aeeec171765a","authorName":"ZyVOP","authorUrl":"https://zyvop.com/author/zyvop","authorSameAs":["https://zyvop.com","https://github.com/zyvop","https://x.com/zyvop1"],"category":"AI Models","tags":["nvidia","small language models","Voice Agents","Edge AI","Nemotron 3.5 Bolt"],"audience":"Software engineers and developers building applications with AI Models","tone":"Instructional, practical, code-first","readingTimeMinutes":9,"wordCount":2187,"faqs":null,"primaryTopic":"AI Models","publishedAt":"2026-11-01T07:00:23.226Z","updatedAt":"2026-11-01T07:00:23.226Z","canonicalUrl":"https://zyvop.com/nvidia-nemotron-3-5-bolt-e4b-4gb-specs-benchmarks-and-how-to-run-it-locally-k7adh"}