{"schemaVersion":"1.0","type":"Article","types":["Article"],"slug":"benchmarking-local-ai-building-a-llama-cpp-vs-ollama-comparison-benchmarking-app-evw56","url":"https://zyvop.com/benchmarking-local-ai-building-a-llama-cpp-vs-ollama-comparison-benchmarking-app-evw56","title":"Benchmarking Local AI: Building a llama.cpp vs. Ollama Comparison & Benchmarking App","subtitle":null,"tldr":"Pros and cons of both tools provided by Bob! Introduction I’ve been reading and seeing blog posts regarding the virtues of llama.cpp versus Ollama. Myself, I'm ...","keywords":["localaisolution","ollama","bob","llamacpp"],"entities":["Alain Airom (Ayrom)","Build Engineer","localaisolution","ollama","bob","llamacpp","ZyVOP"],"keyTakeaways":["Here is how the two stack up across core technical categories: ## Key Takeaways & Comparison Matrix Here is how the two stack up across core technical categories: | Dimension | llama.cpp | Ollama | | ----------------------- | ------------------------------------------------------------ | --------------------------------------------------------- | | **Architecture** | C/C++ native backend engine | Go wrapper daemon around custom `llama.cpp` builds | | **User Experience** | CLI flags, config scripts, manual GGUF management | Single binary / desktop app with REST API & model pulls | | **Model Discovery** | Direct GGUF loading from disk or HF cache | Centralized model library registry (`ollama pull`) | | **Performance Control** | Unrestricted access to context sizing, GPU offloading, threads | Curated defaults optimized for ease of use | | **Ideal For** | Custom pipelines, maximum hardware tuning, offline GGUFs | Rapid prototyping, web wrappers, hassle-free agent setups | Enter fullscreen mode Exit fullscreen mode."],"headings":["Introduction","System Architecture & Overview","Interface & Local Tool Integration","Ollama: Frictionless Desktop & Daemon Workflows","llama.cpp: Granular Binary & Hardware Optimization","Benchmarking Inference: Streaming vs. Subprocess Parsing","Ollama Streaming Runner","llama.cpp Native Dispatcher Runner","Key Takeaways & Comparison Matrix","Conclusion","Links"],"outboundLinks":["https://github.com/aairom/llamacpp-vs-ollama","https://github.com/aairom/llamacpp-vs-ollama/blob/main/Docs/Benchmarking.md","https://bob.ibm.com/"],"contentText":"Pros and cons of both tools provided by Bob! Introduction I’ve been reading and seeing blog posts regarding the virtues of llama.cpp versus Ollama. Myself, I'm an Ollama user since a long time. I decided to ask Bob to build a thorough comparison and if possible a benchmarking code. And amazingly, this is exactly what I got! Bob didn’t just give me a static list of bullet points; he engineered a full-stack Flask application equipped with live environment probes, dynamic GGUF model discovery, and an asynchronous benchmarking engine to test real-time inference side by side. System Architecture & Overview To evaluate both backends objectively, Bob structured the system around a lightweight, single-page application (SPA) frontend paired with a Python/Flask REST backend. ┌─────────────────────────────────────────────────────┐ │ Browser (vanilla JS SPA) │ │ benchRun() → POST /api/bench → poll /api/bench/job │ └────────────────────┬────────────────────────────────┘ │ HTTP ┌────────────────────▼────────────────────────────────┐ │ Flask Backend (app.py) │ │ api_bench() → background worker thread │ │ _run_bench_job() → _bench_ollama() / _bench_llamacpp│ └───────────┬───────────────────┬─────────────────────┘ │ HTTP │ Subprocess ┌───────────▼──────┐ ┌─────────▼────────────────────┐ │ Ollama Daemon │ │ llama CLI / Dispatcher │ │ localhost:11434 │ │ PATH / ~/.local/bin │ │ /api/generate │ │ llama cli -m … -p … -n … │ └──────────────────┘ └──────────────────────────────┘ Enter fullscreen mode Exit fullscreen mode The application serves static comparison data, auto-detects installed local tools and .gguf model files, and executes isolated timing benchmarks. # app.py: Core Flask Application Setup & Data Routes from flask import Flask, jsonify, render_template, requestfrom flask_cors import CORSimport json, os, subprocess, threading, time app = Flask(__name__)CORS(app) DATA_PATH = os.path.join(os.path.dirname(__file__), \"data\", \"comparison.json\")BENCH_PATH = os.path.join(os.path.dirname(__file__), \"data\", \"bench_history.json\") @app.route(\"/\")def index(): return render_template(\"index.html\") @app.route(\"/api/data\")def api_data(): with open(DATA_PATH, \"r\", encoding=\"utf-8\") as fh: return jsonify(json.load(fh)) Enter fullscreen mode Exit fullscreen mode Interface & Local Tool Integration Ollama: Frictionless Desktop & Daemon Workflows Ollama provides an out-of-the-box desktop UI and background daemon process (localhost:11434). It simplifies model management into one-liner commands like ollama run and abstract model tags (e.g., ibm/granite4:3b). The Flask backend probes the live Ollama daemon to discover currently pulled models: @app.route(\"/api/probe/ollama\")def probe_ollama(): result = {\"installed\": False, \"version\": None, \"models\": [], \"error\": None} # Check CLI presence try: proc = subprocess.run([\"ollama\", \"--version\"], capture_output=True, text=True, timeout=5) if \"version\" in proc.stdout.lower() or \"version\" in proc.stderr.lower(): result[\"installed\"] = True except FileNotFoundError: result[\"error\"] = \"ollama binary not found in PATH\" return jsonify(result) # Fetch models via Ollama API try: req = urllib.request.Request(\"http://localhost:11434/api/tags\") with urllib.request.urlopen(req, timeout=5) as resp: data = json.loads(resp.read().decode()) result[\"models\"] = [ { \"name\": m.get(\"name\", \"\"), \"size_gb\": round(m.get(\"size\", 0) / 1e9, 2), \"quantization\": m.get(\"details\", {}).get(\"quantization_level\", \"\"), } for m in data.get(\"models\", []) ] except Exception as exc: result[\"error\"] = f\"Ollama API unreachable: {exc}\" return jsonify(result) Enter fullscreen mode Exit fullscreen mode llama.cpp: Granular Binary & Hardware Optimization llama.cpp focuses on high-performance C/C++ execution with raw hardware access, custom quantizations (such as mxfp4, Q8_0, Q4_0, Q4_K_M), and multi-command dispatchers (llama cli). Bob built a scanner into app.py to auto-discover GGUF weights stored across local Hugging Face caches, custom folders, and system directories: @app.route(\"/api/probe/llamacpp/models\")def probe_llamacpp_models(): models = [] search_dirs = [ os.path.expanduser(\"~/.cache/huggingface/hub\"), os.path.expanduser(\"~/models\"), os.path.expanduser(\"~/Downloads\"), ] for d in search_dirs: if not os.path.isdir(d): continue for dirpath, _, files in os.walk(d): for fname in files: if fname.lower().endswith(\".gguf\"): full = os.path.join(dirpath, fname) models.append({ \"name\": fname, \"path\": full, \"size_gb\": round(os.path.getsize(full) / 1e9, 2) }) return jsonify({\"models\": models}) Enter fullscreen mode Exit fullscreen mode Benchmarking Inference: Streaming vs. Subprocess Parsing The primary technical challenge in comparing both systems is capturing accurate, equivalent performance metrics: Tokens Per Second (TPS): Generation throughput. Time to First Token (TTFT): Perceived initial latency. Prompt Evaluation Time: Time required to ingest input tokens. Ollama Streaming Runner Bob used Ollama’s HTTP streaming API to capture the exact timestamp of the first token chunk received: def _bench_ollama(model: str, prompt: str, n_predict: int) -> dict: url = \"http://localhost:11434/api/generate\" payload = { \"model\": model, \"prompt\": prompt, \"stream\": True, \"options\": {\"num_predict\": n_predict}, } t_start = time.monotonic() t_first_token = None total_tokens = 0 eval_duration_ns = 0 resp = requests.post(url, json=payload, stream=True, timeout=300) for raw_line in resp.iter_lines(): if not raw_line: continue chunk = json.loads(raw_line) # Capture TTFT on first non-empty response token if chunk.get(\"response\") and t_first_token is None: t_first_token = time.monotonic() if chunk.get(\"done\"): eval_duration_ns = chunk.get(\"eval_duration\", 0) total_tokens = chunk.get(\"eval_count\", 0) break t_end = time.monotonic() eval_ms = eval_duration_ns / 1_000_000 if eval_duration_ns else None tps = (total_tokens / (eval_ms / 1000)) if eval_ms else None return { \"time_to_first_token_ms\": round((t_first_token - t_start) * 1000, 2) if t_first_token else None, \"total_time_ms\": round((t_end - t_start) * 1000, 2), \"tokens_per_second\": round(tps, 2) if tps else None, \"tokens_generated\": total_tokens, } Enter fullscreen mode Exit fullscreen mode llama.cpp Native Dispatcher Runner For llama.cpp, the app executes the CLI binary via subprocess.Popen with safe pipe draining to avoid deadlock buffers, parsing standard timing blocks (llama_print_timings): def _bench_llamacpp(model: str, prompt: str, n_predict: int) -> dict: cli_path, _ = _find_llama_cli() cmd = [ cli_path, \"cli\", \"-m\", model, \"-p\", prompt, \"-n\", str(n_predict), \"--no-conversation\", \"--single-turn\", \"--reasoning\", \"off\", \"--log-disable\", \"-e\" ] t_start = time.monotonic() proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True) stdout, stderr = proc.communicate(timeout=300) t_end = time.monotonic() combined = stdout + \"\\n\" + stderr # Extract timings from llama_print_timings log output load_ms = _parse_ms(r\"load time\\s*=\\s*([\\d.]+)\\s*ms\", combined) prompt_ms = _parse_ms(r\"prompt eval time\\s*=\\s*([\\d.]+)\\s*ms\", combined) eval_tps = _parse_tps(r\"eval time.*?([\\d.]+)\\s*tokens per second\", combined) ttft_ms = (load_ms + prompt_ms) if (load_ms and prompt_ms) else prompt_ms return { \"time_to_first_token_ms\": round(ttft_ms, 2) if ttft_ms else None, \"total_time_ms\": round((t_end - t_start) * 1000, 2), \"tokens_per_second\": round(eval_tps, 2) if eval_tps else None, \"command\": \" \".join(cmd), } Enter fullscreen mode Exit fullscreen mode Key Takeaways & Comparison Matrix Here is how the two stack up across core technical categories: ## Key Takeaways & Comparison Matrix Here is how the two stack up across core technical categories: | Dimension | llama.cpp | Ollama | | ----------------------- | ------------------------------------------------------------ | --------------------------------------------------------- | | **Architecture** | C/C++ native backend engine | Go wrapper daemon around custom `llama.cpp` builds | | **User Experience** | CLI flags, config scripts, manual GGUF management | Single binary / desktop app with REST API & model pulls | | **Model Discovery** | Direct GGUF loading from disk or HF cache | Centralized model library registry (`ollama pull`) | | **Performance Control** | Unrestricted access to context sizing, GPU offloading, threads | Curated defaults optimized for ease of use | | **Ideal For** | Custom pipelines, maximum hardware tuning, offline GGUFs | Rapid prototyping, web wrappers, hassle-free agent setups | Enter fullscreen mode Exit fullscreen mode Conclusion Bob’s comparison application made it easy to test and quantify the trade-offs between both tools right on my local machine. I will continue to use ollama, it’s handy, but I’ll use more heavily llama.cpp now that I’ve installed it and begining using it. >>> Thanks for reading <<< Links Github repository for this post: https://github.com/aairom/llamacpp-vs-ollama Detailed comparaison and benchmark: https://github.com/aairom/llamacpp-vs-ollama/blob/main/Docs/Benchmarking.md IBM Bob: https://bob.ibm.com/","contentHash":"sha256:67529c43a46f8b970226000117eb16d5f4cd854a160d97c7f35e98fd520faf38","authorName":"Alain Airom (Ayrom)","authorUrl":"https://zyvop.com/author/alain","authorSameAs":["https://github.com/aairom","https://www.linkedin.com/in/aairom/"],"category":null,"tags":["localaisolution","ollama","bob","llamacpp"],"audience":"Senior software engineers, systems architects, and technical leads working with localaisolution","tone":"Analytical, comparative, and data-driven","readingTimeMinutes":5,"wordCount":1094,"faqs":null,"primaryTopic":"localaisolution","publishedAt":"2026-08-31T05:46:57.511Z","updatedAt":"2026-08-31T13:50:00.100Z","canonicalUrl":"https://dev.to/aairom/benchmarking-local-ai-building-a-llamacpp-vs-ollama-comparison-benchmarking-app-48jj"}