{"schemaVersion":"1.0","type":"Article","types":["Article"],"slug":"building-an-ai-that-never-stops-learning-on-8gb-of-vram-v5co9","url":"https://zyvop.com/building-an-ai-that-never-stops-learning-on-8gb-of-vram-v5co9","title":"Building an AI That Never Stops Learning on 8GB of VRAM","subtitle":"How mini-AGI trains a byte-level, mixture-of-experts model from scratch on a single 8GB GPU, and solves catastrophic forgetting with one learning-rate tweak.","tldr":"Most language models are trained once and frozen forever. mini-AGI is a small byte-level model that keeps learning continuously on a single 8GB consumer GPU, using a mixture-of-experts architecture that pages experts to disk and a simple learning-rate trick to avoid catastrophic forgetting.","keywords":["machine learning","Mixture of Experts","Continual Learning","open-source","local-ai","AI News"],"entities":["Anshu Pathak","machine learning","Mixture of Experts","Continual Learning","open-source","local-ai","AI News","ZyVOP"],"keyTakeaways":["A byte-level model that trains continuously on a single consumer GPU, and doesn't forget what it already knew.","Every large language model you've ever used was trained once, frozen, and shipped.","You can prompt it, fine-tune around its edges, or bolt on a retrieval system, but the model itself stopped learning the moment it left the lab."],"headings":["At a Glance","The Core Problem: You Can't Actually Own a Model","An Architecture That Pages Itself Onto the GPU","Solving Forgetting With One Number in a Config File","It's Not Smart Yet, And It Says So","Trying It Yourself","Why It's Worth Paying Attention To"],"outboundLinks":["https://github.com/volotat/mini-AGI","https://arxiv.org/abs/2107.05407","https://arxiv.org/abs/2401.13660"],"contentText":"A byte-level model that trains continuously on a single consumer GPU, and doesn't forget what it already knew. Every large language model you've ever used was trained once, frozen, and shipped. You can prompt it, fine-tune around its edges, or bolt on a retrieval system, but the model itself stopped learning the moment it left the lab. mini-AGI, a project by developer Alexey Borsky (GitHub: volotat/mini-AGI), asks a different question: what if a language model never had to stop? It's a small, byte-level model that trains from scratch on a single consumer GPU with just 8GB of VRAM, and it keeps learning continuously from whatever text it's pointed at, without the catastrophic forgetting that normally makes continual learning fall apart. At a Glance 🧑‍💻 Author Alexey Borsky · volotat/mini-AGI 🏗️ Architecture Byte-level, mixture-of-experts, adaptive-depth recurrent block 🔢 Total parameters ~468M across 146 experts 💾 Resident in VRAM ~101M (32-expert working set) 🖥️ Hardware needed Single GPU, 8GB VRAM (reference: RTX 3070 Laptop GPU) 📈 Progress so far ~243M characters read · weights not yet released The live training dashboard, showing the model mid-run as it continually reads from its corpus. The Core Problem: You Can't Actually Own a Model The project's motivation is blunt: A personal AI model today is really just someone else's frozen model with a thin layer of customization on top. Try to keep training it on your own data, and it forgets what it already knew. mini-AGI's goal is a model that's actually yours: it lives on hardware you control, it learns from the data and conversations you feed it, and no outside company can revoke access to it or shut it down. Three hard constraints shape the design: Fit in 8GB of VRAM. Brutal, since training needs gradients and optimizer state that roughly triple the size of the weights themselves. Never forget. New information can't come at the cost of what the model already knows. Read anything. No fixed vocabulary, which is why it works directly on raw bytes instead of tokens. An Architecture That Pages Itself Onto the GPU Instead of a fixed stack of transformer layers, each character passes through two dense \"prelude\" blocks and then a single recurrent block that can be applied up to 24 times, with each pass picking its own mixture of experts from a shared pool. A PonderNet-style halting mechanism decides how many passes a character actually needs. Something predictable can get resolved in a single pass, while something ambiguous keeps getting pushed through more rounds. The model learns this depth allocation as part of training. A live capture of the model processing text. Each tile is one expert, each row is one pass through the recurrent block, and the model keeps going until its halting mechanism decides to stop. The clever part is how the experts themselves live on disk. Every expert is a plain file holding its weights and Adam optimizer state. Only a working set of 32 experts sits resident in VRAM at any moment; the rest wait on disk, with a RAM cache in between. Before reading each chunk of text, the model predicts which experts the upcoming content will need and swaps them in. An expert's momentum travels with it rather than being overwritten by whatever replaces it in a VRAM slot. The upshot: how large the model can grow depends on how much free disk space is available, not how much VRAM is on the card. At the time of writing, the model holds about 468M total parameters spread across 146 experts, while only around 101M of those are ever resident on the card at once. The pool isn't fixed in size either: it grows new experts by recombining pieces of existing ones when it's short on capacity, and prunes experts that go unaddressed for too long. Notably, the project found that how often an expert gets picked is a far better signal of whether it's \"alive\" than how confidently it's picked, which upended their first pruning heuristic. Solving Forgetting With One Number in a Config File This is the most interesting result in the whole project. Feed a continual-learning model half a million characters of nothing but chess, and by default it will forget most of what it knew about seven other subjects: held-out loss on those subjects climbs from about 1.12 nats to 3.73 nats. Held-out loss on the seven subjects the model did not read, across three configurations. Two climb sharply; the third, with the trunk learning rate cut to a tenth, barely moves. The fix wasn't the expert pool itself, which the team initially assumed was doing the protective work. Freezing the working set entirely barely helped. What actually mattered was slowing the learning rate of the \"trunk\" (the embeddings, attention, routers, and halting head that every character passes through) to one-tenth the rate used for the experts. That single change took forgetting from a serious regression down to a change so small it's nearly indistinguishable from a model that never specialized on chess at all. Cutting the trunk's learning rate to a tenth of the experts' rate took the forgetting penalty from +2.23 nats down to +0.0067 nats, about 99.84% of the possible improvement recovered. During that same read, only about 40% of the expert pool received any gradient at all, since routing simply never touched the rest. It's Not Smart Yet, And It Says So The README is candid about where the project actually stands: the author calls it a \"toy-level model\" and is upfront that nobody should expect frontier-grade capability from it yet. The model primed with a held-out story (grey) and left to continue on its own (green), using greedy decoding so the same prompt always produces the same output. A sample continuation in the repo, generated by priming the model with a held-out story and letting it write on its own, is grammatically coherent and on-topic but noticeably repetitive, which the author attributes honestly to the model having only read about 243 million characters so far. Weights aren't published yet: the run hasn't finished even one full pass over the training corpus, and at today's pace that's still a little while off. There's a data-scaling comparison against published byte-level models like MambaByte that puts this in perspective: those comparison models trained on well over a hundred times more data. How mini-AGI's held-out loss compares against published byte-level and subword models as training data scales up. The project's own fitted scaling exponent comes out to 0.221. For context, that falls inside the range set by two well-known scaling-law papers: Reference Scaling exponent Kaplan et al. (original estimate) 0.095 mini-AGI (this project) 0.221 Chinchilla (compute-efficient) 0.28 Recent stretches of the run have been trending even steeper, though it's still an early result rather than a finished one. Trying It Yourself Because the model reads raw bytes, there's no dataset prep beyond pointing it at files: python3 train.py read ~/notes # dry read, nothing saved python3 train.py read ~/src ~/docs --passes 3 --saveThe --save flag is what actually commits anything to disk. Leave it off and you get a dry run. Either way, the model scores itself on a held-out set before and after the read, so you get a direct answer about whether feeding it your files cost it anything elsewhere. To run full training you'll need: A CUDA GPU with at least 8GB VRAM (reference machine: RTX 3070 Laptop GPU) Python 3.10+ PyTorch After that, python3 -m corpora all builds an eight-subject training corpus, and serve.py spins up a local web UI to watch the model read and write in real time. Why It's Worth Paying Attention To mini-AGI isn't trying to compete with GPT-class models on capability. What it offers instead is a working demonstration that continual learning without catastrophic forgetting is possible on hardware almost anyone already owns, and that the mechanism protecting it can be as simple as a learning-rate ratio rather than an exotic architecture. The README also discloses that Claude Opus 5 implemented most of the code and helped design and debug the experiments, with the author driving direction, ideas, and verification, a small data point on how AI-assisted research projects are starting to look. For anyone curious about local-first AI, mixture-of-experts systems, or just watching a model's shape reorganize itself live as it reads, the repo's README is worth a full read. It includes GIFs of the model's expert-routing pattern shifting character by character as it works. Images in this post are from the mini-AGI repo.","contentHash":"sha256:e4d67127bbb490474d4bc1e92cd219488aba8cb65945b4e19c87e3375cdaf42d","authorName":"Anshu Pathak","authorUrl":"https://zyvop.com/author/anshu","authorSameAs":[],"category":"AI News","tags":["machine learning","Mixture of Experts","Continual Learning","open-source","local-ai"],"audience":"Developers, software engineers, and students learning AI News","tone":"Practical and evidence-based engineering guidance","readingTimeMinutes":6,"wordCount":1424,"faqs":null,"primaryTopic":"AI News","publishedAt":"2026-09-21T15:42:14.174Z","updatedAt":"2026-09-21T15:42:14.174Z","canonicalUrl":"https://zyvop.com/building-an-ai-that-never-stops-learning-on-8gb-of-vram-v5co9"}