Building an AI That Never Stops Learning on 8GB of VRAM

A byte-level model that trains continuously on a single consumer GPU, and doesn't forget what it already knew.

Every large language model you've ever used was trained once, frozen, and shipped. You can prompt it, fine-tune around its edges, or bolt on a retrieval system, but the model itself stopped learning the moment it left the lab.

mini-AGI, a project by developer Alexey Borsky (GitHub: volotat/mini-AGI), asks a different question: what if a language model never had to stop?

It's a small, byte-level model that trains from scratch on a single consumer GPU with just 8GB of VRAM, and it keeps learning continuously from whatever text it's pointed at, without the catastrophic forgetting that normally makes continual learning fall apart.

At a Glance

The live training dashboard, showing the model mid-run as it continually reads from its corpus.

The Core Problem: You Can't Actually Own a Model

The project's motivation is blunt:

A personal AI model today is really just someone else's frozen model with a thin layer of customization on top. Try to keep training it on your own data, and it forgets what it already knew.

mini-AGI's goal is a model that's actually yours: it lives on hardware you control, it learns from the data and conversations you feed it, and no outside company can revoke access to it or shut it down.

Three hard constraints shape the design:

• Fit in 8GB of VRAM. Brutal, since training needs gradients and optimizer state that roughly triple the size of the weights themselves.

• Never forget. New information can't come at the cost of what the model already knows.

• Read anything. No fixed vocabulary, which is why it works directly on raw bytes instead of tokens.

An Architecture That Pages Itself Onto the GPU

Instead of a fixed stack of transformer layers, each character passes through two dense "prelude" blocks and then a single recurrent block that can be applied up to 24 times, with each pass picking its own mixture of experts from a shared pool.

A PonderNet-style halting mechanism decides how many passes a character actually needs. Something predictable can get resolved in a single pass, while something ambiguous keeps getting pushed through more rounds. The model learns this depth allocation as part of training.

A live capture of the model processing text. Each tile is one expert, each row is one pass through the recurrent block, and the model keeps going until its halting mechanism decides to stop.

The clever part is how the experts themselves live on disk. Every expert is a plain file holding its weights and Adam optimizer state. Only a working set of 32 experts sits resident in VRAM at any moment; the rest wait on disk, with a RAM cache in between.

Before reading each chunk of text, the model predicts which experts the upcoming content will need and swaps them in. An expert's momentum travels with it rather than being overwritten by whatever replaces it in a VRAM slot.

The upshot: how large the model can grow depends on how much free disk space is available, not how much VRAM is on the card.

At the time of writing, the model holds about 468M total parameters spread across 146 experts, while only around 101M of those are ever resident on the card at once.

The pool isn't fixed in size either: it grows new experts by recombining pieces of existing ones when it's short on capacity, and prunes experts that go unaddressed for too long. Notably, the project found that how often an expert gets picked is a far better signal of whether it's "alive" than how confidently it's picked, which upended their first pruning heuristic.

Solving Forgetting With One Number in a Config File

This is the most interesting result in the whole project.

Feed a continual-learning model half a million characters of nothing but chess, and by default it will forget most of what it knew about seven other subjects: held-out loss on those subjects climbs from about 1.12 nats to 3.73 nats.

Held-out loss on the seven subjects the model did not read, across three configurations. Two climb sharply; the third, with the trunk learning rate cut to a tenth, barely moves.

The fix wasn't the expert pool itself, which the team initially assumed was doing the protective work. Freezing the working set entirely barely helped.

What actually mattered was slowing the learning rate of the "trunk" (the embeddings, attention, routers, and halting head that every character passes through) to one-tenth the rate used for the experts. That single change took forgetting from a serious regression down to a change so small it's nearly indistinguishable from a model that never specialized on chess at all.

Cutting the trunk's learning rate to a tenth of the experts' rate took the forgetting penalty from +2.23 nats down to +0.0067 nats, about 99.84% of the possible improvement recovered.

During that same read, only about 40% of the expert pool received any gradient at all, since routing simply never touched the rest.

It's Not Smart Yet, And It Says So

The README is candid about where the project actually stands: the author calls it a "toy-level model" and is upfront that nobody should expect frontier-grade capability from it yet.

The model primed with a held-out story (grey) and left to continue on its own (green), using greedy decoding so the same prompt always produces the same output.

A sample continuation in the repo, generated by priming the model with a held-out story and letting it write on its own, is grammatically coherent and on-topic but noticeably repetitive, which the author attributes honestly to the model having only read about 243 million characters so far. Weights aren't published yet: the run hasn't finished even one full pass over the training corpus, and at today's pace that's still a little while off.

There's a data-scaling comparison against published byte-level models like MambaByte that puts this in perspective: those comparison models trained on well over a hundred times more data.

How mini-AGI's held-out loss compares against published byte-level and subword models as training data scales up.

The project's own fitted scaling exponent comes out to 0.221. For context, that falls inside the range set by two well-known scaling-law papers:

Recent stretches of the run have been trending even steeper, though it's still an early result rather than a finished one.

Trying It Yourself

Because the model reads raw bytes, there's no dataset prep beyond pointing it at files:

python3 train.py read ~/notes                     # dry read, nothing saved
python3 train.py read ~/src ~/docs --passes 3 --save

The --save flag is what actually commits anything to disk. Leave it off and you get a dry run. Either way, the model scores itself on a held-out set before and after the read, so you get a direct answer about whether feeding it your files cost it anything elsewhere.

To run full training you'll need:

• A CUDA GPU with at least 8GB VRAM (reference machine: RTX 3070 Laptop GPU)

• Python 3.10+

• PyTorch

After that, python3 -m corpora all builds an eight-subject training corpus, and serve.py spins up a local web UI to watch the model read and write in real time.

Why It's Worth Paying Attention To

mini-AGI isn't trying to compete with GPT-class models on capability. What it offers instead is a working demonstration that continual learning without catastrophic forgetting is possible on hardware almost anyone already owns, and that the mechanism protecting it can be as simple as a learning-rate ratio rather than an exotic architecture.

The README also discloses that Claude Opus 5 implemented most of the code and helped design and debug the experiments, with the author driving direction, ideas, and verification, a small data point on how AI-assisted research projects are starting to look.

For anyone curious about local-first AI, mixture-of-experts systems, or just watching a model's shape reorganize itself live as it reads, the repo's README is worth a full read. It includes GIFs of the model's expert-routing pattern shifting character by character as it works.

Images in this post are from the mini-AGI repo.