← DAK · Read before you buy

How much memory does a local AI need?

One rule tells you which models fit on a machine, before you buy anything.

An AI model is a long list of numbers, called parameters. To run it, the whole list has to sit in memory: the memory on a graphics card, or the unified memory that a Mac or an AMD Ryzen AI Max shares between its processor and graphics.

The rule: parameters times bits per parameter, divided by eight, gives gigabytes. A 70 billion parameter model stored at 4 bits needs 70 × 4 ÷ 8 = 35 GB. Add 10 to 20 percent for the working memory a conversation needs, and more for very long documents.

Bits are the trade-off. At 8 bits the same model needs about 70 GB and loses almost nothing. At 4 bits it needs 35 GB and stays close to the original, which is why 4 bits is the usual choice. At 2 or 3 bits it fits on much smaller machines, but it gets noticeably less precise.

Rule of thumb: at 4 bits, the gigabytes are about half the billions of parameters, plus a little room.

On our own machines: the Mac mini in Prototype 1.2 has 48 GB, which is comfortable for 27 to 35 billion parameter models and can just fit an 80 billion parameter coder squeezed down to 3 bits. DAK-1 is planned with 128 GB, enough for a 70 billion parameter model and its helpers at the same time.

Memory decides what fits. How fast it answers is a second rule, about memory speed.

Quick answers

How much memory does a 70B model need?

About 35 GB at 4 bits, plus 10 to 20 percent for context, so around 40 GB. About 70 GB at 8 bits.

What is the quick rule for model memory?

Parameters times bits, divided by eight, gives gigabytes. At 4 bits that is about half the billions of parameters.

Keep reading

Tell us about your office.

Headcount, the kind of files you handle and what you'd like it to do. We reply within a business day with a first recommendation.

Request a quote →