Local AI bill of materials

A starting point for running a local model on hardware a small firm or a school might already own. Prices move. Check the current used market before quoting.

Tiers

Tier	Example hardware	What it runs	Watch out for
Laptop	Apple silicon, 16 GB unified memory	7B to 8B models at Q4	Shared memory. Leave 6 to 8 GB for the system.
Laptop	Recent x86 laptop, 32 GB RAM	7B to 8B on CPU	Slow. Fine for chat, painful for long documents.
Desktop	Used GPU with 12 to 16 GB VRAM	8B to 14B at Q4	Power supply, drivers, case clearance.
Desktop	Used GPU with 24 GB VRAM	14B to 32B at Q4, some MoE	Heat, noise, and a power bill.
Small server	Two used 24 GB GPUs	Larger models and batch work	Maintenance, cooling, and someone to own it.

What to check before buying

• Memory bandwidth, not just capacity. It decides tokens per second.
• VRAM per GPU, not total across GPUs. A model has to fit on one card unless you split it.
• Power supply headroom. A used GPU can draw more than the label suggests.
• Cooling and noise. A small office will notice.
• Driver support on the operating system you will actually use.
• Whether the model you need runs on the backend you can maintain.

Software baseline

• A local runtime that fits the hardware. llama.cpp is the safe default.
• A chat interface the staff will use. Keep it simple.
• A document store for retrieval, if the work needs it.
• Logging and access control, even for a small deployment.
• A backup plan for the model files and the index.

What not to buy

• A GPU you cannot power or cool.
• A server with no named owner.
• A subscription that sends client data to a vendor when a local model would do.
• Hardware for a model you have not tested on the work.

Measure before and after

Record tokens per second, time to first token, memory use, and the tasks that fail. Publish the method and the numbers. See the benchmark method.