lyogavin/airllm
★ 41.7k
Python
Apache-2.0
A free, open-source way to run 70B AI models on one small GPU
4GB of memory runs a 70-billion-parameter model.
It keeps just one layer in memory.
Normally a 70-billion-parameter model loads into memory all at once. AirLLM loads it one layer at a time instead.
one layer at a time, not the whole 70B model
That's normally 140GB of GPU memory.
That's normally about a hundred forty gigabytes of GPU memory — several expensive, professional GPUs. AirLLM does it with just four gigabytes.
~140GB normally, 4GB with AirLLM — 35x less
Even a 405-billion model needs just 8GB.
The same trick scales further. Llama 3.1, almost six times bigger at four-oh-five billion parameters, still needs only eight gigabytes.
405B Llama 3.1 → still just 8GB
Built mostly by one person. 28,000+ stars.
It's mostly the work of one person, Gavin Li — a former Airbnb and Alibaba engineer. Twenty-eight thousand stars, and the newest release shipped just last week.
28,198 stars, newest release last week