01What makes a model "small"?

There is no single official cutoff, but researchers who survey the field tend to converge on a working definition: a small language model (SLM) is a decoder-only transformer roughly in the 100 million to 5 billion parameter range. A complementary way to define it leans on capability and constraints — an SLM is one that can still do useful, often specialized work while being small enough to run in resource-constrained settings like a phone or a laptop. Both definitions matter, and crucially, "small" is relative and keeps moving: as frontier models grow, yesterday's "large" can become tomorrow's "small."

• A common convention puts SLMs in the ~100M–5B parameter range — a guideline from surveys, not a hard standard.
• An SLM is also defined by what it enables: useful capability that fits resource-constrained, on-device settings.
• "Small" is a moving target — it's measured relative to the frontier, which keeps rising over time.

02Why a small model can punch above its weight

For years the dominant idea was simple: bigger is better — more parameters and more data meant more capability. Small models challenge that in two ways. First, data quality over raw quantity: Microsoft's Phi line argued that training on carefully curated, "textbook-quality" and synthetic data lets a small model deviate from raw scaling laws. Their phi-1 model (1.3B parameters) reached strong coding performance on a tiny data budget, and Phi-3-mini (3.8B) is reported by Microsoft to rival much larger models. Second, good architecture and training choices: open models like TinyLlama (1.1B) and Google's Gemma (2B/7B) show compact models trained well can outperform older models of similar size.

One important caveat: claims that a small model "rivals" or "beats" a much larger one are almost always vendor-reported benchmark results. They're meaningful signals, but they are not the same as independent, real-world proof — read them as "the maker says," not settled fact.

• Data quality thesis: curated + synthetic "textbook-quality" data lets small models punch above their weight (the Phi line).
• Architecture & training matter: TinyLlama and Gemma show well-trained compact models can beat older same-size ones.
• Read parity claims carefully: "rivals a much bigger model" is usually a vendor-reported benchmark, not independent fact.

03Two ways to shrink a model: distillation & quantization

Beyond training a small model from scratch, there are two well-established techniques for fitting capability into a smaller footprint — and they work on different things.

Knowledge distillation shrinks the model itself. A small "student" model is trained to reproduce a large "teacher" model's outputs — specifically its soft probabilities (how confident it was across all the options), not just the single hard answer. Those soft outputs carry richer information, so the student learns more than it would from labels alone. This idea (Hinton, Vinyals & Dean, 2015) is now used in shipping products: Google reports that Gemma 2's 2B and 9B models were trained with distillation.

Quantization shrinks how the weights are stored. Model weights are normally 16-bit numbers; quantization stores them in lower precision — 8-bit, 4-bit, even lower — which cuts file size and memory and speeds up inference, at the cost of some accuracy. Smart methods limit that cost: GPTQ quantizes to 3–4 bits with little accuracy loss, AWQ protects the small fraction of "salient" weights that matter most, and QLoRA introduced a 4-bit format (NF4) good enough to fine-tune on a single consumer GPU.

• Distillation = a small student learns from a big teacher's soft probabilities — compressing capability into fewer parameters.
• Quantization = store the same weights at lower numeric precision (16-bit → 8/4/lower bit) to cut size, memory, and latency.
• Methods like GPTQ, AWQ, and QLoRA keep quality loss small by being selective about precision — they're the levers behind on-device deployment.

04See the tradeoff: size ↔ capability ↔ where it runs

Picking a model size is a balancing act. Bigger usually means more capable — but also heavier: more memory, slower responses, and harder to run anywhere but a server. Drag the slider to change the model size, then flip on quantization and distillation to watch them shrink the memory footprint and pull capability back up. The "Runs on" badge tells you where a model of that size could realistically live.

05Why run AI on-device at all?

If the cloud has the biggest, most capable models, why bother running a smaller one locally? Because moving inference onto the device changes four things at once. Switch between them below. Real shipped examples include Apple's ~3B on-device model (tuned for Apple silicon with KV-cache sharing and 2-bit quantization-aware training), Google's Gemini Nano on Android via AICore, and Meta's MobileLLM family of sub-billion-parameter models.

06Check your understanding

07Take it with you & go deeper