What "Parameters" Actually Means in AI

Parameters are just the weights, counted up

When a model is described as having "70 billion parameters", that is the total number of adjustable values inside its neural network, the weights tuned during training. You can picture them as billions of tiny dials. Training turns the dials until the model's outputs improve.

Parameters are not a list of stored facts. Knowledge is spread across all of them as patterns, which is why you can't point to a single parameter that "knows" a particular thing.

Why parameter count gets so much attention

More parameters generally means more capacity to represent complex patterns, and larger models have tended to perform better across many tasks. That is why it became a headline number when comparing models, even though it is an imperfect one.

But bigger isn't automatically better. Data quality, training technique and architecture matter enormously. A smaller, well-trained model can beat a larger, poorly trained one on specific tasks.

What the numbers mean for hardware

Size translates directly into memory. As a rough calculation, a model with 7 billion parameters stored at 16-bit precision takes about 14 GB (7 billion times 2 bytes). Compressing the numbers to 4 bits, a technique called quantization, cuts that to roughly 3.5 GB, which is why smaller or compressed models can run on a laptop while the largest ones need data-centre hardware. See the hardware lesson for more.

The trade-offs of more parameters

Higher compute cost to train and to run, which shows up in price and response speed.
Diminishing returns. Gains from adding parameters often shrink at very large scales while costs keep rising.
Harder to run locally, as the memory calculation above shows.

Total parameters versus active parameters

Some modern models use a "mixture of experts" design. Only a portion of the model's parameters is used for any given piece of text, so the model can be very large in total while costing less to run than a model of the same size that uses everything every time. This is one reason the headline parameter number can mislead.

Why companies release smaller variants

Many providers offer a large flagship model alongside smaller, faster and cheaper ones. Not every task needs maximum capability, and a small model is often the better choice for simple, high-volume jobs.

Frequently asked

Is a model with more parameters always smarter?

Not necessarily — training quality and technique matter as much as raw size, and comparisons should be based on actual task performance, not parameter count alone.

Why don't companies always publish exact parameter counts?

Some companies treat exact figures as competitive information and don't disclose them publicly, sharing only general performance benchmarks instead.

How many parameters does a "small" model have compared to a "large" one?

This varies a lot by era and provider, but generally smaller models are in the single-digit billions of parameters, while flagship large models range from tens of billions to over a trillion.

← Back Next: Tokens & Context Windows →