The model outgrew the memory
Weights that no longer fit do not run slowly — they do not load. Past what system memory holds, the answer is a card chosen by memory first, and how much a model needs is worked through on the LLM page.
A Ryzen AI machine serves a small model from its own system memory. A dedicated GPU earns its price at a few well-defined moments. This page is the dividing line between the two, stated plainly.
Before pricing a card, it is worth being clear what a whole Ryzen AI machine does without one — because for a lot of inference, this list is the whole job.
Weights that no longer fit do not run slowly — they do not load. Past what system memory holds, the answer is a card chosen by memory first, and how much a model needs is worked through on the LLM page.
Serving many people at once is what a card’s memory bandwidth and batching are for. That is the point at which a queue on a small machine turns into a case for a dedicated card.
Training holds the weights, the gradients and the optimiser state in memory at once — several times what serving needs. That is card territory, and sometimes more than one card.
When reply pace is what your customer experiences, the card’s faster memory is the honest fix. A small machine is for jobs that think in seconds.