Guide
Why serving models is different from running jobs
Training and inference use the same GPUs but behave very differently. Here’s what changes when a model goes into production.
By the C-Gen.AI engineering team
Jobs finish. Endpoints don’t.
A training job starts, runs and ends, and it can wait in a queue without anyone noticing. An inference endpoint runs all day and is expected to answer each request quickly. Batch schedulers are designed to keep queued work flowing; serving needs capacity ready before the requests arrive.
Demand arrives as requests
Traffic rises and falls with your users’ day, and a launch can multiply it overnight. Serving systems scale on signals such as request rate and queue depth, adding capacity when requests start to wait and handing it back when they don’t.
Memory decides how many requests fit
Every active request needs GPU memory for its KV cache, on top of the model weights. Modern serving engines use paged KV-cache management and continuous batching, adding new requests to work already running instead of waiting for a batch to finish. Both let one GPU serve more requests at once.
Training and serving compete for GPUs
Many teams run one fleet for training and another for serving, each with headroom for its own peaks. A shared pool removes the duplicate headroom, but only works if production endpoints always get the capacity they need, whatever training is doing.
