Guide
What it takes to run an open model in production
Downloading the weights is the easy part. Here’s what to plan before real users depend on the model.
By the C-Gen.AI engineering team
Size GPU memory to the model
At 16-bit precision, weights take about 2 bytes per parameter, so a 70-billion-parameter model needs roughly 140 GB before it answers a single request. Each active request adds memory for its key-value cache, which grows with context length and with the number of requests served at once. Quantizing to 8 or 4 bits shrinks the weights, sometimes at a cost in quality, so test it on your own tasks.
Fine-tune only what you need
Full fine-tuning updates every weight and needs extra memory for gradients and optimizer state. Parameter-efficient fine-tuning (PEFT) methods such as LoRA and QLoRA train a small set of added weights instead, so a model that would need a large cluster to fully fine-tune can often be adapted on far fewer GPUs.
Measure what users feel
Throughput tells you how many tokens a GPU produces. Users notice how long the first token takes and how steadily the rest arrive. Track both for each endpoint, and size the capacity you guarantee for production from real traffic rather than estimates.
Decide where data lives
Teams often self-host for control: prompts, outputs and fine-tuning data stay inside infrastructure they govern, and they decide what is kept for future training. Write those retention rules down before launch, not after the first audit request.
