Skip to content
C-Gen.AI

Inference engine manager

Orchestrate inference across GPUs, ASICs and RISC-V.

Nucleaton™ manages the inference engines you already use, such as vLLM and llama.cpp, on any accelerator, popular clouds or your own servers, with:

  • Intelligent routing
  • Autoscaling
  • Disaggregated inference
  • Fast model loading
Talk to us earlyHow it works
One inference layer, any chipIllustrative

Applications call the same endpoints. Nucleaton™ manages the inference engines (vLLM, llama.cpp and others) and runs them on GPUs, ASICs and RISC-V accelerators, with room for the next accelerator.

Your applications, agents and APIs · same endpoints
Nucleaton™ inference engine managerIntelligent routing · autoscaling · disaggregated inference · fast model loading · observability
The engines you already use
  • vLLM
  • llama.cpp
  • Other engines
  • GPUsOn popular clouds or your own servers
  • ASICs and RISC-VBuilt for high-volume serving
  • Next acceleratorReady when you are. Talk to us early

Why new chips

Inference has different physics from training.

Every request runs in two phases. Prefill reads the whole prompt at once and keeps compute busy. Decode then produces one token at a time, so compute often waits while weights and cache are read from memory.

Because the phases want different things, new chips are designed for inference alone, and the newest serving systems run the two phases as separate pools.

One request, two phasesSimplified

Prefill processes every prompt token at once: compute is busy and memory reads per token are low. Decode produces one new token per step for each request: compute is mostly waiting and memory reads per token are high.

PrefillEvery prompt token at once
ComputeBusy
Memory reads per tokenLow
DecodeOne new token per step, per request
ComputeMostly waiting
Memory reads per tokenHigh

The landscape

Six chip technologies for inference.

Each makes a different bet on where inference spends its time. A plain-language overview of the technologies, not of any product.

  • GPUs

    General-purpose parallel processors with high-bandwidth memory and the broadest software support.

    Good for
    Training, fine-tuning and serving almost any model.
    Watch for
    Power and cost per token at very high serving volume.

  • Inference ASICs

    Chips designed for one job, with matrix engines and memory sized for serving. Many cloud providers now design their own.

    Good for
    High-volume serving of stable, popular models.
    Watch for
    Narrower model support, and availability often tied to one provider.

  • SRAM-first and wafer-scale

    Keep model data in very fast on-chip memory, or build one very large chip from a whole wafer, to get around the memory bottleneck.

    Good for
    Very low latency and fast token streaming.
    Watch for
    Little memory per chip, so large models span many chips.

  • Dataflow architectures

    Compile the model into a pipeline laid out across the chip, so data flows between operations instead of returning to memory.

    Good for
    Large models running steady, long-lived workloads.
    Watch for
    Compiler support for brand-new model architectures.

  • Memory-centric designs

    Pair large, lower-cost memory with compute placed close to it, so big models and long contexts fit on fewer cards.

    Good for
    Big models and long contexts at a lower cost per card.
    Watch for
    Lower raw bandwidth than high-bandwidth memory, and younger software.

  • RISC-V accelerators

    Programmable cores built on the open RISC-V instruction set, combined with matrix and vector engines, often with open-source software.

    Good for
    Teams that want control and less vendor lock-in.
    Watch for
    More tuning work while the software catches up.

Guide

How to evaluate an inference chip

A checklist for teams deciding whether a new accelerator belongs in their serving fleet.

By the C-Gen.AI engineering team

Test your model, not the brochure

Published numbers come from the vendor’s chosen model, precision and batch size. Measure time to first token, tokens per second per user and cost per million tokens on your own model, with your own traffic pattern, before you commit.

Follow the software

A chip is only as useful as the inference engine that runs on it. Check that your models and features run in that engine. Open-source engines add new hardware through backend plugins, and newer chips usually ship their own SDK and compiler. Support for brand-new model architectures can take time to arrive.

Count memory, not just FLOPS

Decode reads weights and cache for every token, so memory bandwidth and capacity often decide speed and cost. SRAM-first chips are very fast but hold little per chip, HBM balances speed and size, and LPDDR cards hold the most at lower bandwidth.

Plan for a mixed fleet

Few teams will run one kind of chip for long: GPUs for training and flexible work, specialized chips where serving volume justifies them, and new designs arriving every year. The hard part is operating them side by side: deploying engines, distributing requests and seeing health and usage in one place.

Nucleaton™

An inference engine manager, not another engine.

Bring the engines you already trust. Nucleaton™ handles everything around them: where they run, how requests reach them, how they scale and how fast they start.

  • Your engines

    vLLM, llama.cpp and other inference engines, deployed and run by Nucleaton™.

  • Intelligent routing

    Each request goes to the replica best placed to serve it, within the priorities you set.

  • Autoscaling

    Replicas scale up and down with request rate and queue depth.

  • Disaggregated inference

    Prefill and decode run as separate pools, each scaled on its own.

  • Fast model loading

    Model checkpoints are staged so new replicas load and start serving sooner.

  • Stable endpoints

    Applications keep calling the same endpoints when the chip behind them changes.

  • Hybrid by design

    Run inference across popular clouds and your own servers, managed as one platform.

  • Observability and usage

    Observability history for every accelerator, and usage reports by project, team and user.

Disaggregated inference

Split prefill from decode. Scale each on its own.

Prefill and decode stress hardware in opposite ways. Disaggregated inference runs them as separate pools and hands each request’s key-value cache from one to the other, so each pool is sized for its own job.

Separating the phases removes the interference between them, so time to first token (TTFT) and time per output token (TPOT) can each be tuned on their own. Nucleaton™ makes it something you run, not something you build.

Disaggregated inferenceSimplified

Requests arrive through Nucleaton™ routing. A prefill pool reads each prompt in parallel and builds its key-value cache, hands the cache to a decode pool, which streams new tokens back through the same endpoint. Each pool scales on its own.

Led by a world-class team

We’ve designed, built and run some of the largest AI training and inference systems in production.

Dr. Sami Kama

Dr. Sami Kama

Founder & CEO
  • 17 years at CERN
  • Inventor, TensorFlow-TensorRT Integration (NVIDIA)
  • Inventor and Architect, AWS SageMaker
  • Inventor and Architect, AWS SageMaker HyperPod
  • Creator, SageMaker Profiler (world’s first distributed AI profiler)
Richard Vencu

Richard Vencu

Head of Engineering
  • Former Head of HPC & MLOps, Stability AI
  • Managed one of the world’s largest AI training clusters
  • Co-Founder & Engineering Lead, LAION

Questions

Inference beyond GPUs

Is Nucleaton™ an inference engine?
No. It’s an inference engine manager. It works with engines such as vLLM and llama.cpp and handles placement, routing, scaling and model loading around them.
Which chips does Nucleaton™ support?
GPUs and inference ASICs. We’re ready to support new accelerators, so tell us early which ones you’re evaluating.
What is disaggregated inference?
Running the prefill and decode phases of each request as separate pools, so each can be sized and scaled for its own job. Nucleaton™ manages both pools.
Do applications change when the chip changes?
No. Applications call Nucleaton™ endpoints, so the accelerator behind an endpoint can change without changing how it’s called.
Do we have to give up our GPUs?
No. Many teams keep GPUs for training and flexible work, and add specialized chips where serving volume justifies them. Nucleaton™ manages both.
Does it work across clouds?
Yes. Inference runs on popular clouds and on your own servers, managed from one platform.

Early access

Evaluating a new accelerator? Talk to us early.

Nucleaton™ is ready to support new kinds of chips. Tell us which accelerators you’re considering and what you serve, and we’ll plan support with you before you commit. Building a chip, or a cloud on one? We’d like to hear from you too.

Our engineers usually reply within 4 business hours. See our Privacy Policy.