NVIDIA Rubin CPX: A New GPU Role for Long-Context Inference

NVIDIA Rubin CPX is hard to summarize as merely “a faster GPU.” NVIDIA’s main point is disaggregated inference: separating the context phase, where a model reads a long input, from the generation phase, where it produces tokens one by one. That distinction matters as coding, research, and video-generation workloads push toward million-token context windows.

Earlier discussion of the AI factory already showed that modern AI infrastructure is designed at rack scale rather than as a single server. NVIDIA’s Vera Rubin platform deep dive makes the same point by treating CPU, GPU, networking, security, and cooling as a co-designed system. Rubin CPX adds a specialized role on top of that system: digesting long inputs quickly. As inference becomes multi-stage, cost depends on which chip performs which part of the work.

Background

NVIDIA’s technical blog divides inference into two phases. The context phase is compute-bound: it processes a large input and produces the first-token-ready internal state. The generation phase is more memory-bandwidth-bound: it reads KV cache and emits output token by token. Because the bottlenecks differ, running both phases on the same undifferentiated GPU pool can waste resources.

Rubin CPX targets the context phase. NVIDIA says Rubin CPX provides 30 petaFLOPs of NVFP4 compute, 128 GB of GDDR7 memory, hardware video decode and encode support, and 3x attention acceleration compared with NVIDIA GB300 NVL72. The Vera Rubin NVL144 CPX rack combines 144 Rubin CPX GPUs, 144 Rubin GPUs, and 36 Vera CPUs, with 8 exaFLOPs of NVFP4 compute, 100 TB of high-speed memory, and 1.7 PB/s of memory bandwidth.

Phase	Main bottleneck	Where Rubin CPX fits
Context	Large-input compute	Reading long prompts, documents, codebases
KV Cache	State transfer	Passing context results into generation
Generation	Memory bandwidth	Sustained token output
Orchestration	Routing and batching	Dynamo-like layers place work on resources

Principle

Disaggregated inference is like splitting kitchen prep from cooking. Preparing ingredients and cooking dishes operate at different speeds. If one station prepares large batches quickly and another serves orders steadily, total throughput rises. LLM inference has a similar split: reading long context and generating output stress different parts of the system.

Imagine an AI coding agent reading an entire large repository plus issue history. The context phase must process many files and logs into a useful internal state. The generation phase then produces a fix explanation, code diff, and test summary step by step. Rubin CPX accelerates the first part, while Rubin GPUs, Vera CPUs, and networking support generation and system operation. This is why AI server bottlenecks are moving from single-chip peak speed toward data movement and phase placement.

A million-token context window does not mean teams should throw everything into the prompt. Even if a system can process long input, poor selection and unmanaged KV cache movement can quickly increase cost and latency.

Structure

<Disaggregated inference phases around NVIDIA Rubin CPX 3.1>

Context represents bulk input processing. KV Cache is the intermediate state. Generation emits tokens, and Output is what users receive. Rubin CPX is aimed especially at accelerating Context so downstream generation spends less time waiting.

<Conceptual Vera Rubin NVL144 CPX rack layout 3.2>

The second diagram simplifies the rack. Vera CPU handles system and control roles. Rubin CPX handles long-context processing. Rubin GPU handles generation. Network moves data inside and outside the rack. Real systems are more complex, but the important point is that different chips behave like one inference factory.

Checkpoints

NVIDIA’s ROI and performance claims are meaningful under specific large-scale utilization, power, and workload assumptions. They should not be read as a guarantee that every enterprise will see 30x to 50x ROI. Smaller teams should start with cloud pricing and actual utilization.

Long context expands data-governance risk. Feeding entire codebases, research archives, or customer logs into inference makes privacy and trade-secret boundaries harder to manage.

Software orchestration matters as much as hardware. If layers such as Dynamo or TensorRT-LLM do not route work and manage KV cache movement well, specialized chips lose part of their advantage.

NVIDIA Rubin CPX sends a clear message: inference is now a system design problem. The million-token context era will be shaped not only by model size, but by how cheaply and reliably infrastructure can read long inputs.