DGX Spark for Running Big Local Models: How Large Can You Go?

A single consumer GPU with 24 GB of VRAM hits a wall around a quantised 30B model. The DGX Spark blows past that wall by giving you 128 GB of unified memory that the CPU and GPU share directly, so a 70B model loads comfortably and a 200B model fits with aggressive quantisation. For anyone in South Africa who wants to run serious models locally rather than renting cloud GPUs by the hour, that ceiling is the whole story.

Quick Answer

DGX Spark's 128 GB of coherent unified memory runs models up to roughly 200 billion parameters for inference and fine-tunes models up to about 70 billion parameters, far beyond what a 24 GB or 32 GB consumer GPU can hold. Link two units over the dual high-speed Ethernet ports and you reach 405B-class models in FP4.

How the memory maths works

The figure that matters is how much model fits in memory at once. A model's footprint is roughly its parameter count multiplied by the bytes per parameter, which depends on quantisation. At 4-bit (FP4 or Q4), the standard for local deployment, you spend about half a gigabyte per billion parameters.

That maths is what makes 128 GB so useful. A 70B model at 4-bit lands around 35 to 40 GB, leaving plenty of headroom for context. A 120B model occupies roughly 60 to 70 GB. A 200B model at aggressive quantisation squeezes into 128 GB with enough room left for the context window and operating overhead. The Spark's GB10 Grace Blackwell chip delivers up to 1 petaFLOP of FP4 AI compute, and its unified LPDDR5x memory runs at up to 273 GB/s shared across CPU and GPU.

Why unified memory beats raw VRAM here

A normal GPU keeps its own VRAM separate from system RAM, so anything larger than the VRAM must be split, swapped or simply will not load. The Spark shares one 128 GB pool between the CPU and GPU coherently, which means a large model loads and runs directly without the overhead of shuffling data between system memory and dedicated VRAM.

The consumer-GPU comparison

A flagship 24 GB consumer card is fast and excellent for image generation and mid-size models, but it cannot hold a 70B model without heavy quantisation and offloading, and a 200B model is simply out of reach. The Spark trades some of that peak bandwidth for sheer capacity, which is the right trade when your goal is running the biggest model you can rather than the fastest small one. If your work is more graphics or single-large-GPU territory, the GPU best sellers show what is moving for that use case, while the Spark sits in a different category entirely.

Where the bandwidth limit shows

The honest caveat is speed. The Spark's 273 GB/s is generous for its size but far lower than a top-end discrete GPU's 1.8 TB/s, so tokens-per-second on the very largest models will feel deliberate rather than instant. Published benchmarks put the Spark at around 3 tokens per second on a 70B model at near-full precision, which is fine for prototyping and experimentation but not for interactive production serving. For local development, validation and fine-tuning of large models, that is an acceptable trade. For high-throughput production serving, it is not the tool.

How large can you actually go

On a single unit, plan around these bands at 4-bit quantisation: 70B models run very comfortably with room for long context, 120B models are a solid fit, and 200B models are the practical ceiling with tight context budgets. For fine-tuning, 70B-class models are the realistic limit on one Spark, since training requires holding the model, gradients and optimiser state simultaneously. A 70B model at BF16 full precision uses around 140 GB for inference alone; fine-tuning at that scale needs more memory than one unit carries.

Two units linked over the 200 Gb/s aggregate interconnect operate as a small cluster and push distributed inference to 405B-class models in FP4. That two-unit setup gives you 256 GB of combined unified memory, which opens up Llama 3 405B and similar frontier-class open-weight models. To see how Evetech positions these alongside other local-AI machines, the AI PC range is the place to compare.

Who this is for

The Spark suits a developer, researcher or studio that needs to keep data on-premises, iterate on large models without per-hour cloud bills, and fine-tune at the 70B level. For SA users specifically, running locally sidesteps cloud latency and data-residency questions, and it turns a recurring cloud cost into a one-time hardware decision. If your largest planned model is 13B or 30B, a strong single GPU is cheaper and faster; the Spark earns its place the moment you genuinely need 70B and up.

Frequently Asked Questions

What is the largest model a single DGX Spark can run?

For inference, roughly a 200 billion parameter model at aggressive 4-bit quantisation, because 128 GB of unified memory is the hard limit. In practice 70B and 120B models run more comfortably with full context, while 200B sits at the edge with a tighter context budget.

Can the DGX Spark fine-tune large models, not just run them?

Yes, it can fine-tune models up to about 70 billion parameters thanks to the 128 GB shared memory pool. Larger models can be loaded for inference but exceed what one unit can practically fine-tune on its own.

How does it reach 405B-parameter models?

By linking two DGX Spark units over their dual high-speed Ethernet ports at 200 Gb/s aggregate bandwidth, so they operate as a small cluster with 256 GB of combined memory. That distributed setup handles 405B-class models in FP4 that a single unit cannot hold.

Is it faster than a high-end consumer GPU?

Not in raw bandwidth or tokens-per-second on small models, where a top discrete GPU wins. The Spark's advantage is capacity: it holds models far too large for any single consumer card, trading some speed for the ability to run them at all.

Why not just use a 24GB GPU with quantisation?

A 24 GB card tops out around a quantised 30B model and cannot hold a 70B model without aggressive offloading that cripples speed. The Spark's 128 GB removes that ceiling, which is the entire reason to choose it over a single consumer GPU.