TL;DR
Thorsten Meyer AI’s Part 7 of its Memory Squeeze series prices the 2026 local-inference alternative to renting cloud GPUs. The report says the main cost driver is whether a model fits in VRAM: fit yields fast output, while spilling into system RAM can sharply reduce speed. Prices and token-per-second figures are late-June 2026, community-sourced estimates, so buyer value could change quickly.
Thorsten Meyer AI has published Part 7 of its Memory Squeeze series, a 2026 cost analysis saying the price of running AI models locally is set less by raw GPU compute than by whether model weights fit in VRAM. The analysis matters for developers, researchers and small companies comparing owned inference hardware with cloud GPU rental.
The central claim is the VRAM cliff: when a model fits entirely in GPU video memory, inference can be fast; when weights spill into system RAM, speed can collapse. Thorsten Meyer AI cites community benchmarks showing an RTX 5090 running a 70B model in VRAM at about 40-50 tokens per second, while the same setup with partial system-memory spill falls to about 1-2 tokens per second.
The report says LLM inference is memory-bandwidth-bound, which makes VRAM capacity the build limit and reduces the relevance of CUDA core counts or teraflop ratings for this workload. Its sizing map, based on Q4 quantization, puts 7-8B models at about 6-8GB, 26-32B models near 18-20GB, 70B models near 43GB and 100B-plus systems at 60-130GB or more.
The cost example at the center of the piece is the used RTX 3090 24GB, listed at about $600-$850 in late June 2026. Thorsten Meyer AI says that card delivers roughly five times the VRAM per dollar of an RTX 5090, while four used 3090s would provide 96GB of pooled memory for under about $3,200, subject to used-hardware risk and software limits.
The real cost of a local-inference rig
Owning beats renting for steady AI work — so what does a local rig cost in 2026? The unintuitive, good news: the most expensive build is almost never the smartest one. It all comes down to one rule.
The difference is only whether the weights fit. LLM inference is memory-bandwidth-bound — VRAM capacity is the hard limit you build around. Compute specs are mostly noise.
The squeeze reframes the rig like everything else in this series: discipline beats maximalism. VRAM is exactly the memory under most pressure, so over-buying it is the 128GB-“to-be-safe” trap, only worse per gigabyte. Take the cheap, high-value step to 24GB (the gateway to the 30B class), reach for used 3090s and MoE models, and use quantization to climb a tier without buying silicon. Sized right, the rig pays for itself against the cloud’s ever-rising hidden bill. Next: Apple Silicon’s quiet memory advantage.
VRAM Costs Shape AI Budgets
The analysis changes the local-AI buying question from which card is fastest to which memory tier fits the work. For steady, high-use inference, Thorsten Meyer AI says ownership can beat renting, but only if the rig is sized for the model class actually being run rather than for a vague safety margin.
That matters because the most visible hardware choice may not be the best value. A disciplined buyer running 7-14B models may need only a modern 16GB card, while a 30B-class workload can fit on one 24GB card and a 70B workload may call for dual GPUs, a 32GB flagship card, or a large-memory Mac. The financial risk is overbuying VRAM that sits idle or underbuying and hitting the speed collapse.
NVIDIA GeForce RTX 3090 Founders Edition Graphics Card (Renewed)
- Package Dimensions: 15.0 x 12.25 x 4.25 inches
- Package Weight: 6 pounds
- Package Quantity: 1
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Cloud Bills Set Up Comparison
The new installment follows a prior chapter that argued cloud rental hides part of the bill for frequent AI use. This chapter prices the alternative: local inference for users who want lower recurring cost, more control over prompts and models, or hardware they own.
The technical backdrop is quantization. The report says FP16 models need about 2GB per billion parameters, while Q8 can roughly halve that and Q4 can cut it to about one quarter. It also points to Mixture-of-Experts models, such as Qwen3 variants, as a way to get higher apparent model quality while activating fewer parameters per token.
“Everything about a local-inference build reduces to a single, unforgiving rule.”
— Thorsten Meyer AI, in Part 7 of The Memory Squeeze
Prices Could Move Fast
Several figures remain fluid. The price range for used RTX 3090 cards, the relative value of the RTX 5090, electricity costs, cooling needs and resale values can shift quickly. The token-per-second data is attributed to community benchmarks, not a single standardized lab test, and results may vary by model file, quantization level, driver stack and multi-GPU setup.
It is also not yet clear how many buyers can rely on pooled VRAM in practice. Multi-card rigs can be more complex than a single GPU or a Mac with large unified memory, and used cards may carry warranty and wear concerns.
Apple Silicon Gets Next Review
The series is set to move next to Apple Silicon’s unified-memory advantage, a comparison that could matter for buyers deciding between multi-GPU PCs and large-memory Macs. For readers pricing a rig now, the next step is to match the target model class, expected utilization and full power-and-cooling cost against current cloud spend.
Key Questions
What is the main finding of the local-inference rig analysis?
The report says VRAM fit is the main cost driver. A model that fits entirely in GPU memory can run quickly, while system RAM spill can make the same setup much slower.
Is an RTX 5090 always the best local AI card in 2026?
No. Thorsten Meyer AI argues that a used RTX 3090 24GB may be better value for many inference buyers because VRAM per dollar can matter more than newest-generation speed. The tradeoff is used-card risk and more complex builds if several cards are used.
How much VRAM does a 70B model need?
The article gives a 70B Q4 model estimate of about 43GB. That places it above a single 24GB card and into dual-GPU, 32GB-plus GPU or large-unified-memory territory, depending on quality and speed targets.
Are the listed GPU prices fixed?
No. The article labels its prices as late June 2026 point-in-time estimates. GPU resale prices, supply, power rates and cloud pricing can move, so the purchase case needs a fresh check before buying.
Does local hardware always beat cloud rental?
No. The report’s ownership argument applies most clearly to steady, high-utilization inference. Occasional workloads, very large models or teams without hardware support may still find cloud rental simpler or cheaper.
Source: Thorsten Meyer AI
