The Transformer architecture has been a breakthrough in AI. As a result, large language models (LLMs) have become increasingly powerful. They can generate human-like text, translate languages, summarize articles, answer complex questions, and even write code.
Their ability to understand and generate text with high accuracy has revolutionized the way we interact with AI.
GPUs were initially designed for 2D/3D graphics and gaming and have become the dominant high-performance hardware architecture for accelerating highly parallel workloads such as matrix multiplications (matmul).
NVIDIA's Jensen Huang believes there is a fundamental platform shift from general-purpose serial CPU computing to accelerated GPU parallel computing. He was right for training. Inference is a different animal. Decode is sequential, memory-bound, and stubbornly serial, and no amount of parallel silicon fixes that. NVIDIA has just proven the point itself by shipping a second rack of chips that are not GPUs.
The Rise of the Attention Mechanism.
In 2017, the Google Paper “Attention is All You Need” proposed a new Transformer architecture, based solely on attention mechanisms, trained on 8 NVIDIA P100 GPUs.
Recurrent Neural Networks (RNNs) process and convert a sequential data input into a specific sequential data output, one step at a time. Because RNNs read one word at a time, training is slow and cannot be parallelized. Worse, as sequences grow long, the learning signal either explodes into instability or vanishes towards zero, and the model increasingly struggles to connect distant words. In short, RNNs are slow, inefficient, and expensive to train, and become increasingly forgetful over distance.
Transformers ended the waiting and unlocked parallelism. Instead of reading one word at a time and passing notes down a slow chain, self-attention lets every word look at every other word at once, and positional encoding keeps track of the order the words came in. Attention decides what matters. Positional encoding preserves order.
That single change fixes what held RNNs back. Nothing gets forgotten over distance. The learning signal stays stable instead of exploding or fading. The payoff is a model that trains faster, connects distant words no matter how far apart, and scales massively on GPUs, which are great at parallel processing.
Freed from reading one word at a time, Transformers could now learn from huge amounts of text faster, cheaper, and better. The same architecture could now expand beyond text to images, audio, and video.
Attention has become the engine that every Transformer runs. It now powers nearly every frontier LLM, closed- and open-weight alike.
The Quadratic Scaling Wall.
Attention has one non-negotiable rule. Every token must compare itself with every other token. It is how each word figures out which others give it meaning.
Now count the comparisons. Token 1 compares with all n tokens. Token 2 compares with all n tokens, and so on. That is n tokens each doing n comparisons. That is n times n, n-squared.
Ten tokens, one hundred comparisons. One thousand tokens, a million. A hundred thousand tokens, ten billion. The work grows with the text squared.
As the context size increases, the attention weight matrix grows quadratically, meaning the number of scores that must be computed and stored grows significantly faster. This leads to higher memory usage and computational cost, making it challenging to scale transformers to very long sequences efficiently.
Such quadratic scaling eventually hits a wall. Each step forward costs more than the last. The first thousand tokens are cheap. The next thousand cost more. The ten thousandth token is punishingly expensive, and it only gets worse.
Fixes have come in three variations. Compute it smarter. Compute less of it. Or replace it. The first two focus on treating the symptoms. The third goes after the disease.
The first fix came via FlashAttention. It computes in small tiles and never keeps the giant n-by-n matrix at once. The compute stays quadratic, but memory becomes linear. That alone made long contexts practical, and it sits inside most models today.
The second fix asks a simple question. If every word does not truly need every other word, why compare them all? Sliding window attention lets each word look at its nearby neighbors. Sparse attention lets each word look at a chosen few. The cost is that a word can miss something far away, so models mix in a few full attention layers to catch the long-range stuff.
The third fix throws attention out of most layers entirely. Instead of looking back at everything, each word carries a small fixed-size memory forward. The memory does not grow as the sentence grows. So the cost is linear, not quadratic. The tradeoff is that a fixed memory can forget or blur important details, which is why pure versions still struggle at frontier scale.
The quadratic wall is real, and no fix has removed it cleanly. So, the frontier settled on hybrids for now. Cheap linear layers do most of the work. A few real attention layers stay to catch what the cheap layers miss. The frontier has settled on hybrids, a truce, not a cure.
Because the quadratic problem was never fully solved at the software architectural level, the industry has been forced to keep compensating with more expensive HBM and more GPU FLOPs at the problem, plus incremental software fixes, especially as workloads shift from training to inference and agentic use cases.
Shifting from Training to Inference to Agentic.
CPUs are built to finish one task fast. GPUs are built to finish many tasks per second. CPUs are great for serial computations, but are architecturally poorly suited for matrix multiplications, which involve numerous parallel calculations.
LLMs are mostly matrix multiplications by arithmetic. NVIDIA did not make GPUs good at matrix multiplication by accident. Matrix multiplication turned out to look exactly like the problem GPUs were already built to solve.
GPUs, which are excellent at parallel computations, have rapidly displaced CPUs in this initial AI infrastructure buildout phase. NVIDIA has been one of the biggest AI beneficiaries, as evidenced by its rapid revenue growth, largely from its Data Center segment from 2023 onwards.
The grounds beneath are shifting. AI LLM workloads started primarily with training, as models are trained, and are increasingly shifting to inference and agentic workloads. Below is how it could look directionally over time.
Splitting Inference into Prefill and Decode.
Inference workloads can be split into prefill and decode.
Prefill (prompt processing) is compute-bound. It processes the entire input prompt in parallel, builds the KV cache for attention, and is dominated by large matrix multiplications.
Decode (token generation) is memory-bound. It generates one output token at a time sequentially and reuses and grows the KV cache on every step autoregressively. It is dominated by memory reads of model weights and KV state. The bottleneck is memory bandwidth.
Each token requires reading the entire model’s weights and the full KV cache from memory, then performing relatively light arithmetic on them. The bottleneck is bandwidth, specifically the latency of getting weights and cache to the compute unit.
The GPU’s FLOPs sit idle waiting for memory. Using a GPU to decode at a batch size of 1 is a Ferrari in a parking lot: expensive silicon and memory doing almost nothing.
Continuous batching can improve throughput by interleaving many independent sequences, but it cannot eliminate the fundamentally low arithmetic intensity of decode, especially for latency-sensitive or single-stream workloads.
Agentic workloads compound this issue further. The industry’s software response is to run both phases on the same chip and optimize for throughput. But the issue remains. One cannot batch your way out of a workload that is inherently sequential. Decode looks like a compute problem but behaves like a memory problem.
AI inference is no longer a single workload that can be served efficiently by a single type of accelerator or memory. AI inference is a dual-stage workload, with very different resource demands for each. The same knife should not be used to cut everything.
The Shortcomings of HBM.
GPUs address this hardware architecture flaw with fast, expensive HBM stacked next to the compute die. Stacking to increase capacity causes thermal, yield, and capacity issues. DRAM chips generate heat, and it is becoming more difficult to remove it effectively.
Each generation of HBM just keeps adding more layers, bandwidth, and pin count. There is a strict JEDEC stack height (currently 775 µm for HBM4). More layers force memory manufacturers to make each DRAM die thinner. Thinner dies are more fragile, warp more easily, and lower manufacturing yield. The industry continues to push the limit higher by relaxing the JEDEC height limit, using thinner dies, hybrid bonding with bump-less direct-to-copper, and better cooling solutions.
There are physical limits to how much thinner DRAM can be shaved and how much higher the shaved layers can be stacked vertically with TSVs (Through-Silicon Vias) and micro-bumps to make HBM.
Yield is also becoming difficult to achieve as layer counts increase. For a given stack yield, total yield decreases as layer count increases.
Capacity-wise, HBM is incredibly inefficient due to HBM’s 3D architecture and larger die size, as it typically consumes 3x more silicon during the manufacturing process as compared to standard DRAM. This is known as the “HBM trade ratio”, and this is expected to rise from 3:1 to 4:1 going into HBM4. Making HBM only pulls capacity out of the DRAM supply chain.
HBM has been scaling slower than compute for several generations now. The memory wall on the hardware side is real. The industry’s response is to keep coming up with new software innovations to incrementally improve, but all it has done is bought time, not escape.
NVIDIA is using HBM to compensate for the flaws of using GPUs for Inference.
Each NVIDIA GPU generation keeps adding more and more HBM memory by increasing gigabits per layer, layers per stack, gigabytes per stack, and HBM stacks. Yet for most of the last decade, compute FLOPs outran bandwidth. Rubin improves the balance by running HBM4 near the top of its range across eight stacks. NVIDIA still built a separate SRAM rack, Groq 3 LPX, to hit low-latency decode targets. HBM was NVIDIA's answer. Rubin shows it was not enough.
NVIDIA GPUs use the most advanced HBMs, and NVIDIA’s share of HBM memory has accelerated through 2025 with the B200 and B300 to ~75% by 4Q25 and should remain strong in 2026.
While more customers are signing 3-5 year long-term agreements (LTAs) for HBM with the three DRAM memory makers (SK Hynix, Samsung, and Micron), no memory maker has publicly disclosed a multi-year HBM LTA with NVIDIA. Because NVIDIA is the largest buyer of HBM, it has the largest bargaining power (i.e., monopsony).
If your biggest customer does not sign an LTA, the argument for LTAs making HBM memory less commoditized should fall flat. Despite LTAs now having prepayments, etc, they remain largely volume-based, with pricing tending to be more floating than fixed, and are signed by many smaller customers with lower bargaining power.
Inference compute remains memory-bound on decode and batch size one. While HBM is a lousy memory, unfortunately, it is the best solution we have right now that can address this structural memory wall problem.
In short, HBM is bit inefficient, power inefficient, and bandwidth inefficient. The memory makers likely know it too and recognize that HBM is not the final answer and will not solve the memory wall problem.
Watch Dr. Hoshik Kim, SVP, SK Hynix, agree with ex-Intel CEO Pat Gelsinger that HBM is a lousy memory and is not the final answer to the memory wall problem.
NVIDIA, being the largest supplier of AI infrastructure, leads the entire stack. It sets the roadmap that memory vendors have to chase. It has a long habit of engineering its way out of its own constraints rather than being disrupted by them.
The memory hierarchy of using more and better HBM to compensate for the GPU’s structural inefficiency in decode is structurally using bad hardware to fix bad software.
HBM itself is a bad memory, but it is just the best solution we got right now. In turn, the memory bottleneck has allowed memory players to price-gorge, profit-maxx, and hit record profitability.
The solution from first principles? Instead of coming up with half-baked software and corresponding hardware solutions. It cannot just be step-wise efficiency gains (e.g., 1-5X better). It has to be significantly transformational (e.g., 10X, 20X, 50X, 100X better) instead, with the rise of new software and hardware architectures.
Disaggregating Inference by Hardware.
The next best solution is to redesign the hardware to better address the inherent software flaws. One way that is increasingly being adopted by the industry is to disaggregate the inference compute workload of prefill, decode, and orchestration/execution.
For hardware, do not choose one chip for everything (i.e., GPUs). Disaggregate and split execution by physics.
While GPUs have been great for training, they are increasingly poorly suited for inference, especially on decode. If inference hardware becomes increasingly disaggregated into prefill and decode and is even less suited for agentic AI, where workloads are dominated by orchestration, GPUs could become less dominant.
Prefill on compute-dense silicon (i.e., NVIDIA and AMD GPUs, custom ASICs). Decode on bandwidth-dense silicon (i.e., Groq LPU, Cerebras WSE, SambaNova RDU, etc). Then orchestrate both from a CPU (i.e., NVIDIA, Intel, AMD, Graviton CPUs) and carry the KV cache on the fabric.
It makes sense to further disaggregate the AI inference hardware stack. Numerous launches to disaggregate prefill and decode have been announced over the last few months by the majors, including NVIDIA, AWS, Intel, AMD, with Groq, Cerebras, and SambaNova for decode-specialized chips.
NVIDIA ran the inference disaggregation experiment on both legs, and only one worked. In September 2025, it announced Rubin CPX, a prefill chip carrying GDDR7 DRAM instead of HBM, on the logic that compute-bound work does not need expensive bandwidth. Six months later, CPX was absent from the GTC 2026 roadmap, with NVIDIA’s Ian Buck reported as saying the concept moves to Feynman, and the redesign putting HBM back in.
The decode leg went the other way. NVIDIA’s Groq 3 LPX shipped as a separate rack of SRAM chips beside Vera Rubin. Cheap memory for prefill was tried and shelved. SRAM for decode was tried and shipped. NVIDIA keeps attention on the GPU and sends only the feed-forward work to the LPUs. Only the decode weights move to SRAM. Prefill and the KV cache stay on HBM.
If this growing hardware disaggregation takes flight, we could have a structural reduction in HBM memory demand. As AI storage evolves, it could further broaden into different storage hierarchies optimized for each workload and across different chips.
There exist many types of memory because of the speed/capacity/cost trilemma in memory. In short, fast memory is small and expensive, while cheap memory is slow and large.
The physical structure of memory determines this trade-off. Semiconductor design works around this physical limitation by stacking different types of memory in a hierarchy. Faster, smaller memory sits closer to the processor, while slower, larger memory sits further away.
Memory can also be thought of in another spectrum: short-term versus long-term memory, best illustrated in the memory hierarchy pyramid below.
The lower it is, the more like long-term memory it is. NAND memory can hold a lot of data (i.e., remembers), but is slower to retrieve and is cheaper. The higher it is, like SRAM and HBM, the more it is like short-term memory (i.e., forgets). It holds less data, is faster, but more expensive.
Concluding Thoughts.
As inference compute disaggregates, memory demand broadens beyond HBM. Weight traffic in decode moves to SRAM. Prefill stays on HBM. So does the KV cache, and it grows with context. HBM intensity per token falls. Absolute HBM demand likely keeps rising for years, just more slowly than before.
Not all thoughts need HBM.
The deeper problem remains upstream. Attention scales quadratically, and nobody has fixed it. Everything downstream is compensation. More HBM, more FLOPs, more racks, more energy, more capital. We need a fundamentally different software architecture first. Without it, we keep buying expensive hardware to paper over a flaw in the software.
This will take time. New hardware needs an ecosystem, and ecosystems hold until they give. But the direction is set. The memory makers are earning record margins solving a problem the industry has already started redesigning around. Cycles end when the thing being paid for stops being the only option.
The information contained in this article is for general informational purposes only. While every effort has been made to ensure the accuracy and reliability of the content, errors or omissions may occur. The author and publisher do not assume any responsibility or liability for any errors, inaccuracies, or omissions in the content, nor for any actions taken based on the information provided. Readers are encouraged to verify any information before relying on it and to seek professional advice as needed.
3 Aug 2026 | Eugene Ng | Vision Capital Fund | eugene.ng@visioncapitalfund.co
Find out more about Vision Capital Fund.
You can read our annual and quarterly letters for Vision Capital Fund.
Check out our book on Investing, Vision Investing: How We Beat Wall Street & You Can, Too. We believe that individual investors can outperform the market over the long term. The book chronicles our entire investment approach. It explains why we invest the way we do, how we invest, what we look for in companies, where we find them, and when we invest. Available via Amazon: paperback and eBook.
Join our email list for more investing insights. Our emails tend to be ad hoc and infrequent, as we aim to write timeless, not timely, content.