Journal article : Letter
Hardware-managed heterogeneous high-bandwidth memory and flash in LLM inference systems
- Abstract:
- Large Language Models (LLMs) have become the driver of many modern applications, but their parameter counts have outgrown the memory capacity of a single GPU. Because GPUs rely on High-Bandwidth Memory (HBM), whose capacity is approaching a fundamental ceiling, fitting today’s models often requires aggregating several accelerators, a cost that places edge deployment out of reach for many users. High-Bandwidth Flash (HBF) offers a denser alternative, providing 16x more capacity per stack at comparable bandwidth. In this work, we show that while replacing HBM with HBF can address the capacity problem, doing so naively severely impacts performance due to HBF’s long tail memory latency starving GPU schedulers. To address this, we propose a Heterogeneous Memory Architecture (HMA) that combines HBM and HBF through a prediction-based migration policy to keep high latency HBF off the GPU’s critical path. We demonstrate that the HMA achieves geo-mean 2.79x improved performance over an HBF Only baseline, enabling the efficient execution of 9.2x larger models on a single edge GPU.
- Publication status:
- Published
- Peer review status:
- Peer reviewed
Actions
Access Document
- Files:
-
-
(Preview, Accepted manuscript, pdf, 575.1KB, Terms of use)
-
- Publisher copy:
- 10.1109/LCA.2026.3723326
Authors
+ Advanced Research + Invention Agency
More from this funder
- Funder identifier:
- https://ror.org/028y4g372
- Grant:
- NACB-PR01-P04
- Publisher:
- IEEE
- Journal:
- IEEE Computer Architecture Letters More from this journal
- Publication date:
- 2026-08-12
- Acceptance date:
- 2026-08-04
- DOI:
- EISSN:
-
1556-6064
- ISSN:
-
1556-6056
- Language:
-
English
- Keywords:
- Subtype:
-
Letter
- Pubs id:
-
2450025
- Local pid:
-
pubs:2450025
- Deposit date:
-
2026-08-11
- ARK identifier:
Terms of use
- Copyright holder:
- IEEE
- Copyright date:
- 2026
- Rights statement:
- © 2026, IEEE
- Notes:
- The author accepted manuscript (AAM) of this paper has been made available under the University of Oxford's Open Access Publications Policy, and a CC BY public copyright licence has been applied.
- Licence:
- CC Attribution (CC BY)
If you are the owner of this record, you can report an update to it here: Report update to this record