HPIM: Heterogeneous Processing-In-Memory-based Accelerator for Large Language Models Inference

👏 Paper title: HPIM: Heterogeneous Processing-In-Memory-based Accelerator for Large Language Models Inference.

HPIM is a memory-centric heterogeneous Processing-In-Memory (PIM) accelerator designed to address the memory bandwidth bottlenecks and strict latency constraints in large language model (LLM) inference, particularly during the memory-bound autoregressive decoding stage.

The framework employs a software-hardware co-design strategy that integrates two complementary PIM subsystems: SRAM-PIM and HBM-PIM. Latency-critical attention operations and complex non-linear functions are dispatched to the SRAM-PIM subsystem to exploit its ultra-low latency and computational flexibility, while weight-intensive GEMV computations are assigned to the HBM-PIM subsystem to leverage its large capacity and massive internal bandwidth.

HPIM introduces a specialized compiler framework featuring stage-specific operator mapping and a hybrid tiling strategy (combining head-wise and tensor-wise parallelism) to balance compute loads and minimize inter-subsystem data movement. Furthermore, a tightly coupled pipelining mechanism between SRAM-PIM and HBM-PIM maximizes intra-token parallelism to overcome the serial execution bottleneck inherent in single-batch LLM inference.

Evaluated on cycle-accurate simulations across OPT family models (350M to 30B), HPIM achieves substantial speedups over conventional GPU architectures (such as NVIDIA A100) and outperforms state-of-the-art homogeneous PIM accelerators, presenting a scalable and energy-efficient solution for large-scale LLM deployment.