14.4 A 51.6TFLOPs/W Full-Datapath CIM Macro Approaching Sparsity Bound and <2^-30 Loss for Compound AI

Abstract

Large-language models (LLM) are widely used with exceptional performance, but their prohibitive size and cost limits deployment on edge devices. The compound-AI combines several specialized small models to achieve matched or even superior accuracy on target downstream tasks [1], [2]: e.g., RevCol [3] fuses multiple convolutional models and surpasses the 7.2B-parameter monolithic LLM model [4] on the ImageNet classification task by 1.64%. Therefore, the shift to compound systems opens opportunities for edge deployment in addition to model-size scaling. SRAM-based floating point (FP) CIM promises accelerated edge-AI models [5]–[14]. Figure 14.4.1 shows a conventional FP CIM macro, which faces three challenges for compound-AI acceleration: (1) Previous FP CIMs rely on a Gaussian data distribution to reduce alignment loss, where data is distributed around its mean [11]. However, compound-AI employs diverse models, with multiple data distributions, leading to an accuracy degradation of more than 5% [15]. (2) In variable compound-AIs, the CIM only performs MAC operations [16]–[23]; however, partial-sum/residual accumulation requires data movement and accounts for 26-72% power consumption, especially for bit-serial CIMs. (3) Conventional CIMs explore sparsity according to specific model features [11], [12], but compound-AI models have flexible sparse patterns, causing a gap between the actual and theoretical speedup (1/1-sparsity).

Topics & Concepts