The Breakthrough Memory Solutions for Improved Performance on LLM Inference
Byeongho Kim, Sang-Hoon Cha, Sangsoo Park, Jieun Lee, Sukhan Lee, Shin-haeng Kang, Jinin So, Kyung-Soo Kim, Jin Chul Jung, Jong-Geon Lee, Sunjung Lee, Yoonah Paik, Hyeonsu Kim, Jinseong Kim, Won-Jo Lee, Yuhwan Ro, Yeongon Cho, Jin Hyun Kim, Joonho Song, Jaehoon Yu, Seungwon Lee, Jeong‐Hyeon Cho, Kyomin Sohn
Abstract
Large Language Model (LLM) changes our lives, while it requires unprecedented computing resources, especially it requires large memory capacity and high bandwidth to process weights. However, while the logic process was developing, the speed of development of the memory process could not keep up, causing problems that resulted in the performance of LLM being hindered by memory. Samsung have introduced breakthrough Processing-in-Memory/Processing-near-Memory (PIM/PNM) solutions that enhance the main memory bandwidth. With the HBM-PIM-based GPU-cluster system and LPDDR5-PIM-based system, the performance of transformer-based LLMs improved by up to 1.9× and 2.7×, respectively. The CXL-based PNM solution serves memory-centric computing systems by implementing logic inside the CXL memory controller. This results in a performance gain of over 4.4× with an energy reduction of about 53% with PNM. Furthermore, we provide PIM/PNM software stacks, including an AI compiler targeting the acceleration of AI models.