CAGR: A Cross-Accelerator Graph Optimization Framework for Efficient Recommender System Inference
Zijian Shen, Wenyu Zhao, B. Wang, Zimeng Wang, Wenbin Shang
Abstract
Recommender systems have become ubiquitous in modern online services, yet their deployment across diverse hardware accelerators remains challenging due to significant performance variations. Contemporary deep learning recommendation models (DLRMs), such as DeepFM and NGCF, exhibit substantial inference latency differences when executed on NVIDIA GPUs,AMDGPUs, and Google TPUs, primarily due to architectural disparities and vendor-specific optimization strategies. Existing graph optimization frameworks are typically designed for specific hardware backends, lacking the flexibility to generate portable high-performance implementations across heterogeneous accelerators. This paper presents CAGR (Cross-Accelerator Graph Rewriting), a novel framework that achieves performance-portable inference optimization for recommendation models through three key innovations: (1) a hardware-aware graph rewriting engine that dynamically selects optimal operator implementations by analyzing compute-to-memory bandwidth ratios and operator density characteristics; (2) a reinforcement learning-based transformation policy that learns cross-platform optimization strategies without exhaustive search; and (3) a heterogeneous pipeline architecture enabling “optimize once, deploy across supported backends” semantics. We implement CAGR with support for multiple kernel backends including Triton, cuBLAS, MIOpen, and oneDNN, and demonstrate its effectiveness on the Avazu CTR prediction dataset. Experimental results show that CAGR achieves 1.8-3.2× speedup over baseline implementations across NVIDIA V100, AMD MI100, and Google TPU v3 platforms, while reducing optimization time by 67% compared to platform-specific auto-tuning approaches. Furthermore, CAGR maintains 92-96% of reference optimized performance with zero manual intervention, demonstrating practical viability for production recommendation systems requiring multi-vendor deployment.