Towards End-to-End Optimization of LLM-based Applications with Ayo
Xin Tan, Yimin Jiang, Yitao Yang, Hong Xu
Abstract
Large language model (LLM)-based applications consist of both LLM and non-LLM components, each contributing to the end-to-end latency. Despite great efforts to optimize LLM inference, end-to-end workflow optimization has been overlooked. Existing frameworks employ coarse-grained orchestration with task modules, which confines optimizations to within each module and yields suboptimal scheduling decisions.
Topics & Concepts
End-to-end principleComputer scienceDead endMathematicsArtificial intelligenceFlow (mathematics)GeometryAdvanced Data Storage TechnologiesParallel Computing and Optimization TechniquesSoftware Testing and Debugging Techniques