Creating Edge AI from Cloud-based LLMs
Qifei Dong, Xiangliang Chen, Mahadev Satyanarayanan
Abstract
Cyber-human and cyber-physical systems have tight end-to-end latency bounds, typically on the order of a few tens of milliseconds. In contrast, cloud-based large-language models (LLMs) have end-to-end latencies that are two to three orders of magnitude larger. This paper shows how to bridge this large gap by using LLMs as offline compilers for creating task-specific code that avoids LLM accesses. We provide three case studies as proofs of concept, and discuss the challenges in generalizing this technique to broader uses.
Topics & Concepts
CompilerComputer scienceCloud computingBridge (graph theory)Enhanced Data Rates for GSM EvolutionMathematical proofLatency (audio)Computer securityArtificial intelligenceProgramming languageTelecommunicationsOperating systemInternal medicineGeometryMedicineMathematicsIoT and Edge/Fog ComputingContext-Aware Activity Recognition SystemsSoftware System Performance and Reliability