Pre-Conference Talk by SHI Jieke | Green Large Language Models of Code

Green Large Language Models of Code

Large language models of code have shown remarkable effectiveness in various software engineering tasks. Despite the availability of many cloud services built upon these powerful models, there remain several scenarios where developers cannot take full advantage of them, stemming from factors such as restricted or unreliable internet access, institutional privacy policies that prohibit external transmission of code to third-party vendors, and more. Therefore, developing a compact, efficient, and yet energy-saving model for deployment on developers' devices becomes essential. To this aim, we propose Avatar, a novel approach that crafts a deployable model from a large language model of code by optimizing it in terms of model size, inference latency, energy consumption, and carbon footprint while maintaining a comparable level of effectiveness. The key idea of Avatar is to formulate the optimization of language models as a multi-objective configuration tuning problem and solve it using a Satisfiability Modulo Theories (SMT) solver and a tailored optimization algorithm. We use Avatar to produce optimized language models of code with a small size (3 MB, 160× smaller). The optimized models significantly reduce energy consumption (up to 184× less), carbon footprint (up to 157× less), and inference latency (up to 76× faster), with only a negligible loss in effectiveness (1.67% on average). 

This is a Pre-Conference talk for 46th International Conference on Software Engineering (ICSE 2024).