On limits of LLMs in adaptation of ensemble-based architectures

Recent developments of Large Language Models (LLMs) show great potential in many areas, including software architecture. Although there is some work on applying LLMs during architecture design, using LLMs for adaptation of the architecture of a collective adaptive system at runtime has not yet been explored enough. In this paper, we explore two approaches for how LLMs can serve for architecture adaptation of collective adaptive systems based on autonomic component ensembles. The first approach employs an LLM during runtime as a part of the adaptation manager; the other one asks the LLM to generate it (in Python), which is then used for the ensemble formation (resolution) at runtime. The prompts for both approaches are automatically generated from an architectural specification, which includes constraints for the architecture. Based on experimental observations of two use cases, we show that LLMs are quite capable in online prompting in particular. Even without being explicitly provided with an adaptation strategy, an LLM can come up with an efficient heuristic for ensemble resolution and realize it. We show that the limiting factor for using LLMs this way is not time complexity (as would be the case when solving the problem as constraint optimization), but the “laziness” of LLMs when prompted with a larger problem instance, as also recently reported in other works. In addition, we map how the correctness of the LLM’s solution scales with different forms of prompting and problem size, which captures the effect of the LLMs’ laziness under different conditions.