Neural Bandit Based Optimal LLM Selection for Pipeline of Tasks

There has been a growing interest in strategies to optimize the cost-performance tradeoff of LLMs, in particular LLM selection, where the aim is to understand which language models perform better than others for a given task. However, there arises some applications where simply finding the LLM which performs the best on the task is not sufficient as the task may be too specialized and difficult for an LLM to handle alone. In such cases considering a pipeline of LLMs seems to be more suitable to decrease the difficulty of the task by breaking it down into smaller subtasks where an LLM's output is fed as the input to another LLM. The task of suggesting diagnosis for patients based on provided medical reports of patients which are lengthy and include medical language is such an application. We believe that for such a task, the LLMs can benefit from being given a summarized version of the medical report (summary obtained through another LLM) which highlights the key points and necessary information which can guide diagnosis decisions.