Towards General Agents with Multimodal Large Language Models

The goal of our research is to explore ways to develop general agents that can solve general tasks in a human-like manner. An ideal general agent must possess the following key features: (1) World Understanding: The agent should have a human-like perception and understanding of the world. It should be able to process multimodal information, including text, images, and audio, and interact with the environment naturally. Furthermore, the agent should possess human-like common sense and a broad knowledge base across various domains, enabling it to handle everyday tasks effectively. (2) Rational Thinking: The agent should be capable of thorough planning and basic reasoning, allowing it to approach problem-solving in a manner similar to humans. This includes the ability to verify the outcomes of its actions and to invoke all available tools in a coordinated and planned fashion, either through a unified interface or by leveraging its perception capabilities to interact with external tools. (3) Self-Improvement: The agent should be able to learn from its past experiences, either by observing human behavior, which is similar to imitation learning or by exploring and identifying patterns on its own, which is similar to reinforcement learning. This would allow the agent to adjust its internal parameters, expand its knowledge, and improve its performance over time, akin to how humans learn and adapt. In summary, our research will concentrate on three key features, world understanding, rational thinking, and self-improvement, to explore novel approaches for enhancing the capabilities of current agents. With these efforts, we aim to make significant strides toward realizing general agents that can operate autonomously and effectively in real-world tasks.