MAD-OGRE: A Multi-Agent Debate Framework for Open Geographic Entity Relation Extraction Based on Large Language Models

# MAD-OGRE: A Multi-Agent Debate Framework for Open Geographic Entity Relation Extraction Based on Large Language Models ## 🧩 Core Logic The core code of this project is located in the `src/` directory, adopting a design pattern that separates **Roles** and **Actions**: ### 1. Core Modules * **`src/role/`**: Defines agent attributes, Prompt templates, and interaction logic. * **`src/action/`**: Implements specific business logic, API calls, and data processing. ### 2. Four Structural Patterns To address the complexity of geographic text, four parallel processing modules are designed: * **Independent Parallel**: Handles simple sentences or coordinate sentences with independent semantics. * **Progressive**: Handles complex sentences involving causality, chronological order, or hierarchical reasoning. * **One-to-Many**: Handles relationships where one head entity corresponds to multiple tail entities. * **Many-to-One**: Handles aggregate relationships where multiple head entities point to the same tail entity. ### 3. Multi-Agent Debate Pipeline Each structure contains a complete closed-loop debate process. Overview of the core workflow: 1. **Basic Extraction**: Entity recognition and preliminary relation extraction. 2. **Debate Optimization**: * Advisory Component (`_Three_advice`): Generates diverse evaluation viewpoints; * Core Debate (`_Debate`): Conducts multi-round interactions; * Decision Module (`_Decide`): Determines and outputs the final result. > Note: Since the core framework process is highly consistent across different sentence patterns, `###` is used below as a placeholder for the sentence pattern to illustrate the structure. Agent Role Definitions (Implementation Files) * `src/role/conventional_template/Extract_entityer.py`: General Entity Extraction Agent * `src/role/###/###_Cleaning.py`: Cleaning Agent * `src/role/###/###_Extract_Relationer.py`: Relation Extraction & Triple Generation Agent * `src/role/###/###_ThreeAdvicer.py`: Evaluation Opinion Generation (Advisor) Agent * `src/role/###/###_Debator.py`: Debate Agent * `src/role/###/###_Decider.py`: Decision Agent * `src/role/###/###_Summarizer.py`: Summarization & Correction Agent Agent Actions (Corresponding Implementations) * `src/action/conventional/Extract_entity.py`: General Entity Extraction Action * `src/action/###/###_Clean.py`: Cleaning Action * `src/action/###/###_Extract_Relationship.py`: Relation Extraction & Triple Generation Action * `src/action/###/###_Three_advice.py`: Evaluation Opinion Generation Action * `src/action/###/###_Debate.py`: Debate Action * `src/action/###/###_Decide.py`: Decision Action * `src/action/###/###_Summarize.py`: Summarization & Correction Action ## ⚙️ Experimental Setup To ensure the reproducibility of experiments, the default parameters used in the paper are as follows (unless overridden in specific experiment scripts): ### Triple Extraction Experiment Parameters | Parameter | Value | Description | | --- | --- | --- | | **Model Version** | `Qwen2.5-14B-Instruct` | The model version used in the main experiments of the paper | | **Temperature** | `0.0` | Set to 0 to ensure determinism and consistency of results | | **Top_p** | `1.0` | Nucleus sampling parameter | | **Max Tokens** | `8192` | Maximum output length limit | | **Debate Rounds** | `3` | Default debate rounds (varies in Debate Rounds experiments) | > **Note:** The LLM parameters mentioned above are mainly managed in `config/`, while the number of debate rounds is controlled by the `DEBATE_ROUNDS` variable in different experiments. ### LLM Parameters for Experimental Result Evaluation | Parameter | Value | Description | | --- | --- | --- | | **Model Version** | `Qwen3-235B-A22B-Instruct` | The model version used in the main experiments of the paper | | **Temperature** | `0.7` | Set to 0.7, as recommended by Qwen official documentation | | **Top_p** | `0.8` | Nucleus sampling parameter, recommended by Qwen official | | **Max Tokens** | `8192` | Maximum output length limit | | **presence_penalty** | `1.5` | Recommended by Qwen official | | **top_k** | `20` | Recommended by Qwen official | ### LLM Parameters in Comparison of Reasoning Capabilities: GRE vs. Open GRE Experiment | Parameter | Value | Description | | --- | --- | --- | | **Model Version** | `Qwen2.5-14B-Instruct` | The model version used in the main experiments of the paper | | **Temperature** | `0.0` | Set to 0 to ensure determinism and consistency of results | | **Top_p** | `1.0` | Nucleus sampling parameter | | **Max Tokens** | `8192` | Maximum output length limit | ## 🛠️ Installation This project supports modern Python environment management. Using `uv` is recommended for faster installation speed. ### Prerequisites * Python 3.11 or higher * An OpenAI-compatible API Key * Configured `uv` environment Since the project includes `uv.lock`, strict environment synchronization can be performed: ```bash uv sync ``` ### 🔑 Configuration Before running experiments, LLM settings must be configured according to the `metagpt` package requirements. 1. Create a `config` folder in the project root directory. 2. Create a `config2.yaml` file inside `config/`. 3. Copy the following content and fill in your actual information: ```yaml # config/config2.yaml llm: api_type: 'openai' # Optional: openai, azure, ollama, etc. api_key: 'sk-...' # Replace with your actual API Key model: 'Qwen2.5-14B-Instruct' # Recommended: Qwen2.5-14B-Instruct base_url: '...' # Optional: Custom endpoint URL # ...... # See metagpt official site for other configurations ``` ## 📊 Data Preparation and Format ### Dataset Description * **`data/CE-GRE-Eval/`**: Self-constructed OpenGRE dataset, containing geomorphic entities, processes, and causal relationships. * **`data/DuIE/`**: DuIE benchmark dataset, used for generalization capability testing. * **`data/QA/`**: Geographic knowledge QA dataset, used to test the impact of GRE and OpenGRE on LLM reasoning capabilities. ### Dataset Field Description | Field Name | Explanation | | --- | --- | | **content_source Version** | `Original text information` | | **triples** | `List of annotated triples information` | | **head_entity_name** | `Head Entity` | | **relationship_name** | `Relation` | | **tail_entity_name** | `Tail Entity` | | **sentence_pattern_classification** | `Sentence pattern type, one of: Independent_Parallel_Structure, Progressive_Structure, One_to_Many_Structure, Many_to_One_Structure` | | **question** | `Input question` | | **GRE** | `Triples extracted by GRE` | | **OpenGRE** | `Triples extracted by OpenGRE` | ### Data Format Examples `DuIE` input data typically uses JSON format, containing original text and (optional) annotated ground truth: ```json [ { "content_source" : "六春湖，又名绿葱湖，位于浙江龙游县庙下乡境内，绿葱湖是龙游最高峰，海拔1390.5米，方圆7平方公里，山势雄伟壮观，周围峰峦叠嶂，终年绿草如茵" , "triples" : [ { "head_entity_name" : "六春湖" , "relationship_name" : "海拔" , "tail_entity_name" : "1390.5米" } ] , "sentence_pattern_classification" : "简单句" } ] ``` `CE-GRE-Eval` input data typically uses JSON format, containing original text and (optional) annotated ground truth: ```json [ { "content_source" : "冰川融水中携带的碎屑物在消融季节沉积并压实成松散的冰碛物，再由冰碛物在反复冻融和压密过程中形成稳定的冰碛堆积体。" , "triples" : [ { "head_entity_name" : "冰川融水中携带的碎屑物" , "relationship_name" : "在消融季节沉积并压实" , "tail_entity_name" : "松散的冰碛物" } , { "head_entity_name" : "松散的冰碛物" , "relationship_name" : "在反复冻融和压密过程中形成" , "tail_entity_name" : "稳定的冰碛堆积体" } ] , "sentence_pattern_classification" : "递进句" } ] ``` `QA` input data typically uses JSON format, containing the question, reference original text for the question, GRE triples, and OpenGRE triples: ```json [ { "question"