Long-context understanding and efficiency in natural language processing

Current NLP systems have achieved increasingly strong performance on short texts. However, their performance on long-form inputs remains significantly lower, due to challenges such as limited context length, degraded comprehension over extended input, and high computational costs. This thesis addresses these ore challenges in long-context processing through four complementary understanding and computational efficiency. First, we propose an approach for document-level multi-event extraction with event proxy nodes and Hausdorff distance minimization. Event proxy nodes serve as representations of events, and can connect with one another to capture global contextual information, providing a graph-structured view of document content. Motivated by the benefits of explicit structure, second, we propose a framework for Table of Contents extraction, consisting of three steps: (1) constructing an initial tree based on reading order and font sizes, (2) enriching each tree node with context information, and (3) modifying the tree via node-specific actions (Keep, Delete, or Move). This results in a hierarchical structure that complements event graphs and organizes long documents to benefit from both local and long-distance contextual cues for downstream use. Building on these structural representations, third, we propose PECAN, a method for long-context question-Answering (QA). PECAN uses Large Language Models (LLMs) to dynamically control the retrieval process, adjusting the retrieved scope based on different queries to achieve a better balance between accuracy and efficiency. It introduces a hierarchical graph-based retrieval strategy, where graph edges are derived by LLM attention weights, aligning retrieval with the document hierarchies above. To promote interoperability and efficiency across model choices, fourth, we propose a unified task embedding framework, that aligns task embeddings form various models, including smaller language models and LLMs with varied prompts, into a shared vector space, supporting broader interoperability and plug-and-play integrations of the preceding components. Extensive experiments across multiple across multiple datasets demonstrate that these methods effectively address key challenges in long-context NLP. Our approaches show notable performance improvements in both comprehension accuracy and computational efficiency.