Design and Implementation of NovaAI: A Hybrid Offline–Online Conversational AI System for Low- Resource Environments

Design and Implementation of NovaAI: A Hybrid Offline–Online Conversational AI System for Low-Resource Environments Harsh Singh Rao Bachelor of Computer Applications (BCA), 4th Semester Pacific Academy of Higher Education & Research University, Udaipur, India [harshsinghrao882@gmail.com] Abstract The widespread adoption of large language models (LLMs) has transformed conversational artificial intelligence; however, most modern systems rely on continuous internet connectivity and cloud-based inference. This dependency introduces challenges related to data privacy, latency, operational cost, and accessibility in low-resource or restricted network environments. This paper presents NovaAI, a hybrid offline–online conversational AI system designed to operate seamlessly across both local and cloud-based inference modes. NovaAI integrates a locally hosted quantized LLM using llama-cpp-python with an automatic fallback mechanism to an online inference API. The system is implemented using Python and Flask, accompanied by a lightweight web-based user interface. Experimental observations demonstrate that NovaAI enables practical AI-assisted interaction on consumer-grade hardware while preserving user privacy and maintaining service availability. The proposed system is particularly suitable for educational use, privacy-sensitive applications, and regions with limited or unreliable internet access. Keywords: Hybrid AI, Offline AI, Large Language Models, Edge AI, Privacy-Preserving Systems, Quantization. 1. Introduction Large language models (LLMs) such as GPT, LLaMA [1], and Mistral [2] have significantly advanced natural language understanding and generation. Despite these advances, most deployed conversational AI systems depend heavily on centralized cloud infrastructure. Such reliance results in multiple limitations, including increased latency, recurring usage costs, privacy risks due to data transmission, and reduced usability in low-bandwidth environments. To address these challenges, this paper introduces NovaAI, a hybrid offline–online AI system that prioritizes local inference while retaining the capability to utilize cloud-based models when required. The system is designed with a modular architecture that ensures uninterrupted operation and flexible deployment across diverse computing environments. The primary contributions of this work are: Architecture: Design of a lightweight hybrid AI architecture combining offline and online inference. Privacy: Implementation of a privacy-aware conversational AI system using open-source technologies. Evaluation: Practical evaluation of offline LLM inference on consumer-grade hardware. 2. Related Work Recent research has emphasized Edge AI and on-device inference to mitigate cloud dependency. Projects such as llama.cpp [3], GPT4All, and quantized model techniques (like QLoRA) demonstrate that LLMs can be executed efficiently on local CPUs. Hybrid inference strategies have also been proposed to balance computational cost and performance by selectively offloading tasks to cloud services. However, many existing systems lack seamless automatic fallback mechanisms or require complex deployment pipelines. NovaAI differentiates itself by offering a simplified, developer-friendly hybrid framework with integrated fallback logic and a web-based interactive interface suited for students and developers. 3. System Architecture NovaAI follows a modular client–server architecture consisting of the following components: User Interface: A browser-based chat interface implemented using HTML, CSS, and JavaScript. Backend Server: A Flask-based server responsible for handling user requests and API routing. AI Engine: Core logic that dynamically selects between offline and online inference modes. Model Layer: Includes a locally stored quantized LLM and access to an external cloud-based inference API. The system prioritizes offline inference using a locally hosted Mistral-7B-Instruct quantized model. If online inference is enabled and available, the system may utilize cloud-based inference; otherwise, it gracefully falls back to offline processing. 4. Implementation 4.1 Offline Inference Module Offline inference is implemented using the llama-cpp-python library, enabling efficient CPU-based execution of quantized LLMs. The system loads the mistral-7b-instruct-v0.2.Q4_K_M.gguf model during initialization. Context length and CPU thread usage are configured dynamically based on system resources to prevent system freezing. 4.2 Online Inference Module The online inference module interfaces with a REST-based AI service using HTTP requests. User prompts are formatted according to a standard chat completion schema. Timeout handling and error detection mechanisms are implemented to ensure robustness during network instability. 4.3 Hybrid Fallback Logic NovaAI employs a hybrid decision mechanism. By default, it prioritizes online inference for higher accuracy. However, in the event of API failure, high latency, or network unavailability, the system automatically switches to the offline local model without interrupting the user interaction. 4.4 Web-Based User Interface The user interface provides real-time conversational interaction, subject-based contextual prompts, and optional text-to-speech output. The interface is designed to be lightweight, responsive, and compatible with low-end hardware devices. 5. Experimental Evaluation The system was evaluated on a consumer-grade Linux-based system (Intel i5/AMD Ryzen equivalent with 8GB RAM). Evaluation criteria included response latency (Time to First Token), memory usage, and system availability during network interruptions. Observations: Latency: Offline inference on CPU showed higher latency compared to cloud APIs but remained within acceptable limits for reading speed. Reliability: The fallback mechanism successfully maintained conversation flow during simulated internet disconnects. Resource Usage: The quantized model required approximately 4-6GB of RAM, leaving sufficient headroom for the OS and browser. Results indicate that offline inference provides acceptable response quality for educational and general-purpose use, while online inference enhances response accuracy when connectivity permits. 6. Security and Privacy Considerations NovaAI enhances user privacy by performing inference locally. Data Sovereignty: During offline operation, sensitive user prompts and data do not leave the host machine. Optional Connectivity: Online inference is optional and can be disabled entirely via configuration. Credential Safety: API credentials for online mode are abstracted from the codebase using environment variables to prevent unauthorized exposure. 7. Code Availability The complete source code for the NovaAI system, including the hybrid AI engine, Flask backend, and web-based user interface, is publicly available in the following GitHub repository to support reproducibility: Repository URL: https://github.com/Harsh595fy/OFFLINE-AI-Nova 8. Conclusion and Future Work This paper presented NovaAI, a hybrid offline–online conversational AI system designed for low-resource and privacy-sensitive environments. The results demonstrate the feasibility of deploying LLM-powered applications without exclusive reliance on cloud infrastructure. Future work will focus on dynamic quality-based inference switching, mobile device deployment (Android/Termux), and secure long-term conversational memory integration. References [1] H. Touvron et al., "LLaMA: Open and Efficient Foundation Language Models," arXiv preprint arXiv:2302.13971, 2023. [2] A. Jiang et al., "Mistral 7B," arXiv preprint arXiv:2310.06825, 2023. [3] G. Gerganov, "llama.cpp: Port of LLaMA Model in C/C++," GitHub repository, 2023. [Online]. Available: https://github.com/ggerganov/llama.cpp [4] T. Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs," arXiv preprint arXiv:2305.14314, 2023.