A Semantic Multimodal Interface Framework for Accessible Navigation in Immersive Virtual Environments

Assistive Orientation Server with Computer Vision This project implements a TCP server in Python that uses multimodal AI (image + text) to assist visually impaired users in understanding and navigating indoor environments (e.g., rooms, corridors, offices). The system processes PNG images generated externally (e.g., from Unreal Engine, cameras, or mobile devices) and returns clear, concise spatial descriptions or navigation guidance in Portuguese or English, following strict accessibility-oriented constraints. Purpose The goal of this system is to provide reliable and safe spatial guidance by: describing the surrounding environment suggesting practical movements when appropriate assuming the user is positioned at the center of the scene This project is designed for: accessibility indoor navigation real-time assistive orientation How It Works (Overview) A TCP client connects to the server. The server monitors a local folder for the most recent PNG image (1.png, 2.png, ...). The image is encoded in Base64 and sent to the GPT-4o model along with an assistive prompt. The model returns a short, objective, and safety-oriented description or guidance. The response is sent back to the client via socket. System Architecture [Unreal Engine / Image Generator] │ ▼ Local folder (.png) │ ▼ [TCP Python Server] │ (image + prompt) ▼ GPT-4o │ ▼ Textual response │ ▼ Assistive Client Unreal Engine Integration (Blueprints) This project is designed to integrate with Unreal Engine, where the environment is simulated and images are captured in real time. The Unreal component is implemented using Blueprints responsible for: capturing the user’s viewpoint (camera) exporting images as sequential PNG files triggering image updates based on user movement or interaction optionally handling haptic feedback and collision events Blueprint Overview The following figures illustrate the Unreal Engine Blueprint logic used in the system: Image Capture Blueprint: captures the camera view and saves it as a PNG file in the monitored directory. Trigger/Update Logic: defines when new images are generated (e.g., movement, interaction, or time interval). Interaction & Feedback Blueprint: manages collision detection and haptic feedback for immersive guidance. (The following repository provides the complete set of Unreal Engine Blueprints used to reproduce the application, along with modular components that can be reused or adapted for other projects: Blueprint Repo) Project Structure serverAcessibilidade.py # Main TCP server README.md # Project documentation /blueprints # Unreal Blueprint images (to be included) Configuration 1. Requirements Python 3.9+ OpenAI SDK (openai >= 1.0) Install dependencies: pip install openai 2. Main Variables In serverAcessibilidade.py, configure: HOST = "127.0.0.1" PORT = 49809 IMAGE_DIR = "PATH/TO/IMAGE/FOLDER" Also configure the environment variable: export OPENAI_API_KEY="your_api_key_here" Assistive Prompt (AI Behavior) The system uses a highly constrained prompt to ensure accessibility and safety: never asks the user for additional information never invents elements not present in the image uses at most 4 short sentences employs clear, objective, spatial language provides tactile-oriented guidance when applicable Example Outputs “Follow the wall on your right.” “When you touch the table with your left hand, turn right.” “There is a chair in front of you, about two steps away.” TCP Communication The server listens on HOST:PORT. The client may send: a question (e.g., "Can I move forward?") or an empty string If no input is provided, the system uses the default instruction: "Describe the environment with as much detail as possible to help my navigation." Usage Example Generate numbered images in the configured folder: 1.png 2.png 3.png Start the server: python serverAcessibilidade.py Connect a TCP client (app, script, or embedded device). Receive real-time spatial descriptions or guidance. Accessibility & Ethics This project was developed with a strong focus on: user autonomy physical safety spatial clarity predictable responses ⚠️ This system should not be used as the sole navigation aid in hazardous environments. Possible Extensions Mobile app integration (Android / iOS) Real-time image streaming Haptic or audio feedback integration Scene memory and caching Critical obstacle detection License Experimental project for educational, academic, and accessibility research purposes.