ediaqi/1_Data_Platform: KnowAIDE: a FAIR-Compliant Data Environment to Accelerate AI Research

Overview KnowAIDE is a dual-storage data infrastructure designed to operationalize FAIR (Findable, Accessible, Interoperable, Reusable) principles for heterogeneous environmental health research. Developed within the scope of the EDIAQI project, this platform resolves the "architectural impedance mismatch" between dynamic IoT sensor streams and static, unstructured research assets. The architecture integrates the iRODS data management system for file-based assets (e.g., surveys, images) with a FROST-Server implementation of the OGC SensorThings API for high-frequency time-series data. To facilitate AI workflows, KnowAIDE features a Semantic Fusion Layer that pre-materializes connections between these modalities, enabling researchers to query quantitative sensor observations alongside qualitative context in a single request. Key Features Dual-Storage Architecture: Couples the OGC SensorThings API (FROST-Server) for continuous time-series ingestion with iRODS for policy-based file management. Semantic Fusion Index: An Elasticsearch-based fusion engine that denormalizes metadata into a unified document structure. This feature provides a 14-fold speedup in data retrieval for multi-year datasets compared to standard API chaining. Automated FAIR Compliance: Programmatically enforces FAIR principles by automating the generation of Submission Information Packages (SIPs), minting Persistent Identifiers (DOIs) via the Zenodo API, and embedding machine-readable licenses (CC BY 4.0). Virtual Research Environment (VRE): Includes pre-configured JupyterLab templates and a Python client (knowaide.get_data) for semantic data retrieval, model training, and analysis. Robust Ingestion Pipelines: Utilizes Apache Airflow to orchestrate schema validation, duplicate detection, and anomaly flagging before persistence. Technical Stack & Requirements The infrastructure follows the EOSC Technical Architecture reference model and is containerized using Docker. Core Services: FROST-Server (v2.5.3), iRODS (v4.3.3), Elasticsearch (v8.12.0). Database: PostgreSQL/PostGIS. Security (AAI): Keycloak for centralized Identity and Access Management. Workflow & UI: Apache Airflow, Streamlit, Metalnx. Monitoring: Zabbix and Prometheus. Deployment This release contains the source code, Docker Compose configurations, and JupyterLab notebook templates used to validate the platform within the EDIAQI pilot network. Quick Start Please refer to the README.md for detailed deployment instructions. The system requires a Docker runtime environment. Funding & Acknowledgements This work was supported by the EDIAQI project, which has received funding from the European Union's Horizon research and innovation programme. This research was also carried out at the Know Center, funded within the COMET Competence Centers for Excellent Technologies Programme by the Austrian Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology. Citation If you use KnowAIDE in your research, please cite the associated paper: Lorenz Dirry et al. (2026). KnowAIDE: a FAIR-Compliant Data Environment to Accelerate AI Research.