Hadoop Infrastructure for Big Data using Docker Compose

Read full text
Authors: Julio C. López-Núñez
Cited by: 0

Abstract

Hadoop Big Data Infrastructure v2.0.0 Version 2.0.0 introduces a major update to the Docker Compose infrastructure used for Big Data learning, experimentation and technical demonstrations. Main improvements Persistent storage for HDFS NameNode and DataNode. Persistent Hive Metastore using PostgreSQL. Persistent storage for Zeppelin notebooks. Persistent storage for Kafka and ZooKeeper. Persistent MySQL storage used by Hue. Persistent storage for StreamSets. Persistent repositories and state for Apache NiFi. Apache NiFi updated and fixed to version 2.11.0. NiFi configured in standalone mode with HTTPS access. StreamSets startup issue fixed by configuring the required nofile limit. Docker services configured with restart: unless-stopped. Improved separation between container lifecycle and persistent data. Updated docker-compose.yml, hadoop-hive.env, README.md and CITATION.cff. Persistence This release introduces explicit Docker volumes for services that maintain state.

Fields of Study & Subjects

Cloud Computing and Resource Management Big Data Technologies and Applications Scientific Computing and Data Management

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
11 March 2026
Language
EN