Harmonized Geospatial Dataset from the 2010 Brazilian Demographic Census by IBGE

Harmonized geospatial dataset from the 2010 Brazilian Demographic Census by IBGE This dataset provides a harmonized geospatial version of the 2010 Brazilian Demographic Census at the census tract level. The data are distributed as separate, gzip-compressed files (one per table and per geography layer) together with the script that builds a single GeoPackage from them. The resulting database contains cleaned, standardized, and consistently typed attribute tables linked to official census tract geometries. Variable names were harmonized, data types were explicitly enforced, and structural inconsistencies were resolved to facilitate reproducible spatial analysis and integration with GIS and data science workflows. The original data source is the 2010 Brazilian Demographic Census conducted by the Instituto Brasileiro de Geografia e Estatística (IBGE): the census tract aggregates ("Agregados por Setores Censitários") and the tract synopsis ("Sinopse por Setores"), and the official census tract, district, subdistrict and municipality boundaries. What is in this deposit Files: attribute tables *.csv.gz — (basico, domicilio01–02, domiciliorenda, entorno01–05, pessoa01–13, pessoarenda, responsavel01–02, responsavelrenda, sinopse). One row per census tract, first column setor_code, then the variables V001, V002, … geospatial layers *.gpkg.gz — setores, subdistritos, distritos and municipios. metadata catalogs: IBGE_CENSO-2010_LAYERS_X_X.csv, IBGE_CENSO-2010_FIELDS_X_X.csv, IBGE_CENSO-2010_VARIABLES_X_X.csv — the metadata catalogs (uncompressed, readable on the repository). builder script build_database.py — the script that builds a GeoPackage database from the sources files. Files are kept separate on purpose, so that new versions of the deposit only replace the files that changed. The .csv.gz files can be read directly by pandas, pyarrow, DuckDB, R and most other tools; the .gpkg.gz files must be decompressed before use in GIS software. Rebuilding the database 1. Download all files into one folder (keep the .gz files as they are; no need to decompress).2. Create and edit build_database_2010.json: set input_folder (the folder above), output_folder and the three catalog paths.3. Run python build_database.py --config build_database_2010.json --overwrite. Requirements: Python 3 (standard library only) and the GDAL command line tool ogr2ogr available in the PATH. The script reads the compressed files directly (geography layers are unpacked temporarily next to the output and removed), builds the database from zero, and writes a log with checks (row counts, column types, sector key cross-check). The header of the script documents every option of the configuration. Layers The Database GeoPackage has 34 layers, named without census or year (e.g. basico, setores): - Geospatial: setores, distritos, subdistritos, municipios.- Attribute tables: the 27 tables listed above, linked to setores by setor_code.- Three structured metadata layers: - layers - Describes all database layers contained in the GeoPackage, including geometry type, spatial reference system, thematic scope, and relational structure where applicable. - fields - Documents major attribute fields and naming conventions, including standardized prefixes, identifier variables, geographic codes, and structural variables used for relational integrity and joins. - variables - Provides a complete listing of original census variables, including official variable names, harmonized names (when applicable), and their corresponding descriptions (Portuguese and English) as defined in the 2010 census documentation. Together, these metadata layers provide full transparency regarding database structure, schema harmonization, and variable provenance. Conventions - Identifiers: every column ending in _code (and code) is stored as text, so leading zeros and long codes are never altered.- Variables: columns named V + number are numeric. They are stored as integers, except the few variables that actually hold decimals, which are detected from the data when the database is built (they are reported in the build log). Counts that the source wrote as 12.0 are stored as whole numbers.- Missing and protected values use fixed codes in every table (documented in the fields layer): -9999 no data (empty in the source), -1 hidden for statistical confidentiality (X in the source), -2 uncertain (. in the source). Text fields use NA for no data. Data quality notes The first 2010 harmonized tables had structural defects that were corrected in this version (details in the log and scripts of the repository): - Sector codes that a spreadsheet had turned into scientific notation (e.g. 2,3001E+14) or into short codes were restored from the order of the household table (domicilio01), only where the match was unambiguous. This affected the entorno02–05 tables (72,521 rows), pessoa08 (2,346 rows) and pessoa11 (47,733 rows).- 47,733 rows of São Paulo in responsavel01 and responsavel02 had been written whole inside the sector code cell; they were parsed back into columns.- 47,733 identical duplicate rows in pessoa08 were removed.- Three cells filled with garbage characters (a truncated buffer in the source files: one in entorno05, two in pessoa04) were set to no data (-9999).- Empty cells are -9999 and X is -1 (for example the 888,338 hidden cells of the sinopse table). Tables do not all cover exactly the same set of tracts. Most have 297,741 tracts; basico has 12 fewer, pessoa04 21 fewer, pessoa07 28 fewer, entorno05 and responsavel01–02 6 fewer, and sinopse has a larger universe (311,052 tracts, 13,317 of them absent from the household table). Join on setor_code and expect unmatched rows. Data sources Instituto Brasileiro de Geografia e Estatística (IBGE). (2010). Censo Demográfico 2010: resultados do universo por setores censitários. https://censo2010.ibge.gov.br/FTP: https://ftp.ibge.gov.br/Censos/Censo_Demografico_2010/ Version log 2.0.0 Second version, with the split method: the tables and geography layers are distributed as separate compressed files, with the harmonization fixes above, and users can rebuild the database by running the build_database.py script. 1.0.0 First version approach with a single database shipped.