HeeeyNathan/Breitenbach_2.0: Data and code for: Comparing species accumulation within taxonomically well- and poorly-resolved insect groups over 37-years

Comparing species accumulation within taxonomically well- and poorly-resolved insect groups over 37-years Overview This repository contains the complete analytical pipeline for examining the influence of "dark taxa" on biodiversity patterns in freshwater ecosystems. Using a unique 37-year quantitative dataset (1969-2006) of aquatic insects from the Breitenbach stream, we compare abundance, richness, turnover, and accumulation patterns between taxonomically well-resolved (EPT: Ephemeroptera, Plecoptera, Trichoptera) and poorly-resolved (Diptera) insect groups. Authors Nathan Jay Baker, Viktor Baranov Table of Contents Repository Structure Data Requirements Dependencies Installation Analysis Workflow Key Analyses Computational Requirements Troubleshooting Citation License Contact Repository Structure ├── Data/ │ └── Breitenbach_community_data_17.12.2024.xlsx # Main dataset ├── Outputs/ # Saved analysis outputs │ ├── ChoasDiversityProfilesDiptera.rds │ ├── ChoasDiversityProfilesEPT.rds │ ├── rarefaction_diptera.rds │ ├── rarefaction_ept.rds │ ├── chaohill_results_diptera.rds │ ├── chaohill_results_ept.rds │ ├── Diptera_TD.rds │ ├── Diptera_TD_red.rds │ └── diptera_family_trends.xlsx ├── Plots/ # Generated figures │ ├── All plots for publication/ # Publication-ready figures │ ├── Diptera/ # Diptera-specific plots │ └── EPT/ # EPT-specific plots ├── Taxonomy information/ # Taxonomic data exports ├── Breitenbach_2.0.Rproj # RStudio project file ├── HighstatLibV15.R # Custom statistical functions ├── EPT_result_summary.qmd # Main EPT analysis ├── Diptera_result_summary.qmd # Main Diptera analysis ├── Chao-Hill plots by trap.qmd # Diversity profile analysis (Chao & Jost 2015 method) ├── Blowes et al. 2022 plots.qmd # Delta analysis (Blowes et al. 2022 method) ├── Rarefaction plots combined.qmd # Combined rarefaction plots └── All model plots combined.qmd # Combined model visualizations Data Requirements Primary Dataset File: Breitenbach_community_data_17.12.2024.xlsx Format: Excel spreadsheet with species abundance data Structure: Taxonomic columns: taxa_id, original_name, harmonized_name, validated_name, family, Order Spatial column: trap (trap locations: A, B, C, E, G, I, III, plus excluded traps) Temporal columns: Annual abundance counts from 1969 to 2006 (38 years) Data type: Integer abundance counts per species-trap-year combination Rows: Individual species records with yearly abundance data across all traps Data Collection Details Study Period: 37 years (1969-2006) Sampling Method: Weekly emergence trap sampling Trap Design: Stationary "greenhouse-like" emergence traps covering 12m stream length Preservation: Automatic collection in ethanol jars Identification: Morphological identification to species level Spatial Replication: 7 traps with observation periods ≥9 years (Traps A, B, C, E, G, I, III) Dependencies R Version R: 4.4.2 Platform: x86_64-w64-mingw32/x64 Core Packages # Data manipulation and visualization library(tidyverse) # Data wrangling and ggplot2 library(patchwork) # Combining plots library(janitor) # Data cleaning library(scales) # Scale formatting # Statistical modeling library(glmmTMB) # Generalized linear mixed models library(DHARMa) # Model diagnostics library(sjPlot) # Model output tables library(ggeffects) # Model predictions library(mgcv) # GAMs for non-linearity testing library(splines) # Spline functions library(mblm) # Median-based linear models # Biodiversity analysis library(vegan) # Community ecology analyses library(iNEXT) # Interpolation/extrapolation of diversity library(SpadeR) # Species richness estimation library(mobr) # Measurement of biodiversity library(codyn) # Community dynamics (turnover) library(entropart) # Entropy partitioning # Data I/O library(readxl) # Reading Excel files library(writexl) # Writing Excel files # Parallel processing library(future.apply) # Parallel computation # Utilities library(gridExtra) # Grid arrangements library(data.table) # Fast data manipulation Custom Functions File: HighstatLibV15.R Source: Zuur, A. F. ., & Ieno, E. N. . (2021). The world of zero-inflated models: Volume 1. Functions: Data standardization (MyStd()), diagnostic plots (Mydotplot()) Installation 1. Clone Repository git clone https://github.com/HeeeyNathan/Breitenbach_2.0 cd breitenbach-ecology 2. Install R Dependencies # Install CRAN packages packages Open Project > Breitenbach_2.0.Rproj # Ensure HighstatLibV15.R is in working directory source("HighstatLibV15.R") Analysis Workflow The analysis follows a hierarchical approach with two main taxonomic groups: Phase 1: Data Preparation Data Import: Load and clean community data from Excel Trap Harmonization: Standardize trap naming conventions Data Transformation: Convert to long format and aggregate by trap/species Quality Control: Filter traps with sufficient temporal coverage (≥9 years) Phase 2: Biodiversity Metrics Basic Indices: Species richness, Shannon diversity, Simpson diversity, evenness Chao Estimators: Species richness estimation for individual communities Diversity Profiles: Hill numbers (q = 0, 1, 2, 3) with confidence intervals Turnover Analysis: Species appearance/disappearance rates between years Rarefaction Curves: Individual-based accumulation curves using iNEXT Phase 3: Statistical Modeling Temporal Trend Analysis: GLMMs for abundance, richness, and turnover Model Selection: AIC-based comparison with various correlation structures and model variants Model Validation: DHARMa residual diagnostics (including residual vs. predicted plots) Predictions: Generate fitted values with confidence intervals Family-Level Trends: Median-based linear models (mblm) to identify which families drove temporal changes Phase 4: Comparative Analysis Cross-group Comparisons: EPT vs. Diptera patterns Delta Analysis: Following Blowes et al. (2022) methodology Visualization: Publication-ready figures and combined plots Key Analyses Script Execution Order Primary Analyses (Run in Parallel) EPT_result_summary.qmd - Complete EPT analysis pipeline Diptera_result_summary.qmd - Complete Diptera analysis pipeline Secondary Analyses (Require Primary Outputs) Chao-Hill plots by trap.qmd - Diversity profiles by trap Blowes et al. 2022 plots.qmd - Delta correlation analysis Rarefaction plots combined.qmd - Combined accumulation curves All model plots combined.qmd - Combined temporal trends Core Analytical Methods 1. Diversity Profile Analysis Method: Chao-Hill numbers with bootstrap confidence intervals Orders: q = 0 (richness), q = 1 (Shannon), q = 2 (Simpson) Output: Trap-specific and overall diversity profiles 2. Temporal Trend Modeling # Example model structure glmmTMB(Response ~ poly(year.std, degree) + (1 | ftrap) + ar1(0 + factor(year.std) | ftrap), family = distribution, data = dataset) Distributions: Negative binomial (abundance), Gamma (richness), Gaussian (turnover) Random Effects: Trap-level intercepts Correlation: AR(1) temporal autocorrelation Predictors: Standardized polynomials for non-linear trends Model Variants: Multiple GLM formulations tested to address residual vs. predicted validation diagnostics 3. Family-Level Trend Analysis Method: Median-based linear models (mblm package) for robust slope estimation Scope: Per-trap and overall trends in family-level abundances for both EPT and Diptera Classification: Families classified as increasing, decreasing, or stable based on slope direction Output: Outputs/diptera_family_trends.xlsx with per-family trend summaries 4. Rarefaction Analysis Method: iNEXT interpolation/extrapolation Type: Individual-based curves (q = 0) Standardization: Common individual count for comparison 5. Delta Analysis (Blowes et al. 2022) Metrics: ΔN (abundance), ΔS (richness), ΔS_rare (rarefied richness) Correlations: Pairwise relationships between delta metrics Visualization: Scatterplots with ellipses and trend lines Computational Requirements Hardware Recommendations RAM: Minimum 8GB, recommended 16GB+ CPU: Multi-core processor (analysis uses parallel processing) Storage: ~500MB for outputs and figures Processing Notes Parallel Processing: Analysis uses future.apply Memory Management: Large objects cached as .rds files Runtime: Complete analysis ~30-60 minutes depending on hardware Computationally Intensive Steps Diversity Profile Calculation: Bootstrap resampling (B = 1000) Rarefaction Curves: iNEXT computations across all traps Chao-Hill Analysis: Incidence-based diversity profiles Model Fitting: GLMM estimation with AR(1) correlation Memory Optimization # Increase memory limits for parallel processing options(future.globals.maxSize = 2000 * 1024^2) # 2GB limit # Clean environment between major steps rm(large_objects) gc() Troubleshooting Common Issues 1. Memory Errors Problem: "C