jqnatividad/csv-nose: 0.2.1

Read full text
Authors: Joel Natividad
Cited by: 0

Abstract

Added Score a dialect candidate on its all-modal window when its field-count variance comes entirely from a non-tabular leading block. Files like dd_Wickenburg_nobmp_623.csv carry metadata lines ($$veh_info, veh_indices=[...]) above a perfectly uniform table; those rows inflate the population sigma so tau_0 = 1/(1+2σ) collapses and the true delimiter loses to a degenerate single-field parse. The window discards a guarded leading block plus at most one truncated trailing record, and because it is all-modal by construction tau_0 = tau_1 = 1.0 is an exact identity rather than an approximation. Guard rails keep this from becoming a general "discard inconvenient rows" escape hatch: at least 2 and at most 25% of rows may be discarded, at least 5 must survive, and each discarded row must hold under 25% of the modal field count — which necessarily excludes every modal row, making a cascading trim structurally impossible Fixed skip_preamble no longer stops at a blank line inside a leading comment block. Blanks are buffered and committed as preamble only when a further comment line follows, so a trailing blank run before data is left in place (#c\n\n\ndata reports one preamble row, not three). Previously a single blank on line 2 of a file meant the rest of its comment header was scored as data, destroying uniformity for every candidate Space is penalised 0.45 rather than 0.75 below 5 parsed rows. A handful of rows does not provide enough repetition to distinguish a real space delimiter from incidental spacing between words. The threshold is measured in parsed rows rather than lines, and is set so that genuinely space-delimited short files remain eligible Changed Benchmark accuracy improvements (zero regressions across all five suites): POLLOCK 97.97% → 98.65%, CSV Wrangling 93.30% → 94.97%, CODEC 92.25% → 94.37%, MESSY 91.27% → 93.65% (W3C-CSVW unchanged at 99.55%). Recovered weapondef.csv, dd_Wickenburg_nobmp_623.csv, memberList.csv and product_feed.csv csv-nose now leads outright on all five benchmark suites on both success ratio and F1. Previously DuckDB's sniff_csv tied on the CODEC subset and narrowly led on MESSY; both now trail. Refreshed the README.md and docs/ACCURACY.md figures and ranking claims accordingly Documented the remaining benchmark failures in docs/ACCURACY.md, separating genuine limitations (whitespace-run delimiters, unquoted nested delimiters) from files that are degenerate, non-tabular, or carry dubious ground-truth annotations Replaced the packaging exclude with an explicit include. The denylist published local tooling to crates.io — .claude/ (agents, settings, skills), .serena/, and CLAUDE.md were all in the 1.1.0 crate because anything not named was included by default. The published crate drops from 41 files to 28 tests/benchmark_accuracy.rs is no longer published. It drives the CSVsniffer corpora, which are not redistributable here, and it panics rather than skipping when they are absent — so the 1.1.0 crate could not pass cargo test from its own packaged source. The packaged crate now runs 120 tests cleanly Relicensed from MIT OR Apache-2.0 to MIT only, copyright datHere, Inc. Releases up to and including 1.1.0 remain available under their original dual-license terms Added LICENSE-MIT. Cargo.toml had always declared a license but no license text existed in the repository, so no license was shipped with the crate through 1.1.0 Full Changelog: https://github.com/jqnatividad/csv-nose/compare/v1.1.0...v1.2.0

Fields of Study & Subjects

Software Engineering Research Data Quality and Management Digital Humanities and Scholarship

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
19 January 2026