Beyond Unit Tests - Advancing Testing Strategies for Research Software

As research software becomes increasingly complex and interdisciplinary, assuring its quality and keeping the software well maintained requires testing approaches that go beyond traditional unit tests. Such traditional testing techniques are often insufficient in research software contexts where expected outcomes may be uncertain, stochastic, or influenced by complex simulation contexts. Not being able to predict what the correct output of a program should be for a given input is known in the literature as the oracle problem. Yet, rigorous testing remains fundamental to long-term maintainability, reproducibility, and trustworthy computational results in the RSE domain. This 90-minute workshop will introduce participants to more advanced testing strategies suited for the challenges in predicting expected outcomes in research software. We will revisit essential testing concepts such as test case design, coverage analysis, and integration testing before we dive into challenges particularly present in research software. Here, we will explore practical strategies to cope with the oracle problem, such as metamorphic testing or property-based testing. We will illustrate each with real-world, scientific software examples. This session is aimed at RSEs and developers with a basic knowledge of automated testing who want to strengthen their testing knowledge with practices suited for hard-to-test research software prototypes. Participants will leave the session armed with practical techniques and templates for implementing robust, reproducible testing frameworks for use in their own projects.