Sebastian Schelter

Aug 16, 2024 | amazon.science | Sebastian Schelter |Stefan Grafberger |Philipp Schmidt |Tammo Rukat

Modern companies and institutions rely on data to guide every single decision. Missing or incorrect information seriously compromises any decision process. In previous work, we presented Deequ, a Spark-based library for automating the verification of data quality at scale. Deequ provides a declarative API, which combines common quality constraints with user-defined validation code, and thereby enables unit tests for data.