Best practice for data validation (that we could adopt)

[rufuspollock]

When doing data integration to prepare a dataset I want to check/validate the dataset (e.g. structure, content) so that I catch errors and problems (e.g. data source has changed and data is now empty!)

Desiderata

• Error messages are excellent: human readable, easy to identify source of problem etc
• Lightweight and simple
• Could reuse any data schema we already have e.g. frictionless schemas
• Nice to have: understandable / readable / editable by people who are not hard-core software engineers
• Easy to add to existing projects

Notes

Known options

• Great expectations
• Frictionless validation (goodtables etc)