Validation of step-through forms

[jacob-a-brown]

Potential problem

Step through forms are elegant ways to have users submit data in an easy-to-understand, organized, and orderly fashion. At the end of the process, though, is when the data is actually submitted to the backend from the frontend and validated. This may cause issues because

• If invalid data is submitted the user will need to navigate back to the appropriate section to correct the invalid data. This was an issue raised in the heuristic audit of the original DataManager; users should not have to navigate too far or back and forth to correct data.
• If there is invalid data for one endpoint, but the other endpoints have valid data, records will be created via the valid endpoints but not the invalid one. This will then create fractured and orphaned data.

It's worth nothing that the frontend does do some data validation, but it does not encompass all of the custom validators from the backend (in particular because those validations are not part of the OpenAPI JSON that the frontend uses for TypeScript).

Proposed solution

When data is submitted to POST endpoints it is immediately validated via Pydantic. If there are any issues with the data, Pydantic raises 422 errors. For general issues, such as data types, Pydantic has pre-built messages. Fields and models can be further validated with custom validators.

To ensure that only valid data is submitted at the end of the step through form, and to ensure that errors are found and displayed to the user in the appropriate section, we can have validate POST endpoints; they would do nothing except for validate the data. They could be named /<endpoint>/validate and would expect the same incoming data. If the data is valid the endpoint would return a 202 status code.

Potential hazards

Some data is not validated until it gets submitted to the database, such as uniqueness and foreign key constraints. Since the proposed solutions do not necessarily validate against the database some errors may not be caught until the very end (which is the issue this proposed solution is trying to prevent).

Other

Database-level validations may be achieved by flushing data to the database via the /<endpoint>/validate and then rolling back the flushed operations before submitted the data to the actual POST endpoints. To achieve this would we have a POST /rollback endpoint? Can this cause other issues, such as security problems?

If we want to do this the data should remain flushed until the end so that foreign/primary keys are recognized and validated (and other database necessities/validations).

Alternative solution

Every POST endpoint could just flush the data, and then we can have a POST /commit endpoint that sends everything to the database. One advantage of this is that we can use try/except statements to rollback if something goes awry.

Potential hazards

POST endpoints are meant to write data to the database, so this alternative solution would be breaking protocol.

[jacob-a-brown]

On second thought the database flush/rollback solution may not be feasible because the endpoint calls use different sessions. So the Other and Alternative solution sections may not be viable. How to perform database-level validations then? It may be moot when one part of the form creates the necessary components for the next, but if the user wants to submit data that relates to other tables how can we validate that? Custom validators for the /<endpoint>/validate that check existence/uniqueness in the database? (e.g. sensor_id when submitting a sample or thing_id when submitting a water level)