ERA-10572: Support EFB schemas in ER > Add ajv validation library #1196

luixlive · 2024-11-28T22:47:31Z

What does this PR do?

Add AJV validations to the new SchemaForm fields on submission and show an error state in the fields.

This PR also updates the project to use Node 20 and its dependencies with exception of dev dependencies (this PR is already too big).

How does it look

Relevant link(s)

Where / how to start reviewing (optional)

I added comments that start with Related to AJV validations in the files where I added the new form validations to make it easier to review. These comments explain the reasoning behind each change. The rest of the files (> 85%) only have minor changes related to the dependency updates. In order to make the review process, I mention here what changes needed to be done after the updates:

Update all our Docker files to use Node 20 (this required finally getting rid of that old and deprecated node-sass library and install sass instead).
Updating MSW from version 0 to version 2 required to use the new http API instead of rest, having to do changes in several test files. Also, I needed to add new polyfills to the setupTests because Jest famous jest-environment-jsdom doesn't include a few. Actually, seems like the community talks about moving from Jest to Vitest since its faster, easier and has better polyfills for newer apps. But that is a conversation for another day.
Some libraries required changes in the way we do imports (mainly because they moved to ES modules) opting for a centralized package where to do named imports. These libraries are:
- react-router
- turf
- react-error-boundary
- redux-thunk
- `uuid``
react-toastify changed the properties under the object toast and removed several utilities that now we have to write like the toast type.
turf also updated some of the math calculations it does to provide more exact areas and perimeters, so some tests had to be updated to match the new values.
react-error-boundary changed a few things of the ErrorBoundary component API, I just had to tweak a bit the fallback component there.
react-to-print changed the way you pass down the content ref.

Thus, my recommendations to start reviewing are:

For the AJV validations: src/ReportManager/DetailsSection/SchemaForm/index.js
For reviewing of the dependencies update: package.json

Any background context you want to provide(if applicable)

AJV is a library that compiles JSON schemas into Javascript code that runs validations against objects that should follow the schema definition. This PR creates a hook that receives a schema and uses memoization to provide a method to run the validations over the compiled validator and returns a flat error object that maps the errors of each field by its id. Errors come as internationalized messages.
The dependencies update is a task we as frontend try to do once in a while. We haven't done it properly for more than a year so I did an update here. This one is specially a big one because I updated Node from 14 to 20, which required certain changes in some old libraries. For this reason, this PR requires a QA regression to make sure all features still work properly.

Future work:

A few dev dependencies weren't updated.
The new Node LTS version is 22, it should be easy to move there from Node 20.
After the updates done here, there are a lot of new warnings:
- PropTypes deprecation: we should remove all our prop typing
- Redux selectors performance issues: the new version of react-redux comes with helpful warnings to see what selectors can be improved to avoid rerenders. Sadly, we have a big bunch of places to optimize. But it's good to have that information and start working on it.
- New version of Sass also provides a lot of warning when building the project about deprecations in our style files (we have a few too).

Note: The translations spreadsheet was updated with the new texts.

swarmia · 2024-11-28T22:47:35Z

✅ Linked to Task ERA-10572 · Support EFB schemas in ER > Add ajv validation library
➡️ Part of Epic ERA-8048 · Event Type Form Builder (EFB)

luixlive · 2024-11-29T17:27:55Z

src/ReportManager/DetailsSection/SchemaForm/SchemaFormContext/index.js


-export const SchemaFormContext = createContext(null);


Related to AJV validations
This file is now just a Context object to import everywhere and avoid circular dependencies. It existed as a Provider component just for the purpose of handling the onFieldChange method, but with the errors object, now it makes more sense to have that method in the parent SchemaForm. So, having the provider as an independent component doesn't have sense anymore.

luixlive · 2024-11-29T17:29:36Z

src/ReportManager/DetailsSection/SchemaForm/fields/Text/index.js

@@ -26,13 +26,13 @@ const TEXT_INPUT_TYPE_TO_INPUT = { [INPUT_TYPE.SHORT]: ShortTextInput, [INPUT_TY
 const TEXT_INPUT_TYPE_STYLES = { [INPUT_TYPE.SHORT]: styles.shortText, [INPUT_TYPE.LONG]: styles.longText };

 const Text = ({ id }) => {
-  const { fields, formData, onFieldChange } = useContext(SchemaFormContext);
+  const { fields, fieldErrors, fieldValues, onFieldChange } = useContext(SchemaFormContext);


Related to AJV validations
Now we read from the context three flat objects: fields, fieldErrors and fieldValues. They map the field UI details, its validation error and the current value it holds. The three of them are mapped by the id of the field. This makes accessing this information trivial and O(1) since we don't have to handle recursivity with collections.

luixlive · 2024-11-29T17:30:00Z

src/ReportManager/DetailsSection/SchemaForm/fields/Text/index.js

@@ -50,17 +50,19 @@ const Text = ({ id }) => {
  }, [details.defaultInput, id, onFieldChange, value]);

  return <div data-testid={`schema-form-text-field-${id}`}>
-    <label className={styles.label} htmlFor={id}>{label}</label>
+    <label className={`${styles.label} ${error ? styles.error : ''}`} htmlFor={id}>{label}</label>


Related to AJV validations
Here I add styles for showing an error state in the text field.

luixlive · 2024-11-29T17:31:00Z

src/ReportManager/DetailsSection/SchemaForm/index.js

+
+  const fields = useMemo(() => makeFieldsFromSchema(schema), [schema]);
+
+  const fieldValues = useMemo(() => Object.entries(formData).reduce((accumulator, [fieldId, fieldValue]) => {


Related to AJV validations
Here we memoize the field values in a flat object. Every time the form data is updated this object will too.

luixlive · 2024-11-29T17:31:20Z

src/ReportManager/DetailsSection/SchemaForm/index.js

+    const newFormData = { ...formData, [fieldId]: value || undefined };
+
+    onFormDataChange(newFormData);
+    setFieldErrors((fieldErrors) => ({ ...fieldErrors, [fieldId]: undefined }));


Related to AJV validations
If a field value is changed we clear the error.

luixlive · 2024-11-29T17:35:31Z

src/utils/events.js

 import isObject from 'lodash/isObject';
-import metaSchemaDraft04 from 'ajv/lib/refs/json-schema-draft-04.json';
+import metaSchemaDraft04 from 'ajv-draft-04/dist/refs/json-schema-draft-04.json';


Related to AJV validations
The latest version of AJV, which I installed, doesn't come with the draft 04. I had to install the ajv-draft-04 library where we can read the metaschema to still support validation of our legacy schemas.

luixlive · 2024-12-04T15:38:53Z

src/EventFilter/index.test.js

@@ -1,54 +1,27 @@
 import React from 'react';
+import cloneDeep from 'lodash/cloneDeep';


This test file was broken due to changes in the new versions of redux and react-bootstrap. I needed to redo most of the tests but I made sure to keep the coverage.

luixlive · 2024-12-04T15:53:16Z

src/ReportManager/DetailsSection/SchemaForm/fields/Text/index.js

@@ -50,20 +50,28 @@ const Text = ({ id }) => {
  }, [details.defaultInput, id, onFieldChange, value]);

  return <div data-testid={`schema-form-text-field-${id}`}>
-    <label className={styles.label} htmlFor={id}>{label}</label>
+    <label className={`${styles.label} ${hasError ? styles.error : ''}`} htmlFor={id}>{label}</label>


Related to AJV validations
I added error state to the input and the corresponding aria tags to make the field accessible.

luixlive · 2024-12-04T15:54:11Z

src/ReportManager/DetailsSection/SchemaForm/fields/Text/index.test.js

@@ -3,7 +3,7 @@ import userEvent from '@testing-library/user-event';

 import { FORM_ELEMENT_TYPES, ROOT_CANVAS_ID, TEXT_ELEMENT_INPUT_TYPES } from '../../constants';
 import { render, screen } from '../../../../../test-utils';
-import { SchemaFormContext } from '../../SchemaFormContext';
+import SchemaFormContext from '../../SchemaFormContext';


Related to AJV validations
Tests updated with the addition of the error state and the aria tags.

luixlive · 2024-12-04T15:55:18Z

src/ReportManager/DetailsSection/SchemaForm/index.js


  const onSubmit = (event) => {
    event.preventDefault();

-    // TODO: Add form validation with AJV before submission and show errors.
-    onFormSubmit({ formData });
+    const fieldErrors = runValidations(formData);


Related to AJV validations
Before confirming the submission, we run the validations and show errors if there are any.

luixlive · 2024-12-04T15:59:08Z

src/ReportManager/DetailsSection/SchemaForm/utils/useSchemaValidations/index.js

@@ -0,0 +1,34 @@
+import { useCallback, useMemo } from 'react';


Related to AJV validations
Hook that receives an schema, memoizes the AJV compilation of its validate function and returns a runValidations memoized method that validates a form data object against the schema and returns a flat errors object mapping an internationalized message per error by the id of the field:

const runValidations = useSchemaValidations(schema); ... const errors = runValidations(formData); if (errors) { // errors is a flat object like { fieldId: 'Meaningful internationalized message with the error!' } } else { // No errors! }

luixlive · 2024-12-04T16:04:31Z

src/ReportManager/DetailsSection/index.js

  formValidator,
  isCollection,
  loadingSchema,
-  onFormChange,
+  onFormDataChange,


Related to AJV validations
I added the new callback to update the form data, and renamed the old one to onLegacyFormChange since we had to change a bit the logic of how we process the changes. They still do essentially the same thing.

luixlive · 2024-12-04T16:05:33Z

src/ReportManager/DetailsSection/index.test.js

 import { report } from '../../__test-helpers/fixtures/reports';
 import { VALID_EVENT_GEOMETRY_TYPES } from '../../constants';

 import DetailsSection from './';

 describe('ReportManager - DetailsSection', () => {
-  const onFormChange = jest.fn(),


Related to AJV validations
Had to redo (and added a few missing) most tests after the changes.

docker-compose.yml

…nused comments from a test file

gaboDev

I'm ready for approve, just asked some questions 🫡

gaboDev · 2024-12-05T16:56:31Z

src/ReportManager/DetailsSection/SchemaForm/utils/useSchemaValidations/index.js

+  const validate = useMemo(() => ajv.compile(schema.json), [schema.json]);
+
+  const runValidations = useCallback((formData) => {
+    const valid = validate(formData);


valid is only being used right in the next line for the if statement, Can we remove it and put it directly into the condition?

gaboDev · 2024-12-05T17:13:41Z

src/ReportManager/DetailsSection/SchemaForm/utils/useSchemaValidations/index.js

+    if (!valid) {
+      const fieldErrors = validate.errors.reduce((accumulator, error) => {
+        if (error.keyword === 'required') {
+          return { ...accumulator, [error.params.missingProperty]: t('required') };


So, by using error.params.missingProperty for the error mapping it means We are going to have the same generic error message for different field types, right?
I'm just asking, because idk if there might be some fields requiring more custom error messages ?

It means we will have the same error message for the same type of error. In this case, it is the error required. Of course, we can add more complexity to the condition that renders the text, in case there is a very specific scenario for some kind of field that wants a different message for the same error. We can totally handle that by passing the field type to the hook 😄 But honestly, I don't think that will happen much, or maybe even at all.

Sounds great!

gaboDev · 2024-12-05T17:20:25Z

src/ReportManager/DetailsSection/SchemaForm/index.js

+    const fieldErrors = runValidations(formData);
+    if (fieldErrors) {
+      setFieldErrors(fieldErrors);
+      document.getElementById(Object.keys(fieldErrors)[0]).focus();


This auto focus feels kind of non-react stuff, using a regular ref was complicating things?

Can you explain a bit more what do you mean by non-react? About using refs, we would need a ref for every field, and we don't know how many fields there are. That is dynamic depending on the schema. Maybe I can refactor it to be a little bit more legible:

const idOfFirstFieldWithError = Object.keys(fieldErrors)[0]; document.getElementById(idOfFirstFieldWithError).focus();

I don't know if that helps.

I meant non-react by accessing directly to a node using document.getElementById When usually We do it with refs, and as you mentioned it, having refs it would cause having unnecessary stuff around all components, so yes, I agree with the little refactor 🫡

Got you. Yeah it's not common to get elements by id in React. But in this specific scenario is pretty handy since we can't have a fixed number of refs because the fields are dynamic. I'll send the changes soon!

gaboDev · 2024-12-05T17:25:39Z

Do We have tickets created already for the points you mentioned on the future work section? If not, it would be great to have them as tech debt tasks

@luixlive

luixlive · 2024-12-05T17:55:01Z

Do We have tickets created already for the points you mentioned on the future work section? If not, it would be great to have them as tech debt tasks

You are right Gabo! I will create them 😄

gaboDev

Great work 🫡

luixlive · 2024-12-05T19:22:01Z

Ready for you guys @AlanCalvillo @fonzy911 and of course feel free to add more feedback regarding the UI @tiffanynwong

ERA-10572: Add AJV validations to form fields and update dependencies

7c2b9d4

luixlive added the deploy label Nov 28, 2024

luixlive marked this pull request as draft November 28, 2024 22:47

luixlive temporarily deployed to ERA-10572 November 28, 2024 22:47 — with GitHub Actions Inactive

luixlive changed the base branch from develop to ERA-10484 November 28, 2024 22:47

luixlive temporarily deployed to padas-app November 28, 2024 22:47 — with GitHub Actions Inactive

luixlive commented Nov 29, 2024

View reviewed changes

ERA-10572: Add translations to the new validation error message

73c5d85

luixlive temporarily deployed to ERA-10572 November 29, 2024 18:10 — with GitHub Actions Inactive

luixlive temporarily deployed to padas-app November 29, 2024 18:10 — with GitHub Actions Inactive

Base automatically changed from ERA-10484 to develop December 2, 2024 22:13

ERA-10572: Fix tests after dependency updates

d5f3fc6

luixlive temporarily deployed to ERA-10572 December 3, 2024 15:05 — with GitHub Actions Inactive

luixlive temporarily deployed to padas-app December 3, 2024 15:06 — with GitHub Actions Inactive

ERA-10572: Add unit tests and missing a11y practices in the text input

e42f221

luixlive added deploy and removed deploy labels Dec 3, 2024

Merge branch 'develop' into ERA-10572

6d39db2

luixlive temporarily deployed to ERA-10572 December 3, 2024 22:22 — with GitHub Actions Inactive

luixlive temporarily deployed to padas-app December 3, 2024 22:22 — with GitHub Actions Inactive

luixlive requested review from AlanCalvillo, gaboDev, JoshuaVulcan and fonzy911 December 3, 2024 22:31

luixlive marked this pull request as ready for review December 3, 2024 22:33

luixlive commented Dec 4, 2024

View reviewed changes

JoshuaVulcan reviewed Dec 4, 2024

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

ERA-10572: Uncomment docker-compose tests paralelization and remove u…

9f812f5

…nused comments from a test file

luixlive temporarily deployed to ERA-10572 December 4, 2024 17:51 — with GitHub Actions Inactive

luixlive temporarily deployed to padas-app December 4, 2024 17:51 — with GitHub Actions Inactive

Merge branch 'develop' into ERA-10572

74f60c6

gaboDev requested changes Dec 5, 2024

View reviewed changes

gaboDev approved these changes Dec 5, 2024

View reviewed changes

ERA-10572: PR feedback from design and minor code improvements

b396fb3

luixlive temporarily deployed to ERA-10572 December 5, 2024 18:43 — with GitHub Actions Inactive

luixlive temporarily deployed to padas-app December 5, 2024 18:43 — with GitHub Actions Inactive

AlanCalvillo approved these changes Dec 6, 2024

View reviewed changes

ERA-10572: Small refactors and improvement to default input autofilling

8d816e4

luixlive temporarily deployed to ERA-10572 December 6, 2024 20:24 — with GitHub Actions Inactive

luixlive temporarily deployed to padas-app December 6, 2024 20:24 — with GitHub Actions Inactive

ERA-10572: Turn off feature flag

db3b6d4

luixlive temporarily deployed to ERA-10572 December 6, 2024 22:01 — with GitHub Actions Inactive

luixlive merged commit 2070210 into develop Dec 6, 2024
2 checks passed

luixlive deleted the ERA-10572 branch December 6, 2024 22:01

luixlive temporarily deployed to padas-app December 6, 2024 22:01 — with GitHub Actions Inactive


		const fields = useMemo(() => makeFieldsFromSchema(schema), [schema]);

		const fieldValues = useMemo(() => Object.entries(formData).reduce((accumulator, [fieldId, fieldValue]) => {

		@@ -1,54 +1,27 @@
		import React from 'react';
		import cloneDeep from 'lodash/cloneDeep';

		@@ -0,0 +1,34 @@
		import { useCallback, useMemo } from 'react';

ERA-10572: Support EFB schemas in ER > Add ajv validation library #1196

ERA-10572: Support EFB schemas in ER > Add ajv validation library #1196

Uh oh!

Conversation

luixlive commented Nov 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

How does it look

Relevant link(s)

Where / how to start reviewing (optional)

Any background context you want to provide(if applicable)

Uh oh!

swarmia bot commented Nov 28, 2024

Uh oh!

luixlive Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luixlive Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luixlive Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luixlive Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luixlive Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luixlive Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luixlive Dec 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gaboDev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaboDev commented Dec 5, 2024

Uh oh!

luixlive commented Dec 5, 2024

Uh oh!

gaboDev left a comment

Choose a reason for hiding this comment

luixlive commented Nov 28, 2024 •

edited

Loading

luixlive Nov 29, 2024 •

edited

Loading

luixlive Nov 29, 2024 •

edited

Loading

luixlive Nov 29, 2024 •

edited

Loading

luixlive Nov 29, 2024 •

edited

Loading

luixlive Nov 29, 2024 •

edited

Loading

luixlive Nov 29, 2024 •

edited

Loading

luixlive Dec 4, 2024 •

edited

Loading