Validation of nested params and the operation order (sanitisation before validation or vice versa)

[ZimNovich]

Hello!

Thanks for this great module! I am using it on a JSON-RPC API backend for validation of incoming JSON payloads.
A request body payload is composed of two parts: the header and the nested params. It looks like the below:

            const payload = {
                method: "fetch",
                api_key: "0ab124ccd129999ss",
                userid: "oxAdd34",
                params: {
                    filter: { datetime: "2021-05-05T14:48:00.000Z", module: "mobile", query: "a sample search text" },
                    start: 0,
                    limit: 12
                }
            };

I am receiving and processing these requests on the backend using endpoints like "/api/v2/posts", "/api/v2/posts", etc.
After extracting the payload from the request body, I validate this payload with a validation scheme like the one below:

const schemaValidate = {
    type: "object",
    properties: {
        method: { type: "string", pattern: "alphaDash", optional: false },
        api_key: { type: "string", pattern: "alphaNumeric", optional: false },
        format: { type: "string", eq: "json", optional: false },
        userid: { type: "string", pattern: "alpha", optional: false },
        params: {
            type: "object",
            properties: {
                filter: {
                    type: "object",
                    properties: {
                        datetime: { type: "date", pattern: "date-time", optional: true },
                        module: { type: "string", pattern: "alpha", optional: true },
                        year: { type: "number", optional: true },
                        query: { type: "string", pattern: "alphaNumeric", optional: true }
                    }
                },
                start: { type: "number", optional: true },
                limit: { type: "number", optional: true }
            }
        }
    }
};

If the payload passed the validation stage, I check the method field, extract the nested params field, sanitize it with another scheme and do all the further processing.

Now are the questions:

1. Is it better to sanitize the entire payload before validating it?
2. I don't like the fact that I need to constantly repeat the same header part for every payload validation scheme. Would it be better to do some prevalidation with some more shorter scheme like ...

const schemaValidatePayload = {
    type: "object",
    properties: {
        method: { type: "string", pattern: "alphaDash", optional: false },
        api_key: { type: "string", pattern: "alphaNumeric", optional: false },
        format: { type: "string", eq: "json", optional: false },
        userid: { type: "string", pattern: "alpha", optional: true },
        params: { type: "object", optional: true }
    }
};

... and then validate the nested params field using some more specific params validation scheme?

[mattwelke]

@ZimNovich Would you mind moving these questions to the GitHub Discussions for this repo? I'd like to keep the issues section dedicated to people reporting problems.

[mattwelke]

HAH this is discussions. I'm going to keep my comment here for posterity.

[mattwelke]

1. I'm not the creator of the library, but my company started using it before I joined the company, I think around 2016. The pattern they chose to use, which is the pattern I think works best, is to have a schema that is first used with the "sanitize" step and then used with the "validate" step. The reason for that is some input may be considered invalid before sanitization rules are applied, but considered valid after the rules are applied. In that case, you want to accept the data, not return an error.

For example, you might have a string that should be all lowercase and no more than 4 characters and the input data is "aBcD ". You could use the "trim" and "lower" sanitization rules and run it through a sanitize step, and you'd get "abcd", which would pass the validation rules. Therefore, no need to fail validation.

2. I think the use case of having some properties or validation rules common to multiple schemas is a common user case, but everyone will have a different preferred style of solving this problem. Some people like inheritance patterns, some people like composition. Some people would rather the library control it, some people would rather compose it themselves. So I've noticed that the library over the 9 or so years it's existed has never taken on responsibility for this. It's left it up to the user to DRY their code the way the user prefers.

For example, at my work, we do stuff like this:

// name common to multiple schemas
const nameSan = /* name property sanitization */

const sanSchema1 = {
  properties: {
    name: nameSan,
    foo: /* foo specific sanitization */
  }
};

const sanSchema2 = {
  properties: {
    name: nameSan,
    bar: /* bar specific sanitization */
  }
};

So we used composition done outside the library. We've never felt this was too onerous.