Feature Request: Add an option to allow ListSerializer to continue with errors

[EmperorArthur]

Hello, I often deal with a data access pastern that is probably quite common in the wild, and wanted to discuss the possibility of addressing it.

We pull in tens of thousands of machine generated data points, then pass them to a serializer before saving. The problem is that there are often a few records which are incorrect. In our case, it's often a string that can not be blank in the CSV file, but the same general thing applies to any validator.

Proposal

Add an allow_failures kwarg to ListSerializer. In this mode, is_valid would succeed in setting validated_data based on all the data which had no errors, but everything else would stay the same.

Details

• to_internal_value returns a list of the objects which succeeded, but also sets "self._errors" to a list of errors that fails.
  • This just changes raise ValidationError(errors) to self._errors = errors
• Modify is_valid to clear "self.errors" before calling self.run_validation(self.initial_data), instead of afterwards.

This presents a more granular approach versus all or nothing, and explicitly allows gratefully handling systems which occasionally "hickup".

I have personally use code similar to the above for over a year at this point, and have found it invaluable. Maybe others will as well.

[tomchristie]

Good first step here would be to address this yourself perhaps with a customized ListSerializer subclass.

Once you've demo'ed something there, either as a code snippet or as a third party package, then others would be able to reuse it too, and we could consider linking to it from the docs.

[EmperorArthur]

Sorry I should have been more clear. We currently do that, but it's sort of hacky, since any change to the base class is not reflected in those two functions. There is no way to do a super() call and get the functionality we need.

This is what we currently use:

class PartialListSerializer(ListSerializer):
    """
    Modified ListSerializer which allows for saving records even if not all of them are valid
    See https://github.com/encode/django-rest-framework/discussions/7828
    """

    def to_internal_value(self, data):
        ret = []
        errors = []

        for item in data:
            try:
                validated = self.child.run_validation(item)
            except ValidationError as exc:
                errors.append(exc.detail)
            else:
                ret.append(validated)
                errors.append({})

        if any(errors):
            self._errors = errors  # noqa This is how the superclass works...

        return ret

    def is_valid(self, raise_exception=False):
        assert hasattr(self, 'initial_data'), (
            'Cannot call `.is_valid()` as no `data=` keyword argument was '
            'passed when instantiating the serializer instance.'
        )

        if not hasattr(self, '_validated_data'):
            self._errors = []  # noqa This is how the superclass works...
            try:
                self._validated_data = self.run_validation(self.initial_data)
            except ValidationError as exc:
                self._validated_data = []  # noqa This is how the superclass works...
                self._errors = exc.detail  # noqa This is how the superclass works...

        if self._errors and raise_exception:
            raise ValidationError(self.errors)

        return not bool(self._errors)

[tomchristie]

any change to the base class is not reflected in those two functions. There is no way to do a super() call and get the functionality we need.

Which is actually probably absolutely fine. ListSerializer isn't in flux, and there's no hard reason why calling into super is always necessary or somehow notionally "correct".

[EmperorArthur]

Thanks for the reply. It just seems a bit hacky to basically copy and paste the DRF code and change all of one line as in the first function.

It also seems like the sort of thing that someone else may have a use for as well. There are plenty of systems that occasionally hiccup and give bad data. That's why this is a feature request. I saw a need, and don't think everyone should have to re-invent the wheel.