Deserialisers should be able to skip fields

[fabianoliver]

Hi,

This is an idea for a potential improvement.

A common use case for chr.avro is of course to deserialise avro encoded data into some DTO or model type.

Now, I'm not quite sure how common this is among the wider user base, but personally I tend to create the DTOs by hand, mapping out only the properties I am interested in.

To give a simple real world example, imagine you are publishing a change stream from some database to kafka, and then have a .NET service consume that. One message represents the change of one row of the table, and its avro schema has fields for every column. Oftentimes, my consuming service does not care about all colums; often, I just need to load up a subset. So, I would model the DTO to only have whatever columns I care for, and then deserialise via chr.avro.

At the moment, deserialisation is a bit wasteful in those circumstances; the generated binary deserialiser would fully deserialise every single field, and then just bind whatever is mapped out to my DTO.

In theory, this could be improved a bit if deserialisers supported skipping fields. Of course due to avro's binary representation, skipping a field isn't a no-op either, but it could be a fair bit cheaper. Say for example the schema has a string field that I haven't bound to my object; instead of converting the data to utf16 and allocating a string for it, just to throw it away quite immediately, the string deserialiser would be able to just read in the length, and then skip over those bytes in the message. And similarly for other primitive types.

The biggest problem is arguably that it might hard to fit this into the existing contract in order for eg BinaryRecordDeserialiser to convey to the other serialisers whether it would like to read or skip some field, possibly in a way so third party deserialisers aren't broken.

From the top of mind, maybe new ISkippableBinaryDeserializerBuilder/ISkippableBinaryDeserializerBuilderCase could be introduced, and BinaryRecordDeserializerBuilderCase etc could check whether these are implemented, and use skip methods if appropriate it so.
Alternatively, if supporting something like this only in newer .NET versions, it might also be a decent option to add Skip() as default interface methods which by default just forward to the regular read methods, but can then be specialised.

Anyways - not sure if there is any wider interest in this, but thought it might be good to open up a discussion as I somewhat often run into cases where this would be useful.

[dstelljes]

Agreed; this would be nice. There was another issue tracking this early on (#4) that was closed following the refactoring that went into 8.0.0. Now would be a good time to revisit; I'd like to do something about #314 in the next major release, and that will also involve some changes to the serde builder contracts.

[fabianoliver]

Thanks - I noticed you tagged these against a new milestone, so figured now might be a good time to lodge suggestions :-)
I might add a few more minor ones as separate tickets then, just as potential ideas.

Would be happy to help with some of those as well if needed, but I reckon if you are planning somewhat wider reaching changes, might be too early at this point / before you've landed on a set of api changes you are happy with?

[dstelljes]

Thanks - I noticed you tagged these against a new milestone, so figured now might be a good time to lodge suggestions :-)
I might add a few more minor ones as separate tickets then, just as potential ideas.

Please do!

might be too early at this point / before you've landed on a set of api changes you are happy with?

I think it'd be good to work out API changes for #314 and this one simultaneously (or at least make any changes needed for #314 with an eye toward skip support later on). I don't think that needs to block work on other changes, though, provided those changes aren't also contingent on serde architecture adjustments.