The Mapper should be allowed to optimise for batch mapping

[Bertg]

We have set up our EventStore to encrypt our data. The way it works for us is that each event can derive a token, which can be looked up in a data_key table, which we can then use to fetch the actual encryption key from our KMS. This means that for each event we want to read we have both a DB and an API call.

Now, both our DB and API allow fetching multiple values at once. Imagine that :)

How could we optimise the EventStore mapping logic so that it could pre-fetch this information in 1 DB call, and 1 API call (per batch result maybe?).

I know we are looking at the SpecificationReader and this method specifically:

def each(specification_result)
  repository.read(specification_result).each { |batch| yield batch.map { |record| mapper.record_to_event(record) } }
end

It would be nice if we could do something like this:

def each(specification_result)
  repository.read(specification_result).each { |batch_of_records| yield mapper.records_to_events(batch_of_records) }
end

It would be fairly trivial to remain backwards compatible for custom mappers:

def each(specification_result)
  if mapper.respond_to?(:records_to_events)
    repository.read(specification_result).each { |batch_of_records| yield mapper.records_to_events(batch_of_records) }
  else
    warn "Some warning here to update your custom mappers to support `records_to_events`
    repository.read(specification_result).each { |batch| yield batch.map { |record| mapper.record_to_event(record) } }
  end
end

What do you think?

[mpraglowski]

Interesting idea. If I understood it correctly your use case is to fetch all encryption keys in single call (DB & API) for whole batch of events and keep them in key repository's cache for use while transforming events, correct? I will play a little bit with this.

[mpraglowski]

Modifying the Repository itself, I'm not a big fan of.

Me neither :) But that's a workaround that will allow to implement your use case using existing RES version.

I like the idea of batch processing. I'm not sure how to implement it now but I will play with it soon.

[Bertg]

I created a small (untested) POC here: c27ab2f

• It puts the batch logic in a helper, making it very easy to adopt in custom mappers
• All old code should still work
• This POC forces people with custom mappers to add the BatchHelper, but we could work around that (for example with the suggestion in my opening comment).

I also changed the batch name from the previously suggested records_to_events to each_record_to_event. I think this is more intention revealing and a much lower chance of being read wrong.

[Bertg]

@mpraglowski If you think this is a good way to go, I can work it out into a full PR

[mpraglowski]

I still don't have strong opinion on this solution. But don't let this to stop you from contributing to the RES project - please send the PR and it could be the solution or starting point to work on final solution :)

[Bertg]

Created a PR here: #1854