ConsumeWorker: add abstraction on channels

[apfitzge]

Problem

• For scheduler bindings we largely will do the exact same processing loop as our normal workers. But the current workers are tied to using crossbeam channels.
• We don't (yet) want to migrate the internal to using shaq until vetted further
• We don't want to duplicate all the code/metrics reporting just to effectively swap out the type of channel we're using.

Summary of Changes

• Abstract a pair of channels for try_recv/try_send
• Implement the abstraction for crossbeam channels

Future plan

• Add support for Resolving
• Add implementation (and translation) for scheduler-bindings message types:
  • pointer->transaction translation, etc.
  • these actions will be done within the try_recv/try_send fns

Fixes #

[apfitzge]

note: for crossbeam these are not necessary to be mut. However for shaq we update some local usizes internally.

Likely when we get to point of using shaq we may need to add something like sync/finalize. Don't need those with crossbeam either, but trying to keep this PR simpler. We don't want to call them every time we recv/send for efficiency.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 96.36364% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.1%. Comparing base (fe56408) to head (0514b44).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #8483   +/-   ##
=======================================
  Coverage    83.1%    83.1%           
=======================================
  Files         840      840           
  Lines      367637   367659   +22     
=======================================
+ Hits       305859   305879   +20     
- Misses      61778    61780    +2     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.