Performance Optimize Experiment

[dameng324]

Motivation

In Protobuf, when serializing a message, a MessageLength (encoded as a Varint) must be written before the message body.
This requires two serialization passes — one to compute the MessageLength, and another to actually write the message.

Although the size calculation step has been highly optimized, it can still incur noticeable overhead when messages are deeply nested.

This PR aims to explore the possibility of performing serialization in a single pass — that is, writing the message only once, while determining its length during the write process, and then backfilling the MessageLength field afterward.

---

Current Progress

The first challenge encountered was that Varint is a variable-length format, so we cannot know in advance how many bytes to reserve for the message length.

This issue was solved in a somewhat hacky but effective way:
by always reserving 5 bytes for the Varint and encoding it in a way that remains compatible with both protobuf-net and Google.Protobuf decoders.

Although this encoding is technically non-standard, it is still valid according to the Protobuf Varint decoding rules.

public void WriteLength(Span<byte> span, int value)
{
    span[0] = (byte)((value & 0x7F) | 0x80);
    value >>= 7;
    span[1] = (byte)((value & 0x7F) | 0x80);
    value >>= 7;
    span[2] = (byte)((value & 0x7F) | 0x80);
    value >>= 7;
    span[3] = (byte)((value & 0x7F) | 0x80);
    value >>= 7;
    span[4] = (byte)((value & 0x7F) | 0x00);
}

So far, everything worked quite well — until the next issue arose.

---

The Main Challenge

The IBufferWriter interface only provides GetSpan/GetMemory and Advance methods.
When we reserve space for the message length using GetSpan, this memory is no longer guaranteed to be valid once Advance is called.

Due to the buffer writer’s internal growth mechanism, the reserved span for MessageLength might become invalid when writing the message body — meaning the memory could have been reallocated or replaced.
By the time we want to backfill the length, the original span may refer to outdated memory, and there’s no reliable way to access the new span corresponding to that same position.

---

Solutions Considered

Use a custom IBufferWriter implementation.
Not feasible, since this serializer aims to work with any user-provided IBufferWriter.

Use unsafe code to access the pointer of the message body and move backward to the length prefix region.
Unsafe and unreliable — there’s no guarantee that the length prefix and message body reside in contiguous memory.

Write the message body into a temporary buffer first, compute its length, then write both length and body to the user’s writer.
This works functionally but introduces extra allocations and copying, which can be expensive for large or deeply nested messages — essentially defeating the purpose of this optimization.

---

Request for Feedback

At this point, the PR is paused here.
If anyone has better ideas or knows of an elegant way to solve this problem while maintaining compatibility with the IBufferWriter interface, I would love to hear your thoughts.

Thank you for reading and for any suggestions or insights you might share.