Switch from `rust_protobuf` to `prost` for protobuf generation

[tiziano88]

ref:

• https://github.com/stepancheg/rust-protobuf
• https://github.com/danburkert/prost

We already have a few PRs that switched parts of the source tree to use prost; if we prefer that, we should go ahead and switch to it fully.

pros:

• no_std compatible (almost)
• more idiomatic and compact generated code

cons:

• will likely require re-writing the custom gRPC generation logic
  • this may actually be implemented in a more idiomatic way than currently by using https://docs.rs/prost-build/0.6.1/prost_build/trait.ServiceGenerator.html
• not sure whether it will be harder to do things using proto extensions or attributes, if we decided to use them

[tiziano88]

@blaxill assigning to you since you are already working on it with #683

[blaxill]

@tiziano88 rust_protobuf is no longer used for the core components, but is still used in the examples through sdk/rust/oak_utils. Do you want to include changing the examples in this issue?

[tiziano88]

Good question, now that we don't have a hard requirement to be no_std compatible in the short term, perhaps we should reconsider this decision based on other factors, for instance which API and code generator we prefer. Given we already have gRPC code generation for rust_protobuf, perhaps we should keep that and not worry about prost?

@daviddrysdale @conradgrobler @rbehjati WDYT?

[conradgrobler]

I personally prefer Prost, but that could just be because i am more familiar with it.

Also, Prost is used by Tonic, so if we plan on using that in future as a gRPC server it might be more natural to use Prost. But, if we are just going to proxy the request body bytes to the nodes, then it probably doesn't matter and the nodes can decide how they interpret it.

[rbehjati]

I think it is not quite maintainable to have both. I already ran into some problems with conflicting blanket implementations, when trying to use Prost for LogMessage. So, I think, eventually we should either change the sdk and the examples to use Prost, or change the core components to use rust_protobuf. (I am assuming that the sdk and the core components are not completely disjoint, and may want to share some code in the future.)

Having said that, I don't really have a preference. Although with my very limited experience, I found Prost to be a bit neater (I did not look into code generation for gRPC).

[tiziano88]

@wildarch as part of your experiments, did you figure out whether it is possible to rely on FieldOptions from prost? I thought it would not be possible because they are usually represented as extensions, which prost does not support, but now I looked at the generated files that prost uses, and it seems perhaps they just stash the options into a regular field:

https://github.com/danburkert/prost/blob/6f3c60f136be096194a3c6e0e77e25a9e1356669/prost-types/src/protobuf.rs#L564-L565

Could you try to define a custom option, assign it to a field, and see whether during code generation time (i.e. as part of running prost itself), the option is visible? If so, I think that means that we should be able to rely on extensions to mark fields as handles, if we wanted to, though of course that would require patching / forking prost.

[wildarch]

I am actually in the process of doing exactly this! For now I got stuck at the situation described in https://github.com/danburkert/prost/issues/256, where for some weird reason the uninterpreted_option field remains empty. Still debugging that issue, but I think once that is fixed we can use Prost!

[tiziano88]

@wildarch did you try to dump the incoming message that protoc itself sends to the prost plugin?

[wildarch]

I did, and unfortunately it doesn't contain the custom option.

[tiziano88]

Do you have the code you used to test that? I wonder if we could try something similar using rust-protobuf and see if there is any difference at that level already.

[wildarch]

I have put the code up at https://github.com/wildarch/handle_extract.

It contains a dump of the file descriptor set that prost-build uses to generate rust types: https://github.com/wildarch/handle_extract/blob/master/descriptor_set.textproto. Note line https://github.com/wildarch/handle_extract/blob/master/descriptor_set.textproto#L10763 specifically, which shows an options message that is set, but does not contain any elements.
I have yet to contrast this with how rust-protobuf does this.

[tiziano88]

What if you set a standard option on that field, e.g. deprecated = true?