How about writing data to databend via flink ？

[artorias1024]

If I implement a flink connector, is it possible to bypass the compute layer and write data directly to storage, similar to hudi?

[xudong963]

Hi @artorias1024, I think we can directly use flink.streaming.api.functions.sink.RichSinkFunction.
I have done some small experiments before, flink->clickhouse, I think by this idea, flink->databend is also possible. https://github.com/xudong963/realtime-system/blob/master/src/main/java/org/data/dataflow/FlinkToCK.java.
Please try it, feel free to ping me when you encounter problems!

[xudong963]

Hi @artorias1024 , how it's going？

[artorias1024]

My idea is to avoid building the data written to DataBend via SQL parsing, which would introduce some performance loss (in batch processing)? Is it possible to write directly to the storage layer by means of RPC directly?

[BohuTANG]

Makes sense, but databend there is no storage RPC yet, any ideas on the RPC API design?
Thanks.

[artorias1024]

So far I have only a very rough idea that the API for the storage layer should be relatively simple. For example, the interface exposed to the public should be similar to 'Put', 'Get', 'Scan'. For Databend, these APIs need to get the metadata information, which can be obtained through Meta-Server. For Flink, different data sources are abstracted into different 'Sources', which are not only used for reading data, but also involved in Join for streaming computation.