Reaching out from the Ray Team!

[richardliaw]

Search before continuing 先搜索，再继续

• I have searched the Data-Juicer issues and found no similar feature requests. 我已经搜索了 Data-Juicer 的 issue 列表但是没有发现类似的功能需求。

Description 描述

Hi! This is a really exciting project. I'm one of the authors of Ray, and I am currently working on Ray Data. We're seeing more users mention Data-Juicer as a key library on top of Ray.

I'd love to better understand how we can improve Ray Data for your users and use cases.

In particular, I noticed that you might not be using some of the recent Ray Data improvements (hash-shuffle, joins) that were introduced in 2.46.

It also seems like data-juicer has a lot of standalone operators; I wonder if we can feature some of these as examples on the Ray Data side in a streaming execution pipeline.

If you're interested, we'd love to connect (perhaps on email, or wechat). Let us know!

Use case 使用场景

No response

Additional 额外信息

No response

Are you willing to submit a PR for this feature? 您是否乐意为此功能提交一个 PR？

• Yes I'd like to help by submitting a PR! 是的！我愿意提供帮助并提交一个PR！

[cyruszhang]

Thanks for reaching out, Richard! We also believe the synergy between Ray and DataJuicer provides a great joint adventure!

One of the biggest challenges we face in massive parallel data processing is how to make it more fault tolerant. One op runs into an issue and our whole pipeline goes to moot and have to be started from scratch. We are actively working on implementation of checkpointing/partitioning/event-logging on top of Ray to solve the problem. Do you see any features in Ray that might help in this area?

[richardliaw]

would love to better understand your failure modes here; checkpointing will help but it is a big hammer and I think some of failure recovery should be taken care of by ray data transparently.

do you have a couple situations or stacktraces I can look at?

[cyruszhang]

here is our PR for comprehensively handling the fault tolerance and job life cycle management; ray has infrastructure level fault tolerance, we are building on top of that
#748

the key contributions:

1. introduce dj job as a first class citizen; one can monitor, stop, resume jobs
2. partition of input data to achieve Independent Partition Execution
3. DAG backed execution planning and tracking
4. checkpointing (with strategies of different granularities) to aid intermediate data persistence and job resumption/robustness
5. event logging infrastructure to aid all the above

let me know if you see something that ray already provides in some capacity so we can piggy back on