How to integrate Auron with RSS (Celeborn, Uniffle) ?

[merrily01]

Integrating Auron with Remote Shuffle Services (RSS)

Auron supports integration with external Remote Shuffle Services to improve shuffle performance and scalability.

Currently, the following versions are supported: Apache Celeborn 0.5.4 and 0.6, and Apache Uniffle 0.9.2.

Apache Celeborn

Auron can work with Celeborn as a shuffle manager. Integration involves configuring Auron/Spark to use the AuronCelebornShuffleManager and pointing it to the appropriate Celeborn master endpoints and storage locations. This allows Spark jobs running on Auron to leverage Celeborn for distributed shuffling.

You can integrate using the following example configuration:

# change celeborn endpoint and storage directory to the correct location
spark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.celeborn.AuronCelebornShuffleManager
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.celeborn.master.endpoints localhost:9097
spark.celeborn.client.spark.shuffle.writer hash
spark.celeborn.client.push.replicate.enabled false
spark.celeborn.storage.availableTypes HDFS
spark.celeborn.storage.hdfs.dir hdfs:///home/celeborn
spark.sql.adaptive.localShuffleReader.enabled false

Apache Uniffle

Similarly, Auron supports Uniffle, a high-performance remote shuffle service. To enable Uniffle, you need to configure Auron/Spark to use the AuronUniffleShuffleManager and specify the Uniffle coordinator endpoints.

You can integrate using the following example configuration:

spark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.uniffle.AuronUniffleShuffleManager
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.rss.coordinator.quorum <coordinatorIp1>:19999,<coordinatorIp2>:19999
spark.rss.enabled true

Notes

1. Ensure the relevant RSS client JARs are included in your Spark application's classpath.

2. Replace endpoints and directories with the actual addresses in your cluster.

3. For detailed setup and advanced configuration, refer to the official documentation for the respective

• Celeborn
• Uniffle

[merrily01]

The detailed solution is in the post.