Custom component node in RayPipeline

[alex-nork]

Is it possible to include a custom component, one which inherits from BaseComponent, in a RayPipeline? I see that components are registered when imported so I'm using a helper function that imports all custom components in a given directory. That works fine with a regular Pipeline, but there seems to be a problem registering custom components with a RayPipeline.

Since RayPipeline can only be configured via YAML I can't just add nodes either. Here's the stacktrace I'm getting from ray:

(ServeController pid=46961) 2022-12-21 15:50:33,275	ERROR worker.py:400 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::ServeReplica:Downloader.reconfigure() (pid=47118, ip=..., repr=<ray.serve._private.replica.ServeReplica:Downloader object at 0x1a2472bf0>)
(ServeController pid=46961)   File "/private/var/alex/.pyenv/versions/3.10.8/lib/python3.10/concurrent/futures/_base.py", line 451, in result
(ServeController pid=46961)     return self.__get_result()
(ServeController pid=46961)   File "/private/var/alex/.pyenv/versions/3.10.8/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
(ServeController pid=46961)     raise self._exception
(ServeController pid=46961)   File "/tmp/ray/session_2022-12-19_12-52-31_464653_85100/runtime_resources/pip/01e37659cb2471029d4b3a822b13eeb3794cc8c5/virtualenv/lib/python3.10/site-packages/ray/serve/_private/replica.py", line 224, in reconfigure
(ServeController pid=46961)     await self._initialize_replica()
(ServeController pid=46961)   File "/tmp/ray/session_2022-12-19_12-52-31_464653_85100/runtime_resources/pip/01e37659cb2471029d4b3a822b13eeb3794cc8c5/virtualenv/lib/python3.10/site-packages/ray/serve/_private/replica.py", line 146, in initialize_replica
(ServeController pid=46961)     await sync_to_async(_callable.__init__)(*init_args, **init_kwargs)
(ServeController pid=46961)   File "/tmp/ray/session_2022-12-19_12-52-31_464653_85100/runtime_resources/pip/01e37659cb2471029d4b3a822b13eeb3794cc8c5/virtualenv/lib/python3.10/site-packages/haystack/pipelines/ray.py", line 293, in __init__
(ServeController pid=46961)     self.node = self.load_from_pipeline_config(pipeline_config, component_name)
(ServeController pid=46961)   File "/tmp/ray/session_2022-12-19_12-52-31_464653_85100/runtime_resources/pip/01e37659cb2471029d4b3a822b13eeb3794cc8c5/virtualenv/lib/python3.10/site-packages/haystack/pipelines/ray.py", line 318, in load_from_pipeline_config
(ServeController pid=46961)     component_instance = BaseComponent._create_instance(
(ServeController pid=46961)   File "/tmp/ray/session_2022-12-19_12-52-31_464653_85100/runtime_resources/pip/01e37659cb2471029d4b3a822b13eeb3794cc8c5/virtualenv/lib/python3.10/site-packages/haystack/nodes/base.py", line 157, in _create_instance
(ServeController pid=46961)     subclass = cls.get_subclass(component_type)
(ServeController pid=46961)   File "/tmp/ray/session_2022-12-19_12-52-31_464653_85100/runtime_resources/pip/01e37659cb2471029d4b3a822b13eeb3794cc8c5/virtualenv/lib/python3.10/site-packages/haystack/nodes/base.py", line 126, in get_subclass
(ServeController pid=46961)     raise PipelineSchemaError(f"Haystack component with the name '{component_type}' not found.")
(ServeController pid=46961) haystack.errors.PipelineSchemaError: Haystack component with the name '<custom_component>' not found.

[julian-risch]

Hi @alex-nork thank you for bringing up this topic. We have a related issue here: #3770 but it's not about Ray pipelines in particular. I'll check with my colleagues whether they can help here.

[zoltan-fedor]

@julian-risch Any update on this?

I have a nasty workaround to make the custom nodes to run in the RayPipeline, but it would be nice to have an official method.

[mkumar3010]

@jzoltan-fedor - could you please let us know the work around for the custom nodes to run in Raypipeline. Appreciate your help

[zoltan-fedor]

Basically you can write your custom modules as a library and then just use Ray's py_modules option to upload them to Ray.
See https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#library-development

Then you need to import the custom modules locally before initializing the RayPipeline - so Haystack's RayPipeline won't complain about unknown components and then on the Ray cluster you need to patch Haystack to load those same custom modules before Haystack would be using them.