How can a pipeline connection or component in a pipeline be updated?

[SurealCereal]

Take for example the ElasticsearchDocumentStore - once its created, the index cannot be changed. That's fine, but I don't see a clear way to remove the component from the pipeline to replace it for example. Pipeline has add_component() but no clear way to remove an existing component. I've tried digging into protected functions and the pipeline graph, but everything gets very heavily interconnected with dynamic properties being added to components like node.data['instance'].__haystack_input__._sockets_dict and more.

[mrm1001]

Hi @SurealCereal are you trying to change the definition of a pipeline at runtime? Or are you trying to deserialise a pipeline and then change its definition?

[SurealCereal]

Hi @mrm1001. I'd like to change the definition of a pipeline (update or swap components) at runtime. Serialization/de-serialization was not involved.

[vblagoje]

@SurealCereal we don't support runtime graph changes. Perhaps you can use ConditionalRouter to route your data through different parts of already built graph

[SurealCereal]

@vblagoje Conditional routing is not going to help when I need to have dynamic Elastic search indexes, i.e new ones can be created at runtime. Is there some way to destroy/free the entire graph? At present trying to create a new graph results in component name conflicts, not to mention the memory that is not freed.