Migrating from FARM to Haystack: File_to_dicts NotImplemented for TextClassificationProcessor

[brycecf]

This is building on #3239, which was closed with the possibility of a future discussion. @ZanSara if you're still willing :)

Currently, my team is using DataSilo to split the data and feed it to a TextClassificationProcessor.

from farm.data_handler.data_silo import DataSilo
from farm.data_handler.processor import TextClassificationProcessor

processor = TextClassificationProcessor(
                tokenizer=tokenizer,
                max_seq_len=self.config.max_seq_length,
                data_dir=data_loc,
                label_list=label_list,
                metric=self.config.metric,
                label_column_name=label_field,
            )

...

data_silo = DataSilo(
            processor=processor,
            batch_size=config.batch_size,
            max_multiprocessing_chunksize=training_config.max_multiprocessing_chunksize,
            max_processes=training_config.max_processes
        )

Consequently, because TextClassificationProcessor no longer has an implemented file_to_dicts, an exception is raised. If there is interest, we are interested in providing an implementation, but the below quote (from the above issue) suggests there is a more Haystack appropriate solution:

I've rarely seen users use DataSilo directly with Haystack. That class comes directly from FARM where it had a much clearer scope. I'd dare saying DataSilo is in Haystack to support FARMReader and other related classes, not to be used as user-facing API.

If so, what should that look like now?