document_identifier in annotation tool

[dm1618]

Hi, I have a question about importing documents in the Haystack annotation tool.

As a dummy example, let's say I create a Pandas dataframe then convert to csv as follows:

test_df = pd.DataFrame()
test_df['document_identifier'] = [123, 456]
test_df['document_text'] = ['foo', 'bar']
test_df.to_csv(‘test.csv’, index=False)

Then I perform the CSV batch upload for the ‘test.csv’ file (under the Import -> Documents tab in the annotation tool). This is the result:

As you can see, the Id column in this table does not match the document_identifier column in my Pandas dataframe. When I try creating labels and exporting them in SQUAD format, the document_id in the output is equal to the Id in the above table, not the document_identifier in my Pandas dataframe.

I’ve also tried converting the document_identifier to a string, but this didn’t work.

Could you please let me know if it is possible to make the Id equal to document_identifier? If so, how can I amend my current code to make this work? This would be very helpful for linking the table of documents I want to annotate to the documents uploaded in the annotation tool.

Thanks in advance for your help (and for your work on Haystack!)

[bilgeyucel]

Hi @dm1618, sorry for the late answer. Unfortunately, it's not possible to make the id equal to document_identifier. id is assigned by the database, and document_identifier is used to refer to a document in the annotation tool for example when you want to also import questions for documents.

[dm1618]

Thanks for getting back to me!