rethink publication handling via zotero

[stefanprobst]

currently, publications are not stored in the DKB database, but in zotero.

the DKB app uses the zotero API

• to retrieve publications for a specific country for a specific reporting year. each country has their own collection in the dariah zotero group, which can be queried separately via API.
• to retrieve all publications for a specific reporting year. the sum is displayed in the yearly "statistics" screen in the admin dashboard.

however, the zotero API is not only slow and prone to timeout errors - the biggest issue is that it does not allow filtering by publication year via API.

currently, we have to retrieve all items for a country collection (or the whole group, in case of the "statistics" admin screen), and filter the full result set by publication year ourselves.

this approach is not sustainable, as each year, the size of the zotero collections will grow, thus the required API requests will increase, leading to more timeouts and slow responses.

open to suggestions how to solve this.

[babslgam]

create subcollections per year maybe?

[stefanprobst]

also good to clarify how these collections are being used, i.e. whether national nodes manage their bibliographies in these collections in the DARIAH zotero group, or if they have their separate zotero collections and just sync relevant entries to the DARIAH one.

[vronk]

this performance problem will be growing every year..
because filtering by year is not possible via zotero API.
one option would be to create subcollections per year pre country in the DARIAH zotero-library.

But maybe handling of publications should be rethought principally.

[vronk]

considering ingesting Zotero into TypeSense and then using information from Typesense for UNR

[vronk]

considering ingesting Zotero into TypeSense and then using information from Typesense for UNR

see DARIAH-ERIC/dariah-website#6

[babslgam]

if we use typesense we have to communicate that changes in zotero won't be reflected in reports immediately.

#198