Towards an even larger SYNERGY dataset

[J535D165]

SYNERGY is a free and open dataset on study selection in systematic reviews, comprising 169,288 academic works from 26 systematic reviews. Although this is already a large dataset compared with other resources, we aim to develop an even larger dataset: SYNERGY+ (SYNERGY Plus). This dataset contains even more records, datasets, a test-train split, and many more improvements. For the development of this dataset, we would appreciate your help. In the upcoming weeks, we will update this thread with information and instructions on how to contribute to this dataset.

Subscribe to this thread to keep up-to-date 🙏

[sidgoogle]

Hi, just curious if there is an update. Thanks for the valuable resource.

[EmilyWes]

Hi sidgoogle,

We are long due for an update, so thank you for the question :).

• We have started actively gathering more datasets, both by searching online and reaching out to people.
• We are standardizing and improving the current pipeline for making those datasets available. A number of changes are already in (see the development/v2 branch). When this is done we will start adding datasets, of which a number are already finalized.
• We are also busy refining and extracting information about the datasets, which we plan add over time. An example is the addition of the eligibility criteria, that can be found here. Other things are labels for abstract screening and indicators of various quality criteria of the datasets.

So keep an eye on mainly the development/v2 branch, where most of this will be added first. And feel free to come with any suggestions / possible improvements :).

[SamyAteia]

Are the studies from the CLEF TAR challenge included in the SYNERGY dataset, or are there plans to include them?
https://github.com/CLEF-TAR/tar

[EmilyWes]

Hi SamyAteia,

Thank you for the suggestion!
I have actually looked in to the CLEF dataset. Reading back my notes I understood that the creators of the dataset reproduced the search(es) rather than having the original list of identifiers or DOI's. Since we found that reproducing a search always gives different results, we did not trust the records of the dataset enough to include it in SYNERGY.
However if I understood incorrectly, please let met know so that I can check it again :).

Also nice to know: we also plan to put an overview of the reviews/data we have looked at here on the github repo, but are still deciding what would be the format for that. Reasoning why something is included/excluded in Synergy will definitely be information provided!

[Mohammadsaknini]

I have collected some data in a similar fashion to yours. Feel free to add it to your dataset Repo

[EmilyWes]

Thank you very much! I will look into it and let you know if I have any questions :)

[puzzledhedgehog]

Hi folks -- very interesting project!
Q1 I see that the title-abstract screening decision is optional; only the full-text screening decision must be included. Is that wise? In many cases, a record can't be excluded by humans based on the title and abstract; only when the full text is considered can the paper be excluded. I don't see how we can reasonably expect a ML tool that doesn't have access to the full text to make accurate full-text screening decisions. (OK, I will admit that in simulations, I was able to get higher performance from ASReview when I tried to achieve 95% recall of the included-after-full-text-screening records, rather than 95% recall of the not-yet-excluded-after-title-abstract-screening records. But the edge cases still worry me.)
Q2 Would you be interested in knowing whether a given document that was screened in or out was identified via database searching or via other methods (citation chaining, hand-searching, etc)?

[EmilyWes]

Hi there,

Q1:
I split the question in to 2 parts:

• Why are full-text labels required and title-abstract (ti-ab) labels not?
  Ideally we want both the full-text and the title-abstract labels, but as the main goal for a systematic review is to find the full text inclusions, we need those labels to say something about how well ASReview performs. The title-abstract labels enable a lot more ways to simulate and test strategies, so when published, we include these labels as well.
  Note that in general the title-abstract labels are a lot more noisy, making the full-text labels a better candidate to use.

• "I don't see how we can reasonably expect a ML tool that doesn't have access to the full text to make accurate full-text screening decisions."
  ASReview is a ranking tool. It helps the screener to look at the most promising records first, but it won't make a decision about a record itself. So the task is to find the records that are most likely a full-text inclusion, for which we have seen in a lot of cases it can do very well.

Q2:
That's a very good question. For the new improved version of SYNERGY we are looking at these type of things. Currently, that means registering how many records are found via these methods, and how many are in the actual data published online.
For most datasets, it is the case that the records published online are those from the database searching only.

[J535D165]

I would like to add to this the following:

Synergy contains an extensive set of metadata fields, the title and abstract of which are the most well-known and used. You can find more information on the available metadata fields in the project's README. These metadata fields can be used to derive rich features for classification. For example, authors, keywords, publication year, and concepts are available. You can even access the open-access PDF or link to the full text. Synergy is just the start of rethinking how we conduct systematic reviews or sparse data retrieval for academic literature. We would love to hear about your creative ideas.