ParisReproducibility

Discussion on Reproducibility from the 6th DELPH-IN Summit

Moderator: StephanOepen; Scribe: FrancisBond

Start with mail from Rebecca:

Now we are slowly seeing more DELPH-IN work getting published, it
should be easier to tick off the requirement for "comparison to
previous work", but that is not as easy as it could be.  One thing
that would make it easier is to have an up-to-date publications list,
perhaps even by topic, so anyone who gets, for example, a parse
selection paper accepted adds it to the wiki under parse selection.
Another, more necessary thing would be using comparable data.  Many of
our papers seem to use slightly different (often unspecified) test and
training sets, even when we are using the same basic data. A few
suggestions for that problem:

* clearly document test and training splits on released profiles
* where possible, stick to previously used data sets
* when creating new corpora, consider creating a held-out test set
(perhaps by taking out every x item) and designating it as testing
from the start
* annotate the publication list above with technical details that
weren't perhaps relevant to the paper, but would help someone else
replicate the results (data set, SVN version, cheap options etc)

I'd like to see a discussion of whether any of the above suggestions
could work, and also whether there are other practical suggestions to
help get more DELPH-IN work published (eg particular results that need
to be published so they can be referenced?)