Removed incorrect not-entailed examples in VerbNet
The method to recast VerbNet annotations/examples into NLI created an issue in some examples. Here we provide a brief example and below we include statistics about the change.
Example
Consider the sentence Jackie chased the thief . which comes from the example in VN class chase-51.6. The semantics for this example from VN are motion(during(E), Agent) motion(during(E), Theme) indicating that both the Agent, i.e. Jackie, and the Theme, i.e. the thief moved. To create negative, i.e. not-entailed examples, we swapped thematic roles in our templates.
Consequently, this example sentence would then incorrectly label the example where the hypothesis would be The thief moved as not-entailed. In this version/release of the dataset, we removed such examples
Stats
| train | dev | test | |
|---|---|---|---|
| unique contexts | 23 | 6 | 2 |
| all unique pairs | 1386 | 143 | 160 |
| bad pairs | 43 | 11 | 4 |
| % bad pairs | 3.10 | 7.69 | 2.50 |
| ------------------ | ------- | ------ | ------ |
| # v0.1 examples | 1441 | 154 | 164 |
| # v0.1.1 examples | 1398 | 143 | 160 |