Add eraser_multi_rc dataset

PiperOrigin-RevId: 291732980

Author: TensorFlow Datasets Team

tensorflow_datasets/testing/test_data/fake_examples/eraser_multi_rc/multirc/docs/test1.txt

@@ -0,0 +1,31 @@
+The judge leaned back in his chair and beckoned to Mr. Andrews .
+It was finished .
+Spear was free , and from different parts of the courtroom people were moving
+toward the door .
+Their numbers showed that the friends of the young man had been many .
+Mr. Thorndike felt a certain twinge of disappointment .
+Even though the result relieved and pleased him , he wished , in bringing it
+about , he had had some part .
+He begrudged to Isaacs & Sons the credit of having given Spear his liberty .
+His morning had been wasted .
+He had neglected his own interests , and in no way assisted those of Spear .
+He was moving out of the railed enclosure when Andrews called him by name .
+" His honor , " he said impressively , " wishes to speak to you . " The judge
+leaned over his desk and shook Mr. Thorndike by the hand .
+Then he made a speech .
+The speech was about public - spirited citizens who , to the neglect of their
+own interests , came to assist the ends of justice , and fellow - creatures in
+misfortune .
+He purposely spoke in a loud voice , and every one stopped to listen .
+" The law , Mr. Thorndike , is not vindictive , " he said .
+" It wishes only to be just .
+Nor can it be swayed by wealth or political or social influences .
+But when there is good in a man , I , personally , want to know it , and when
+gentlemen like yourself , of your standing in this city , come here to speak a
+good word for a man , we would stultify the purpose of justice if we did not
+listen .
+I thank you for coming , and I wish more of our citizens were as unselfish and
+public - spirited . " It was all quite absurd and most embarrassing , but
+inwardly Mr. Thorndike glowed with pleasure .
+It was a long time since any one had had the audacity to tell him he had done
+well .

tensorflow_datasets/testing/test_data/fake_examples/eraser_multi_rc/multirc/docs/train1.txt

@@ -0,0 +1,34 @@
+As his car slid downtown on Tuesday morning the mind of Arnold Thorndike was
+occupied with such details of daily routine as the purchase of a railroad , the
+Japanese loan , the new wing to his art gallery , and an attack that morning ,
+in his own newspaper , upon his pet trust .
+But his busy mind was not too occupied to return the salutes of the traffic
+policemen who cleared the way for him .
+Or , by some genius of memory , to recall the fact that it was on this morning
+young Spear was to be sentenced for theft .
+It was a charming morning .
+The spring was at full tide , and the air was sweet and clean .
+Mr. Thorndike considered whimsically that to send a man to jail with the memory
+of such a morning clinging to him was adding a year to his sentence .
+He regretted he had not given the probation officer a stronger letter .
+He remembered the young man now , and favorably .
+A shy , silent youth , deft in work , and at other times conscious and
+embarrassed .
+But that , on the part of a stenographer , in the presence of the Wisest Man in
+Wall Street , was not unnatural .
+On occasions , Mr. Thorndike had put even royalty— frayed , impecunious royalty
+, on the lookout for a loan — at its ease .
+The hood of the car was down , and the taste of the air , warmed by the sun ,
+was grateful .
+It was at this time , a year before , that young Spear picked the spring flowers
+to take to his mother .
+A year from now where would young Spear be ?
+It was characteristic of the great man to act quickly , so quickly that his
+friends declared he was a slave to impulse .
+It was these same impulses , leading so invariably to success , that made his
+enemies call him the Wisest Man .
+He leaned forward and touched the chauffeur 's shoulder .
+" Stop at the Court of General Sessions , " he commanded .
+What he proposed to do would take but a few minutes .
+A word , a personal word from him to the district attorney , or the judge ,
+would be enough .

tensorflow_datasets/testing/test_data/fake_examples/eraser_multi_rc/multirc/docs/train2.txt

@@ -0,0 +1,34 @@
+As his car slid downtown on Tuesday morning the mind of Arnold Thorndike was
+occupied with such details of daily routine as the purchase of a railroad , the
+Japanese loan , the new wing to his art gallery , and an attack that morning ,
+in his own newspaper , upon his pet trust .
+But his busy mind was not too occupied to return the salutes of the traffic
+policemen who cleared the way for him .
+Or , by some genius of memory , to recall the fact that it was on this morning
+young Spear was to be sentenced for theft .
+It was a charming morning .
+The spring was at full tide , and the air was sweet and clean .
+Mr. Thorndike considered whimsically that to send a man to jail with the memory
+of such a morning clinging to him was adding a year to his sentence .
+He regretted he had not given the probation officer a stronger letter .
+He remembered the young man now , and favorably .
+A shy , silent youth , deft in work , and at other times conscious and
+embarrassed .
+But that , on the part of a stenographer , in the presence of the Wisest Man in
+Wall Street , was not unnatural .
+On occasions , Mr. Thorndike had put even royalty— frayed , impecunious royalty
+, on the lookout for a loan — at its ease .
+The hood of the car was down , and the taste of the air , warmed by the sun ,
+was grateful .
+It was at this time , a year before , that young Spear picked the spring flowers
+to take to his mother .
+A year from now where would young Spear be ?
+It was characteristic of the great man to act quickly , so quickly that his
+friends declared he was a slave to impulse .
+It was these same impulses , leading so invariably to success , that made his
+enemies call him the Wisest Man .
+He leaned forward and touched the chauffeur 's shoulder .
+" Stop at the Court of General Sessions , " he commanded .
+What he proposed to do would take but a few minutes .
+A word , a personal word from him to the district attorney , or the judge ,
+would be enough .

tensorflow_datasets/testing/test_data/fake_examples/eraser_multi_rc/multirc/docs/train3.txt

@@ -0,0 +1,32 @@
+He continued to lie in place , still breathing heavily .
+I tried to keep myself calm .
+Then , with a sigh , he said , " I have need of a certain stone that I once
+possessed .
+It would allow me to leave this bed , despite my current illness .
+My problem is that it was not made in this realm , but by a human , and only a
+human may handle it .
+The herb vendor will tell you where you can find it .
+If you would fetch me this stone , I would tell you what I can . " " Of course
+we will fetch you the stone , " Nepthys assured him .
+" You have our thanks . " Outside , the thing at the herb cart nodded when we
+mentioned the stone .
+He told Nepthys the name of a woman who lived on a certain street , and we set
+off .
+When we reached the street , I expected some sort of temple , or a row of
+antiquaries , or at least a sleazy , black - market relics merchant .
+Instead , it looked like we were in the red - light district : women who
+probably wanted more than our money eyed us from the alleys .
+Nepthys did n't need to warn me not to speak to them .
+He took us to the door of a building like a short pagoda that had a sign hanging
+over it showing a face with hands uplifted , like a saint under G d 's light ,
+and handcuffed .
+Inside , the entrance room was hung with bright silks and scattered with velvet
+cushions .
+Sails of obscene orange and a green that insulted springtime hung draped across
+reds that might have been sensual elsewhere but here were only offensive .
+A large woman , similarly decorated , quickly drifted up to us .
+" Good evening , gentlemen , " she said .
+" May I invite you to have a cup of tea ? " " No , thank you , " said Nepthys ,
+" we regret that we can not take up your kind offer .
+Are you the proprietor of this shop ? " " Yes , " answered the woman .
+" I am Madame Entera , at your service .

tensorflow_datasets/testing/test_data/fake_examples/eraser_multi_rc/multirc/docs/valid1.txt

@@ -0,0 +1,24 @@
+Allan sat down at his desk and pulled the chair in close .
+Opening a side drawer , he took out a piece of paper and his inkpot .
+After filling his pen , Allan looked at his paper in the orange glow from the
+lantern set back in the desk 's right - hand corner .
+His pen cast a forbidding line of shadow slanting across the page , echoing the
+inky darkness crouching in the edges of the lantern 's struggling glow .
+The only other illumination came from a lurid moonlight filtered through thin
+branches and clouds , casting its bone - pale glow onto the pine floorboards .
+Allan unfolded another page , this one crowded with ranks of letters in tight
+formation from left to right .
+The lines of letters stepped into their divisions , in the shape of a story 's
+outline : the loose , dry skeleton of a tale lay exposed beneath their feet ,
+awaiting tendons , muscle and blushing skin .
+Allan reviewed the troops , all prepared to disembark , their task to form the
+tale of a young man returning home from Life Abroad to find his childhood friend
+a bride to - be , thus upsetting the apple cart of his life 's plan , clarified
+– of course – by his very time away from her he loved best .
+Although the concept was a simple one , Allan thought it had potential .
+Besides , the public liked a good , simple romance .
+Perhaps this will be more saleable , he thought and began to write .
+They gazed at each other , lost in the rapture of love based so deeply within
+their hearts that they had never seen it before .
+" What about Roger ? "
+she asked , knowing that the answer no longer mattered .

tensorflow_datasets/testing/test_data/fake_examples/eraser_multi_rc/multirc/test.jsonl

@@ -0,0 +1 @@
+{"annotation_id": "Fiction-stories-masc-A_Wasted_Day-11.txt:0:0", "classification": "True", "docids": null, "evidences": [[{"docid": "test1.txt", "end_sentence": 15, "end_token": 233, "start_sentence": 14, "start_token": 218, "text": "\" The law , Mr. Thorndike , is not vindictive , \" he said ."}, {"docid": "test1.txt", "end_sentence": 16, "end_token": 241, "start_sentence": 15, "start_token": 233, "text": "\" It wishes only to be just ."}, {"docid": "test1.txt", "end_sentence": 17, "end_token": 254, "start_sentence": 16, "start_token": 241, "text": "Nor can it be swayed by wealth or political or social influences ."}]], "query": "What did the judge tell Mr. Thorndike about the law ? || Cannot be swayed by wealth or political or social influences", "query_type": null}

tensorflow_datasets/testing/test_data/fake_examples/eraser_multi_rc/multirc/train.jsonl

@@ -0,0 +1,3 @@
+{"annotation_id": "Fiction-stories-masc-A_Wasted_Day-2.txt:0:0", "classification": "False", "docids": null, "evidences": [[{"docid": "train1.txt", "end_sentence": 16, "end_token": 349, "start_sentence": 15, "start_token": 327, "text": "It was these same impulses , leading so invariably to success , that made his enemies call him the Wisest Man ."}, {"docid": "train1.txt", "end_sentence": 17, "end_token": 359, "start_sentence": 16, "start_token": 349, "text": "He leaned forward and touched the chauffeur 's shoulder ."}, {"docid": "train1.txt", "end_sentence": 18, "end_token": 372, "start_sentence": 17, "start_token": 359, "text": "\" Stop at the Court of General Sessions , \" he commanded ."}, {"docid": "train1.txt", "end_sentence": 20, "end_token": 405, "start_sentence": 19, "start_token": 384, "text": "A word , a personal word from him to the district attorney , or the judge , would be enough ."}]], "query": "How does Mr. Thorndike act upon his impulse ? || It depended on the impulse", "query_type": null}
+{"annotation_id": "Fiction-stories-masc-A_Wasted_Day-2.txt:0:1", "classification": "False", "docids": null, "evidences": [[{"docid": "train2.txt", "end_sentence": 16, "end_token": 349, "start_sentence": 15, "start_token": 327, "text": "It was these same impulses , leading so invariably to success , that made his enemies call him the Wisest Man ."}, {"docid": "train2.txt", "end_sentence": 17, "end_token": 359, "start_sentence": 16, "start_token": 349, "text": "He leaned forward and touched the chauffeur 's shoulder ."}, {"docid": "train2.txt", "end_sentence": 18, "end_token": 372, "start_sentence": 17, "start_token": 359, "text": "\" Stop at the Court of General Sessions , \" he commanded ."}, {"docid": "train2.txt", "end_sentence": 20, "end_token": 405, "start_sentence": 19, "start_token": 384, "text": "A word , a personal word from him to the district attorney , or the judge , would be enough ."}]], "query": "How does Mr. Thorndike act upon his impulse ? || He didn't - he first thought things through", "query_type": null}
+{"annotation_id": "Fiction-stories-masc-Nathans_Bylichka-6.txt:12:2", "classification": "True", "docids": null, "evidences": [[{"docid": "train3.txt", "end_sentence": 3, "end_token": 40, "start_sentence": 2, "start_token": 18, "text": "Then , with a sigh , he said , \" I have need of a certain stone that I once possessed ."}, {"docid": "train3.txt", "end_sentence": 5, "end_token": 79, "start_sentence": 4, "start_token": 54, "text": "My problem is that it was not made in this realm , but by a human , and only a human may handle it ."}]], "query": "Who can handle the stone ? || Only a human can handle it", "query_type": null}

tensorflow_datasets/testing/test_data/fake_examples/eraser_multi_rc/multirc/val.jsonl

@@ -0,0 +1 @@
+{"annotation_id": "Fiction-stories-masc-The_Black_Willow-0.txt:0:0", "classification": "False", "docids": null, "evidences": [[{"docid": "valid1.txt", "end_sentence": 1, "end_token": 13, "start_sentence": 0, "start_token": 0, "text": "Allan sat down at his desk and pulled the chair in close ."}, {"docid": "valid1.txt", "end_sentence": 2, "end_token": 29, "start_sentence": 1, "start_token": 13, "text": "Opening a side drawer , he took out a piece of paper and his inkpot ."}, {"docid": "valid1.txt", "end_sentence": 3, "end_token": 57, "start_sentence": 2, "start_token": 29, "text": "After filling his pen , Allan looked at his paper in the orange glow from the lantern set back in the desk 's right - hand corner ."}]], "query": "Name few objects said to be in or on Allan 's desk || Eraser", "query_type": null}

tensorflow_datasets/text/__init__.py

@@ -19,6 +19,7 @@
 from tensorflow_datasets.text.civil_comments import CivilComments
 from tensorflow_datasets.text.cos_e import CosE
 from tensorflow_datasets.text.definite_pronoun_resolution import DefinitePronounResolution
+from tensorflow_datasets.text.eraser_multi_rc import EraserMultiRc
 from tensorflow_datasets.text.esnli import Esnli
 from tensorflow_datasets.text.gap import Gap
 from tensorflow_datasets.text.glue import Glue

tensorflow_datasets/text/eraser_multi_rc.py

@@ -0,0 +1,121 @@
+# coding=utf-8
+# Copyright 2020 The TensorFlow Datasets Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Passage, query, answers and answer classification with explanations."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import json
+import os
+import tensorflow as tf
+import tensorflow_datasets.public_api as tfds
+
+_CITATION = """
+@unpublished{eraser2019,
+    title = {ERASER: A Benchmark to Evaluate Rationalized NLP Models},
+    author = {Jay DeYoung and Sarthak Jain and Nazneen Fatema Rajani and Eric Lehman and Caiming Xiong and Richard Socher and Byron C. Wallace}
+}
+@inproceedings{MultiRC2018,
+    author = {Daniel Khashabi and Snigdha Chaturvedi and Michael Roth and Shyam Upadhyay and Dan Roth},
+    title = {Looking Beyond the Surface:A Challenge Set for Reading Comprehension over Multiple Sentences},
+    booktitle = {NAACL},
+    year = {2018}
+}
+"""
+
+_DESCRIPTION = """
+Eraser Multi RC is a dataset for queries over multi-line passages, along with
+answers and a rationalte. Each example in this dataset has the following 5 parts
+1. A Mutli-line Passage
+2. A Query about the passage
+3. An Answer to the query
+4. A Classification as to whether the answer is right or wrong
+5. An Explanation justifying the classification
+"""
+
+_DOWNLOAD_URL = 'http://www.eraserbenchmark.com/zipped/multirc.tar.gz'
+
+
+class EraserMultiRc(tfds.core.GeneratorBasedBuilder):
+  """Multi Sentence Reasoning with Explanations (Eraser Benchmark)."""
+
+  VERSION = tfds.core.Version('0.1.0')
+
+  def _info(self):
+    return tfds.core.DatasetInfo(
+        builder=self,
+        description=_DESCRIPTION,
+        features=tfds.features.FeaturesDict({
+            'passage': tfds.features.Text(),
+            'query_and_answer': tfds.features.Text(),
+            'label': tfds.features.ClassLabel(names=['True', 'False']),
+            'evidences': tfds.features.Sequence(tfds.features.Text())
+        }),
+        supervised_keys=None,
+        homepage='https://cogcomp.seas.upenn.edu/multirc/',
+        citation=_CITATION,
+    )
+
+  def _split_generators(self, dl_manager):
+    """Returns SplitGenerators."""
+
+    dl_dir = dl_manager.download_and_extract(_DOWNLOAD_URL)
+    data_dir = os.path.join(dl_dir, 'multirc')
+    return [
+        tfds.core.SplitGenerator(
+            name=tfds.Split.TRAIN,
+            # These kwargs will be passed to _generate_examples
+            gen_kwargs={'data_dir': data_dir,
+                        'filepath': os.path.join(data_dir, 'train.jsonl')},
+        ),
+        tfds.core.SplitGenerator(
+            name=tfds.Split.VALIDATION,
+            # These kwargs will be passed to _generate_examples
+            gen_kwargs={'data_dir': data_dir,
+                        'filepath': os.path.join(data_dir, 'val.jsonl')},
+        ),
+        tfds.core.SplitGenerator(
+            name=tfds.Split.TEST,
+            # These kwargs will be passed to _generate_examples
+            gen_kwargs={'data_dir': data_dir,
+                        'filepath': os.path.join(data_dir, 'test.jsonl')},
+        ),
+    ]
+
+  def _generate_examples(self, data_dir, filepath):
+    """Yields examples."""
+
+    multirc_dir = os.path.join(data_dir, 'docs')
+    with tf.io.gfile.GFile(filepath) as f:
+      for line in f:
+        row = json.loads(line)
+        evidences = []
+
+        for evidence in row['evidences'][0]:
+          docid = evidence['docid']
+          evidences.append(evidence['text'])
+
+        passage_file = os.path.join(multirc_dir, docid)
+        with tf.io.gfile.GFile(passage_file) as f1:
+          passage_text = f1.read()
+
+        yield row['annotation_id'], {
+            'passage': passage_text,
+            'query_and_answer': row['query'],
+            'label': row['classification'],
+            'evidences': evidences
+        }