Document which datasets require manual download.

PiperOrigin-RevId: 282954909

Author: cyfra

docs/catalog/_toc.yaml

@@ -9,7 +9,7 @@ toc:
   title: Audio
 - section:
   - path: /datasets/catalog/abstract_reasoning
-    title: abstract_reasoning
+    title: abstract_reasoning (manual)
   - path: /datasets/catalog/aflw2k3d
     title: aflw2k3d
   - path: /datasets/catalog/bigearthnet
@@ -33,7 +33,7 @@ toc:
   - path: /datasets/catalog/celeb_a
     title: celeb_a
   - path: /datasets/catalog/celeb_a_hq
-    title: celeb_a_hq
+    title: celeb_a_hq (manual)
   - path: /datasets/catalog/cifar10
     title: cifar10
   - path: /datasets/catalog/cifar100
@@ -55,13 +55,13 @@ toc:
   - path: /datasets/catalog/colorectal_histology_large
     title: colorectal_histology_large
   - path: /datasets/catalog/curated_breast_imaging_ddsm
-    title: curated_breast_imaging_ddsm
+    title: curated_breast_imaging_ddsm (manual)
   - path: /datasets/catalog/cycle_gan
     title: cycle_gan
   - path: /datasets/catalog/deep_weeds
     title: deep_weeds
   - path: /datasets/catalog/diabetic_retinopathy_detection
-    title: diabetic_retinopathy_detection
+    title: diabetic_retinopathy_detection (manual)
   - path: /datasets/catalog/dmlab
     title: dmlab
   - path: /datasets/catalog/downsampled_imagenet
@@ -85,11 +85,11 @@ toc:
   - path: /datasets/catalog/i_naturalist2017
     title: i_naturalist2017
   - path: /datasets/catalog/image_label_folder
-    title: image_label_folder
+    title: image_label_folder (manual)
   - path: /datasets/catalog/imagenet2012
-    title: imagenet2012
+    title: imagenet2012 (manual)
   - path: /datasets/catalog/imagenet2012_corrupted
-    title: imagenet2012_corrupted
+    title: imagenet2012_corrupted (manual)
   - path: /datasets/catalog/imagenet_resized
     title: imagenet_resized
   - path: /datasets/catalog/kmnist
@@ -127,7 +127,7 @@ toc:
   - path: /datasets/catalog/quickdraw_bitmap
     title: quickdraw_bitmap
   - path: /datasets/catalog/resisc45
-    title: resisc45
+    title: resisc45 (manual)
   - path: /datasets/catalog/rock_paper_scissors
     title: rock_paper_scissors
   - path: /datasets/catalog/scene_parse150
@@ -193,15 +193,15 @@ toc:
   - path: /datasets/catalog/multi_news
     title: multi_news
   - path: /datasets/catalog/newsroom
-    title: newsroom
+    title: newsroom (manual)
   - path: /datasets/catalog/reddit_tifu
     title: reddit_tifu
   - path: /datasets/catalog/scientific_papers
     title: scientific_papers
   - path: /datasets/catalog/wikihow
-    title: wikihow
+    title: wikihow (manual)
   - path: /datasets/catalog/xsum
-    title: xsum
+    title: xsum (manual)
   title: Summarization
 - section:
   - path: /datasets/catalog/c4
@@ -247,19 +247,19 @@ toc:
   - path: /datasets/catalog/ted_multi_translate
     title: ted_multi_translate
   - path: /datasets/catalog/wmt14_translate
-    title: wmt14_translate
+    title: wmt14_translate (manual)
   - path: /datasets/catalog/wmt15_translate
-    title: wmt15_translate
+    title: wmt15_translate (manual)
   - path: /datasets/catalog/wmt16_translate
-    title: wmt16_translate
+    title: wmt16_translate (manual)
   - path: /datasets/catalog/wmt17_translate
-    title: wmt17_translate
+    title: wmt17_translate (manual)
   - path: /datasets/catalog/wmt18_translate
-    title: wmt18_translate
+    title: wmt18_translate (manual)
   - path: /datasets/catalog/wmt19_translate
-    title: wmt19_translate
+    title: wmt19_translate (manual)
   - path: /datasets/catalog/wmt_t2t_translate
-    title: wmt_t2t_translate
+    title: wmt_t2t_translate (manual)
   title: Translate
 - section:
   - path: /datasets/catalog/bair_robot_pushing_small

docs/catalog/abstract_reasoning.md

@@ -10,7 +10,7 @@
   <meta itemprop="citation" content="@InProceedings{pmlr-v80-barrett18a,&#10;  title =   {Measuring abstract reasoning in neural networks},&#10;  author =   {Barrett, David and Hill, Felix and Santoro, Adam and Morcos, Ari and Lillicrap, Timothy},&#10;  booktitle =    {Proceedings of the 35th International Conference on Machine Learning},&#10;  pages =   {511--520},&#10;  year =    {2018},&#10;  editor =      {Dy, Jennifer and Krause, Andreas},&#10;  volume =      {80},&#10;  series =    {Proceedings of Machine Learning Research},&#10;  address =     {Stockholmsmassan, Stockholm Sweden},&#10;  month =     {10--15 Jul},&#10;  publisher =     {PMLR},&#10;  pdf =     {http://proceedings.mlr.press/v80/barrett18a/barrett18a.pdf},&#10;  url =   {http://proceedings.mlr.press/v80/barrett18a.html},&#10;  abstract =    {Whether neural networks can learn abstract reasoning or whetherthey merely rely on superficial statistics is a topic of recent debate. Here, we propose a dataset and challenge designed to probe abstract reasoning, inspired by a well-known human IQ test. To succeed at this challenge, models must cope with various generalisation 'regimes' in which the training data and test questions differ in clearly-defined ways. We show that popular models such as ResNets perform poorly, even when the training and test sets differ only minimally, and we present a novel architecture, with structure designed to encourage reasoning, that does significantly better. When we vary the way in which the test questions and training data differ, we find that our model is notably proficient at certain forms of generalisation, but notably weak at others. We further show that the model's ability to generalise improves markedly if it is trained to predict symbolic explanations for its answers. Altogether, we introduce and explore ways to both measure and induce stronger abstract reasoning in neural networks. Our freely-available dataset should motivate further progress in this direction.}&#10;}&#10;" />
 </div>
 
-# `abstract_reasoning`
+# `abstract_reasoning` (Manual download)
 
 Procedurally Generated Matrices (PGM) data from the paper Measuring Abstract
 Reasoning in Neural Networks, Barrett, Hill, Santoro et al. 2018. The goal is to
@@ -113,10 +113,18 @@ Versions:
 
 *   **`0.0.2`** (default):
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/abstract_reasoning/`):
+Data can be downloaded from
+https://console.cloud.google.com/storage/browser/ravens-matrices Please put all
+the tar.gz files in manual_dir.
+
 ### Statistics
+
 None computed
 
 ### Features
+
 ```python
 FeaturesDict({
     'answers': Video(Image(shape=(160, 160, 1), dtype=tf.uint8)),
@@ -133,6 +141,7 @@ FeaturesDict({
 *   [https://github.com/deepmind/abstract-reasoning-matrices](https://github.com/deepmind/abstract-reasoning-matrices)
 
 ## `abstract_reasoning/interpolation`
+
 As in the neutral split, $S$ consisted of any \
 triples $[r, o, a]$. For interpolation, in the training set, when the \
 attribute was "colour" or "size" (i.e., the ordered attributes), the values of \
@@ -145,6 +154,12 @@ Versions:
 
 *   **`0.0.2`** (default):
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/abstract_reasoning/`):
+Data can be downloaded from
+https://console.cloud.google.com/storage/browser/ravens-matrices Please put all
+the tar.gz files in manual_dir.
+
 ### Statistics
 
 None computed
@@ -176,6 +191,12 @@ Versions:
 
 *   **`0.0.2`** (default):
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/abstract_reasoning/`):
+Data can be downloaded from
+https://console.cloud.google.com/storage/browser/ravens-matrices Please put all
+the tar.gz files in manual_dir.
+
 ### Statistics
 None computed
 
@@ -208,6 +229,12 @@ Versions:
 
 *   **`0.0.2`** (default):
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/abstract_reasoning/`):
+Data can be downloaded from
+https://console.cloud.google.com/storage/browser/ravens-matrices Please put all
+the tar.gz files in manual_dir.
+
 ### Statistics
 None computed
 
@@ -228,6 +255,7 @@ FeaturesDict({
 *   [https://github.com/deepmind/abstract-reasoning-matrices](https://github.com/deepmind/abstract-reasoning-matrices)
 
 ## `abstract_reasoning/attr.rels`
+
 In our dataset, there are 29 possible unique \
 triples $[r,o,a]$. We allocated seven of these for the test set, at random, \
 but such that each of the attributes was represented exactly once in this set. \
@@ -238,6 +266,12 @@ Versions:
 
 *   **`0.0.2`** (default):
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/abstract_reasoning/`):
+Data can be downloaded from
+https://console.cloud.google.com/storage/browser/ravens-matrices Please put all
+the tar.gz files in manual_dir.
+
 ### Statistics
 None computed
 
@@ -270,6 +304,12 @@ Versions:
 
 *   **`0.0.2`** (default):
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/abstract_reasoning/`):
+Data can be downloaded from
+https://console.cloud.google.com/storage/browser/ravens-matrices Please put all
+the tar.gz files in manual_dir.
+
 ### Statistics
 None computed
 
@@ -299,6 +339,12 @@ Versions:
 
 *   **`0.0.2`** (default):
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/abstract_reasoning/`):
+Data can be downloaded from
+https://console.cloud.google.com/storage/browser/ravens-matrices Please put all
+the tar.gz files in manual_dir.
+
 ### Statistics
 None computed
 
@@ -328,6 +374,12 @@ Versions:
 
 *   **`0.0.2`** (default):
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/abstract_reasoning/`):
+Data can be downloaded from
+https://console.cloud.google.com/storage/browser/ravens-matrices Please put all
+the tar.gz files in manual_dir.
+
 ### Statistics
 None computed
 

docs/catalog/celeb_a_hq.md

@@ -2,13 +2,15 @@
   <div itemscope itemprop="includedInDataCatalog" itemtype="http://schema.org/DataCatalog">
     <meta itemprop="name" content="TensorFlow Datasets" />
   </div>
+
   <meta itemprop="name" content="celeb_a_hq" />
   <meta itemprop="description" content="High-quality version of the CELEBA&#10;dataset, consisting of 30000 images in 1024 x 1024 resolution.&#10;&#10;WARNING: This dataset currently requires you to prepare images on your own.&#10;&#10;&#10;To use this dataset:&#10;&#10;```python&#10;import tensorflow_datasets as tfds&#10;&#10;ds = tfds.load('celeb_a_hq', split='train')&#10;for ex in ds.take(4):&#10;  print(ex)&#10;```&#10;&#10;See [the guide](https://www.tensorflow.org/datasets/overview) for more&#10;informations on [tensorflow_datasets](https://www.tensorflow.org/datasets).&#10;&#10;" />
   <meta itemprop="url" content="https://www.tensorflow.org/datasets/catalog/celeb_a_hq" />
   <meta itemprop="sameAs" content="https://github.com/tkarras/progressive_growing_of_gans" />
   <meta itemprop="citation" content="@article{DBLP:journals/corr/abs-1710-10196,&#10;  author    = {Tero Karras and&#10;               Timo Aila and&#10;               Samuli Laine and&#10;               Jaakko Lehtinen},&#10;  title     = {Progressive Growing of GANs for Improved Quality, Stability, and Variation},&#10;  journal   = {CoRR},&#10;  volume    = {abs/1710.10196},&#10;  year      = {2017},&#10;  url       = {http://arxiv.org/abs/1710.10196},&#10;  archivePrefix = {arXiv},&#10;  eprint    = {1710.10196},&#10;  timestamp = {Mon, 13 Aug 2018 16:46:42 +0200},&#10;  biburl    = {https://dblp.org/rec/bib/journals/corr/abs-1710-10196},&#10;  bibsource = {dblp computer science bibliography, https://dblp.org}&#10;}&#10;" />
 </div>
-# `celeb_a_hq`
+
+# `celeb_a_hq` (Manual download)
 
 High-quality version of the CELEBA dataset, consisting of 30000 images in 1024 x
 1024 resolution.
@@ -54,6 +56,12 @@ Versions:
 *   **`0.1.0`** (default):
 *   `2.0.0`: New split API (https://tensorflow.org/datasets/splits)
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/celeb_a_hq/`): manual_dir
+should contain multiple tar files with images (data2x2.tar, data4x4.tar ..
+data1024x1024.tar). Detailed instructions are here:
+https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training
+
 ### Statistics
 
 Split | Examples
@@ -81,6 +89,12 @@ Versions:
 *   **`0.1.0`** (default):
 *   `2.0.0`: New split API (https://tensorflow.org/datasets/splits)
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/celeb_a_hq/`): manual_dir
+should contain multiple tar files with images (data2x2.tar, data4x4.tar ..
+data1024x1024.tar). Detailed instructions are here:
+https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training
+
 ### Statistics
 
 Split | Examples
@@ -108,6 +122,12 @@ Versions:
 *   **`0.1.0`** (default):
 *   `2.0.0`: New split API (https://tensorflow.org/datasets/splits)
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/celeb_a_hq/`): manual_dir
+should contain multiple tar files with images (data2x2.tar, data4x4.tar ..
+data1024x1024.tar). Detailed instructions are here:
+https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training
+
 ### Statistics
 
 Split | Examples
@@ -135,6 +155,12 @@ Versions:
 *   **`0.1.0`** (default):
 *   `2.0.0`: New split API (https://tensorflow.org/datasets/splits)
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/celeb_a_hq/`): manual_dir
+should contain multiple tar files with images (data2x2.tar, data4x4.tar ..
+data1024x1024.tar). Detailed instructions are here:
+https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training
+
 ### Statistics
 
 Split | Examples
@@ -162,6 +188,12 @@ Versions:
 *   **`0.1.0`** (default):
 *   `2.0.0`: New split API (https://tensorflow.org/datasets/splits)
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/celeb_a_hq/`): manual_dir
+should contain multiple tar files with images (data2x2.tar, data4x4.tar ..
+data1024x1024.tar). Detailed instructions are here:
+https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training
+
 ### Statistics
 
 Split | Examples
@@ -189,6 +221,12 @@ Versions:
 *   **`0.1.0`** (default):
 *   `2.0.0`: New split API (https://tensorflow.org/datasets/splits)
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/celeb_a_hq/`): manual_dir
+should contain multiple tar files with images (data2x2.tar, data4x4.tar ..
+data1024x1024.tar). Detailed instructions are here:
+https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training
+
 ### Statistics
 
 Split | Examples
@@ -216,6 +254,12 @@ Versions:
 *   **`0.1.0`** (default):
 *   `2.0.0`: New split API (https://tensorflow.org/datasets/splits)
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/celeb_a_hq/`): manual_dir
+should contain multiple tar files with images (data2x2.tar, data4x4.tar ..
+data1024x1024.tar). Detailed instructions are here:
+https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training
+
 ### Statistics
 
 Split | Examples
@@ -243,6 +287,12 @@ Versions:
 *   **`0.1.0`** (default):
 *   `2.0.0`: New split API (https://tensorflow.org/datasets/splits)
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/celeb_a_hq/`): manual_dir
+should contain multiple tar files with images (data2x2.tar, data4x4.tar ..
+data1024x1024.tar). Detailed instructions are here:
+https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training
+
 ### Statistics
 
 Split | Examples
@@ -270,6 +320,12 @@ Versions:
 *   **`0.1.0`** (default):
 *   `2.0.0`: New split API (https://tensorflow.org/datasets/splits)
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/celeb_a_hq/`): manual_dir
+should contain multiple tar files with images (data2x2.tar, data4x4.tar ..
+data1024x1024.tar). Detailed instructions are here:
+https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training
+
 ### Statistics
 
 Split | Examples
@@ -297,6 +353,12 @@ Versions:
 *   **`0.1.0`** (default):
 *   `2.0.0`: New split API (https://tensorflow.org/datasets/splits)
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/celeb_a_hq/`): manual_dir
+should contain multiple tar files with images (data2x2.tar, data4x4.tar ..
+data1024x1024.tar). Detailed instructions are here:
+https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training
+
 ### Statistics
 
 Split | Examples
@@ -324,6 +386,12 @@ Versions:
 *   **`0.1.0`** (default):
 *   `2.0.0`: New split API (https://tensorflow.org/datasets/splits)
 
+WARNING: This dataset requires you to download the source data manually into
+manual_dir (defaults to `~/tensorflow_datasets/manual/celeb_a_hq/`): manual_dir
+should contain multiple tar files with images (data2x2.tar, data4x4.tar ..
+data1024x1024.tar). Detailed instructions are here:
+https://github.com/tkarras/progressive_growing_of_gans#preparing-datasets-for-training
+
 ### Statistics
 
 Split | Examples