Skip to content

Commit 64bbcef

Browse files
Conchylicultorcopybara-github
authored andcommitted
Restore beam Py3 tests
PiperOrigin-RevId: 257005506
1 parent edfd9f7 commit 64bbcef

File tree

2 files changed

+10
-19
lines changed

2 files changed

+10
-19
lines changed

docs/beam_datasets.md

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,6 @@ Note: This mode is still experimental, so the API may change in the future
88
depending on user feedback. Do not hesitate to
99
[submit your feedback](https://github.com/tensorflow/datasets/issues/new?assignees=&labels=enhancement&template=feature_request.md&title=).
1010

11-
Warning: Currently we only support using Apache Beam with Python 2. We will add
12-
Python 3 support with the next release of Beam which includes a needed
13-
[bug fix](https://github.com/apache/beam/commit/46a70a8b4691a169c9e018299765e1bdb88239f4).
14-
1511
* [Prerequisites](#prerequisites)
1612
* [Instructions](#instructions)
1713
* [Example](#example)

tensorflow_datasets/testing/dataset_builder_testing.py

Lines changed: 10 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@
2727
from absl.testing import absltest
2828
from absl.testing import parameterized
2929
import numpy as np
30-
import six
3130
import tensorflow as tf
3231

3332
from tensorflow_datasets.core import dataset_builder
@@ -70,28 +69,31 @@ class DatasetBuilderTestCase(parameterized.TestCase, test_utils.SubTestCase):
7069
"""Inherit this class to test your DatasetBuilder class.
7170
7271
You must set the following class attributes:
73-
DATASET_CLASS: class object of DatasetBuilder you want to test.
72+
73+
* DATASET_CLASS: class object of DatasetBuilder you want to test.
7474
7575
You may set the following class attributes:
76-
VERSION: `str`. The version used to run the test. eg: '1.2.*'.
76+
77+
* VERSION: `str`. The version used to run the test. eg: '1.2.*'.
7778
Defaults to None (canonical version).
78-
BUILDER_CONFIG_NAMES_TO_TEST: `list[str]`, the list of builder configs
79+
* BUILDER_CONFIG_NAMES_TO_TEST: `list[str]`, the list of builder configs
7980
that should be tested. If None, all the BUILDER_CONFIGS from the class
8081
will be tested.
81-
DL_EXTRACT_RESULT: `dict[str]`, the returned result of mocked
82+
* DL_EXTRACT_RESULT: `dict[str]`, the returned result of mocked
8283
`download_and_extract` method. The values should be the path of files
8384
present in the `fake_examples` directory, relative to that directory.
8485
If not specified, path to `fake_examples` will always be returned.
85-
EXAMPLE_DIR: `str`, the base directory in in which fake examples are
86+
* EXAMPLE_DIR: `str`, the base directory in in which fake examples are
8687
contained. Optional; defaults to
8788
tensorflow_datasets/testing/test_data/fake_examples/<dataset name>.
88-
OVERLAPPING_SPLITS: `list[str]`, splits containing examples from other
89+
* OVERLAPPING_SPLITS: `list[str]`, splits containing examples from other
8990
splits (e.g. a "example" split containing pictures from other splits).
90-
MOCK_OUT_FORBIDDEN_OS_FUNCTIONS: `bool`, defaults to True. Set to False to
91+
* MOCK_OUT_FORBIDDEN_OS_FUNCTIONS: `bool`, defaults to True. Set to False to
9192
disable checks preventing usage of `os` or builtin functions instead of
9293
recommended `tf.io.gfile` API.
9394
9495
This test case will check for the following:
96+
9597
- the dataset builder is correctly registered, i.e. `tfds.load(name)` works;
9698
- the dataset builder can read the fake examples stored in
9799
testing/test_data/fake_examples/${dataset_name};
@@ -228,13 +230,6 @@ def _download_and_prepare_as_dataset(self, builder):
228230
manual_dir=self.example_dir,
229231
):
230232
if isinstance(builder, dataset_builder.BeamBasedBuilder):
231-
232-
# TODO(b/129148632): The current apache-beam 2.11.0 do not work with Py3
233-
# Update once the new version is out (around April)
234-
skip_beam_test = bool(six.PY3)
235-
if skip_beam_test:
236-
return
237-
238233
import apache_beam as beam # pylint: disable=g-import-not-at-top
239234
# For Beam datasets, set-up the runner config
240235
beam_runner = None

0 commit comments

Comments
 (0)