Lesson 1, FL: MNIST last mini project - Hotfix does not work

[gonzalo-munillag]

Hi :)

I executed the first cell in the last notebooks with the MNIST dataset, however, even though the hotfix is included, the download of the MNIST dataset does not work.
How could I solve this issue?

Thank you in advance!
Cheers!

---

HTTPError Traceback (most recent call last)
in
----> 1 torchvision.datasets.MNIST(get_root_data_path(), train=True, download=True)
2 torchvision.datasets.MNIST(get_root_data_path(), train=False, download=True)

/usr/local/anaconda3/envs/privacy/lib/python3.9/site-packages/torchvision/datasets/mnist.py in init(self, root, train, transform, target_transform, download)
77
78 if download:
---> 79 self.download()
80
81 if not self._check_exists():

/usr/local/anaconda3/envs/privacy/lib/python3.9/site-packages/torchvision/datasets/mnist.py in download(self)
144 for url, md5 in self.resources:
145 filename = url.rpartition('/')[2]
--> 146 download_and_extract_archive(url, download_root=self.raw_folder, filename=filename, md5=md5)
147
148 # process and save as torch files

/usr/local/anaconda3/envs/privacy/lib/python3.9/site-packages/torchvision/datasets/utils.py in download_and_extract_archive(url, download_root, extract_root, filename, md5, remove_finished)
312 filename = os.path.basename(url)
313
--> 314 download_url(url, download_root, filename, md5)
315
316 archive = os.path.join(download_root, filename)

/usr/local/anaconda3/envs/privacy/lib/python3.9/site-packages/torchvision/datasets/utils.py in download_url(url, root, filename, md5, max_redirect_hops)
138 _urlretrieve(url, fpath)
139 else:
--> 140 raise e
141 # check integrity of downloaded file
142 if not check_integrity(fpath, md5):

/usr/local/anaconda3/envs/privacy/lib/python3.9/site-packages/torchvision/datasets/utils.py in download_url(url, root, filename, md5, max_redirect_hops)
130 try:
131 print('Downloading ' + url + ' to ' + fpath)
--> 132 _urlretrieve(url, fpath)
133 except (urllib.error.URLError, IOError) as e: # type: ignore[attr-defined]
134 if url[:5] == 'https':

/usr/local/anaconda3/envs/privacy/lib/python3.9/site-packages/torchvision/datasets/utils.py in _urlretrieve(url, filename, chunk_size)
27 def _urlretrieve(url: str, filename: str, chunk_size: int = 1024) -> None:
28 with open(filename, "wb") as fh:
---> 29 with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": USER_AGENT})) as response:
30 with tqdm(total=response.length) as pbar:
31 for chunk in iter(lambda: response.read(chunk_size), ""):

/usr/local/anaconda3/envs/privacy/lib/python3.9/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
212 else:
213 opener = _opener
--> 214 return opener.open(url, data, timeout)
215
216 def install_opener(opener):

/usr/local/anaconda3/envs/privacy/lib/python3.9/urllib/request.py in open(self, fullurl, data, timeout)
521 for processor in self.process_response.get(protocol, []):
522 meth = getattr(processor, meth_name)
--> 523 response = meth(req, response)
524
525 return response

/usr/local/anaconda3/envs/privacy/lib/python3.9/urllib/request.py in http_response(self, request, response)
630 # request was successfully received, understood, and accepted.
631 if not (200 <= code < 300):
--> 632 response = self.parent.error(
633 'http', request, response, code, msg, hdrs)
634

/usr/local/anaconda3/envs/privacy/lib/python3.9/urllib/request.py in error(self, proto, *args)
559 if http_err:
560 args = (dict, 'default', 'http_error_default') + orig_args
--> 561 return self._call_chain(*args)
562
563 # XXX probably also want an abstract factory that knows when it makes

/usr/local/anaconda3/envs/privacy/lib/python3.9/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
492 for handler in handlers:
493 func = getattr(handler, meth_name)
--> 494 result = func(*args)
495 if result is not None:
496 return result

/usr/local/anaconda3/envs/privacy/lib/python3.9/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
639 class HTTPDefaultErrorHandler(BaseHandler):
640 def http_error_default(self, req, fp, code, msg, hdrs):
--> 641 raise HTTPError(req.full_url, code, msg, hdrs, fp)
642
643 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 503: Service Unavailable

[gonzalo-munillag]

I found a workaround, which still fails down the line.

1. Install wget from terminal

2. Then run this in a notebook cell:

        !wget www.di.ens.fr/~lelarge/MNIST.tar.gz
        !tar -zxvf MNIST.tar.gz
        from torchvision.datasets import MNIST
        from torchvision import transforms
        
        mnist_train = MNIST('./', download=False,
                            transform=transforms.Compose([
                                transforms.ToTensor(),
                            ]), train=True)
   

The problem is that then you do not have the format that the notebook template has.
The template also does not use .send(duet, pointable=True) as with the other exercises, which is weird, cz otherwise how is the data scientist going to train anything?

I did this:

      mnist_train = mnist_train.tag("mnist-data")
      mnist_test = mnist_test.tag("mnist-target")
      mnist_train = mnist_train.describe(
          "This is a dataset for number identification with 60000 samples and 1 feature" 
      )
      mnist_test = mnist_test.describe("Labels for numbers from 0 to 9. (10 labels)")
      data_pointer = mnist_train.send(duet, pointable=True)
      target_pointer = mnist_test.send(duet, pointable=True)

But I get:
AttributeError: 'MNIST' object has no attribute '_object2proto'

[znreza]

Hi @gonzalo-munillag torchvision does not have duet functionalities out-of-the-box. You need to use duet.torchvision instead of torchvision directly like this

remote_torchvision = duet.torchvision

Then you can get the dataset pointer to the remote data like below

train_data_ptr = remote_torchvision.datasets.MNIST(str(get_root_data_path()), train=True, download=True, transform=transforms)
train_loader_ptr = remote_torch.utils.data.DataLoader(train_data_ptr,**train_kwargs)

In this example, instead of the data owner having to upload the dataset to the duet server, we are just using a reference to the dataset already available in torchvision, using remote_torchvision. syft has a new feature coming up soon with the ability to upload and send big data from the data owner's end. You can already try it here.

[gonzalo-munillag]

Thank you for the answer @znreza.
However, @cbs-rschreijen went through the same problem as I did and he had the exact answer to the problem.
Cheers!

[cbs-rschreijen]

Hi, I had the same problem. Seems the http://yann.lecun.com server has some form of request throttling to manage load. If you keep executing the cell in the notebook it will eventually succeed. There are multiple files to be downloaded and it can fail at every step but luckily the files which are downloaded successfully will not be redownloaded again. So keep trying until all files are downloaded and no errors are presented. After that, it will work fine.

[gonzalo-munillag]

Hi @cbs-rschreijen
Indeed, that worked! Thank you for the input!