Skip to content

mnist: use mirror only when original server is not available #196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Jul 9, 2024
Merged

mnist: use mirror only when original server is not available #196

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
f567318
mnist: use mirror only when original server is not available
tikkss Jun 19, 2024
4dfebd4
Introduce #base_urls instead of BASE_URL
tikkss Jun 22, 2024
784a1d1
Retry to attempt download until successful
tikkss Jun 25, 2024
aafbddb
Require at least one URL
tikkss Jun 26, 2024
d5cf668
Include the failed URL and fallback URL
tikkss Jun 26, 2024
01e273e
Use `->`, `<>` as separator
tikkss Jul 1, 2024
2acc578
Use index instead of idx
tikkss Jul 2, 2024
714a405
Remove < and > from XXX_url
tikkss Jul 2, 2024
d22645d
Propagate raised exception as-is
tikkss Jul 2, 2024
3191f18
downloader: extract normalize url
tikkss Jul 5, 2024
8f85fb8
downloader: add suppport for fallback
tikkss Jul 5, 2024
f92ec1a
Avoid change @fallback_urls destructively
tikkss Jul 7, 2024
7b78fc2
Avoid an infinite loop
tikkss Jul 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions lib/datasets/mnist.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
module Datasets
class MNIST < Dataset
BASE_URL = "http://yann.lecun.com/exdb/mnist/"
FALLBACK_URL = "https://ossci-datasets.s3.amazonaws.com/mnist/"

class Record < Struct.new(:data, :label)
def pixels
Expand Down Expand Up @@ -45,9 +46,26 @@ def each(&block)
image_path = cache_dir_path + target_file(:image)
label_path = cache_dir_path + target_file(:label)
base_url = self.class::BASE_URL

download(image_path, base_url + target_file(:image))
download(label_path, base_url + target_file(:label))
fallback_url = self.class::FALLBACK_URL

begin
download(image_path, base_url + target_file(:image))
rescue Net::HTTPClientException => error
if error.response.code == "403"
download(image_path, fallback_url + target_file(:image))
else
raise error
end
end
begin
download(label_path, base_url + target_file(:label))
rescue Net::HTTPClientException => error
if error.response.code == "403"
download(label_path, fallback_url + target_file(:label))
else
raise error
end
end

open_data(image_path, label_path, &block)
end
Expand Down
Loading