Skip to content

Handle 429 and prevent 301 in gp_downloadmetadata #406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Sep 6, 2022
Merged

Handle 429 and prevent 301 in gp_downloadmetadata #406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b8c44c4
Update `gp_downloadmetadata` to offer to retry when receiving a 429
mokagio Sep 6, 2022
80af526
Update a couple of `puts` in `gp_downloadmetadata` to `UI.message`
mokagio Sep 6, 2022
31daea9
Add more UI messages to `gp_downloadmetadata`
mokagio Sep 6, 2022
0af9cfa
Add trailing `/` to URL used by `gp_downloadmetadata` to avoid 301
mokagio Sep 6, 2022
60106a5
Show user unhandled `gp_downloadmetadata` errors and ask to continue
mokagio Sep 6, 2022
eafb2ed
Add `CHANGELOG.md` entry for #406
mokagio Sep 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@

### Breaking Changes

_None_
- Propose to retry when `gp_downloadmetadata` receives a `429 - Too Many Requests` error. [#406]
Copy link
Contributor

@AliSoftware AliSoftware Sep 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mokagio I'm not sure this is really a breaking change — as in, that won't require any change to client repos adopting the new release-toolkit to update their call sites or config or whatnot, will it?

I think it's not worth a major version bump on the next release, while a New Feature and minor bump would make more sense for this change 🙃

Copy link
Contributor Author

@mokagio mokagio Sep 12, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, not at all a breaking change. That was an oversight which I realized only after merging this. It's fixed in 39808da.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, perfect, thanks for following-up 👍


### New Features

_None_

### Bug Fixes

_None_
- Update the URL used by `gp_downloadmetadata` to prevent consistent `301` responses. [#406]

### Internal Changes

Expand Down
8 changes: 4 additions & 4 deletions lib/fastlane/plugin/wpmreleasetoolkit/actions/common/gp_downloadmetadata_action.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@ def self.run(params)

params[:locales].each do |loc|
if loc.is_a?(Array)
puts "Downloading language: #{loc[1]}"
complete_url = "#{params[:project_url]}#{loc[0]}/default/export-translations?filters[status]=current&format=json"
UI.message "Downloading language: #{loc[1]}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Converted the puts to UI.message for more consistent logs.

complete_url = "#{params[:project_url]}#{loc[0]}/default/export-translations/?filters[status]=current&format=json"
downloader.download(loc[1], complete_url, loc[1] == params[:source_locale])
end

if loc.is_a?(String)
puts "Downloading language: #{loc}"
complete_url = "#{params[:project_url]}#{loc}/default/export-translations?filters[status]=current&format=json"
UI.message "Downloading language: #{loc}"
complete_url = "#{params[:project_url]}#{loc}/default/export-translations/?filters[status]=current&format=json"
downloader.download(loc, complete_url, loc == params[:source_locale])
end
end
Expand Down
35 changes: 29 additions & 6 deletions lib/fastlane/plugin/wpmreleasetoolkit/helper/metadata_download_helper.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,7 @@ def initialize(target_folder, target_files)
def download(target_locale, glotpress_url, is_source)
uri = URI(glotpress_url)
response = Net::HTTP.get_response(uri)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a thought, not related to this PR: it might be good to use faraday which has features like following redirects and retry with an interval (which could be a replacement for handling 429 error?).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah... I don't know about faraday or other Ruby networking clients, but given the number of network requests we do with this toolkit, having a feature-rich way to do requests would be handy.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iinm Farady is also the network client that fastlane itself uses, so it should already be part of our ruby dependencies anyway.

This is one of those items part of a long list of little things I'd love to improve / refactor to clean up the release-toolkit in many areas and modernize it… but never got the bandwidth to 😅

response = Net::HTTP.get_response(URI.parse(response.header['location'])) if response.code == '301'

@alternates.clear
loc_data = JSON.parse(response.body) rescue loc_data = nil
parse_data(target_locale, loc_data, is_source)
reparse_alternates(target_locale, loc_data, is_source) unless @alternates.length == 0
handle_glotpress_download(response: response, locale: target_locale, is_source: is_source)
end

# Parse JSON data and update the local files
Expand Down Expand Up @@ -100,6 +95,34 @@ def delete_existing_metadata(target_locale)
def get_target_file_path(locale, file_name)
"#{@target_folder}/#{locale}/#{file_name}"
end

private

def handle_glotpress_download(response:, locale:, is_source:)
case response.code
when '200'
# All good, parse the result
UI.success("Successfully downloaded `#{locale}`.")
@alternates.clear
loc_data = JSON.parse(response.body) rescue loc_data = nil
parse_data(locale, loc_data, is_source)
reparse_alternates(target_locale, loc_data, is_source) unless @alternates.length == 0
when '301'
# Follow the redirect
UI.message("Received 301 for `#{locale}`. Following redirect...")
download(locale, response.header['location'], is_source)
when '429'
# We got rate-limited, offer to try again
if UI.confirm("Retry downloading `#{locale}` after receiving 429 from the API?")
download(locale, response.uri, is_source)
else
UI.error("Abandoning `#{locale}` download as requested.")
Copy link
Contributor Author

@mokagio mokagio Sep 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided to make this error so it would print red to mark the difference in behavior.

image
(The 301 is because I hadn't updated the URLs yet)

I can see how error/red might be misleading though, and look like a wholesale failure. If you think that's too confusing, I can update to a message:

Suggested change
UI.error("Abandoning `#{locale}` download as requested.")
UI.message("Abandoning `#{locale}` download as requested.")

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the error message is pretty clear and it's in reaction of us explicitly replying n in the prompt on the previous line, I vote for keeping it red with UI.error 👍

end
else
message = "Received unexpected #{response.code} from request to URI #{response.uri}."
UI.abort_with_message!(message) unless UI.confirm("#{message} Continue anyway?")
end
end
end
end
end