Skip to content

Commit 62aa408

Browse files
author
Krystle Salazar
committed
Update DAG docs
1 parent 2928370 commit 62aa408

File tree

2 files changed

+17
-0
lines changed

2 files changed

+17
-0
lines changed

catalog/DAGs.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ The following are DAGs grouped by their primary tag:
1919
1. [Database](#database)
2020
1. [Maintenance](#maintenance)
2121
1. [Oauth](#oauth)
22+
1. [Other](#other)
2223
1. [Provider](#provider)
2324
1. [Provider Reingestion](#provider-reingestion)
2425

@@ -63,6 +64,12 @@ The following are DAGs grouped by their primary tag:
6364
| [`oauth2_authorization`](#oauth2_authorization) | `None` |
6465
| [`oauth2_token_refresh`](#oauth2_token_refresh) | `0 */12 * * *` |
6566

67+
## Other
68+
69+
| DAG ID | Schedule Interval |
70+
| --------------------------------------------------------- | ----------------- |
71+
| [`flickr_thumbnails_removal`](#flickr_thumbnails_removal) | `None` |
72+
6673
## Provider
6774

6875
| DAG ID | Schedule Interval | Dated | Media Type(s) |
@@ -113,6 +120,7 @@ The following is documentation associated with each DAG (where available):
113120
1. [`finnish_museums_workflow`](#finnish_museums_workflow)
114121
1. [`flickr_audit_sub_provider_workflow`](#flickr_audit_sub_provider_workflow)
115122
1. [`flickr_reingestion_workflow`](#flickr_reingestion_workflow)
123+
1. [`flickr_thumbnails_removal`](#flickr_thumbnails_removal)
116124
1. [`flickr_workflow`](#flickr_workflow)
117125
1. [`freesound_workflow`](#freesound_workflow)
118126
1. [`image_data_refresh`](#image_data_refresh)
@@ -394,6 +402,11 @@ Output: TSV file containing the images and the respective meta-data.
394402

395403
Notes: https://www.flickr.com/help/terms/api Rate limit: 3600 requests per hour.
396404

405+
## `flickr_thumbnails_removal`
406+
407+
One-time run DAG to remove progressively all the old Flickr thumbnails, as they
408+
were determined to be unsuitable for the Openverse UI requirements.
409+
397410
## `flickr_workflow`
398411

399412
Content Provider: Flickr

catalog/dags/flickr_thumbs_removal.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
"""
2+
One-time run DAG to remove progressively all the old Flickr thumbnails,
3+
as they were determined to be unsuitable for the Openverse UI requirements.
4+
"""
15
import logging
26
from datetime import timedelta
37
from textwrap import dedent

0 commit comments

Comments
 (0)