You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat: split transaction marker and known versions in npm follower lambda (#1694)
#1693 introduced versioning to the staging bucket but we are now
creating a new version ~75 times an hour. The object is 16MB. That's
29GB of data per day we are now storing.
Initially the strategy of this PR was to introduce a lifecycle rule on
nonCurrentVersions at 30 days, but that's 864GB of data we're committing
to storing. Much of this data is the same anyway -- we're creating a new
version if the `marker` changes and the `knownVersions` (where all the
data is) does not change.
Instead this PR approaches the problem by splitting up these two pieces
into separate files and only update `knownVersions` if it changes. We
then have separate lifecycle rules for each -- we delete the transaction
marker noncurrent versions after a day since we don't need them, and
keep knownVersions for a month.
In my experience, `knownVersions` updates about once an hour. It's 16MB,
so 16*24*30 = 11.52GB. At $0.023/month storage cost, this results in
$0.26 cost per month.
I've tested this in my local env.
----
*By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache-2.0 license*
---------
Signed-off-by: github-actions <github-actions@github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Rico Huijbers <rix0rrr@gmail.com>
0 commit comments