Skip to content

Commit 7c29041

Browse files
kaizenccgithub-actionsrix0rrr
authored
feat: split transaction marker and known versions in npm follower lambda (#1694)
#1693 introduced versioning to the staging bucket but we are now creating a new version ~75 times an hour. The object is 16MB. That's 29GB of data per day we are now storing. Initially the strategy of this PR was to introduce a lifecycle rule on nonCurrentVersions at 30 days, but that's 864GB of data we're committing to storing. Much of this data is the same anyway -- we're creating a new version if the `marker` changes and the `knownVersions` (where all the data is) does not change. Instead this PR approaches the problem by splitting up these two pieces into separate files and only update `knownVersions` if it changes. We then have separate lifecycle rules for each -- we delete the transaction marker noncurrent versions after a day since we don't need them, and keep knownVersions for a month. In my experience, `knownVersions` updates about once an hour. It's 16MB, so 16*24*30 = 11.52GB. At $0.023/month storage cost, this results in $0.26 cost per month. I've tested this in my local env. ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license* --------- Signed-off-by: github-actions <github-actions@github.com> Co-authored-by: github-actions <github-actions@github.com> Co-authored-by: Rico Huijbers <rix0rrr@gmail.com>
1 parent de3baba commit 7c29041

File tree

5 files changed

+717
-101
lines changed

5 files changed

+717
-101
lines changed

0 commit comments

Comments
 (0)