Move Dataset from GitHub

[andrebhu]

The dataset is getting a little large. Even though GitHub's website does a lot of the caching and provides a nice search feature, it may be good to move this to somewhere more usable (Kaggle).

https://www.kaggle.com/docs/datasets

[aarchangel64]

How big is this data? Kaggle says a max size of 200GB. Also I'm not super familiar with Kaggle, is that free? We get free GitHub Pro storage being part of an educational institution, although IIRC GitHub caps repos to 50gb (without git LFS at least).

[andrebhu]

About 512MB, but generally, I think Kaggle may be better just as a general data storage solution (and is free according to ChatGPT, but with requirements).

Can likely add workflow steps to push to Kaggle. It may get higher visibility there, too. This is just an idea, though, if someone wants to pick it up.

Side note: The script doesn't check if there are any duplicate pastes, but given the 6-hour cron, the likelihood of this happening feels low.

[andrebhu]

There's some funky things, recently looked and saw some:

• CloudTrail logs
• buckets
• weird SSH servers
• lots of GDrive document scans
• suspicious PDFs