Amazing product!

[butzhang]

it works out of the box with lots of features! Thank you. The quality of it make it hard to believe it is a open source project.

it would be nice to abstract the framework so that I can easily build a geniusHarbor. 😄 (https://genius.com/)

[nick-sh-oh]

Hi @butzhang!

Thank you so much for the feedback! I'm thrilled to hear that you find RedditHarbor feature-rich and high-quality. It's incredibly motivating to know that our open-source project is resonating with users like yourself. 🙌

Absolutely, I'd be more than happy to provide some guidance on abstracting the framework for your "geniusHarbor" idea - Would a pseudocode outline be a starting point?

[butzhang]

yeah, that would be great!

[butzhang]

One feedback on the reddit harbor is that sometimes pulling reddit is slow. I don't know if that's limited by reddit api or something else. Happy to contribute if you can shed some lights on it.

[nick-sh-oh]

1. Reddit API speed

The speed of API requests and responses is governed by Reddit's rate-limiting policies, which is outside of our control. While there's not much we can do to directly improve the API speed, one potential workaround is to use two separate API keys. With multiple keys, you can run two separate command-line instances in parallel, effectively doubling the rate limit and potentially improving overall throughput.

2. Pseduocode for pipeline.py

This would be somewhat lengthy, so I will write them down in the comment section below.

3. Contribution

I'm glad to hear your enthusiasm for contributing to the project! One potential area for exploration could be optimising the efficiency of the code, especially in regard to API requests. Your question about API speed prompted me to reconsider whether RedditHarbor is truly "hyper-efficient" in this aspect.

Specifically, it would be beneficial to investigate whether changing the way we use getattr() could further reduce the number of API calls. For example, let's compare the original code in submission_from_user with a potential alternative:

Original Code:

submissions = [submission for submission in getattr(redditor.submissions, "top")(limit=limit)]

for submission in submissions:
    submission_id, submission_inserted = self.submission_data(submission=submission)[:2]

Potential Alternative:

submissions = redditor.submissions.top(limit=limit)

for submission in submissions:
    submission_id, submission_inserted = self.submission_data(submission=submission)[:2]

In the original code, we use getattr(redditor.submissions, sort_type)(limit=limit) to dynamically choose the sorting type based on the sort_type variable. While this approach works, it's worth double-checking whether it might be making unnecessary API requests compared to the alternative, which directly calls the desired sorting method (submissions.top(limit=limit) in this case).

[nick-sh-oh]

Define class Collect:
    Constructor:
        Initialize Reddit client, Supabase client, database table configurations
        Check and create "error_log" folder

    Method redditor_data(praw_models, insert):
        IF insert is True:
            Get redditor ID
            IF redditor ID exists in database:
                PASS
            ELSE:
                Collect redditor data (name, creation date, karma, moderator status, etc.)
                Insert redditor data into database
        ELSE:
            Get redditor ID
        Return redditor ID, boolean indicating if redditor was inserted

    Method submission_data(submission, mask_pii, insert_redditor):
        Get submission ID
        IF submission ID exists in database:
            PASS
        ELSE:
            Call redditor_data method to get redditor ID 
            Collect submission data (title, text, subreddit, flair, awards, etc.)
            IF mask_pii is True:
                Anonymize text content
            Insert submission data into database
        Return submission ID, booleans indicating if submission and redditor were inserted

    Method comment_data(comments, mask_pii, insert_redditor):
        FOR each comment:
            Get comment ID
            IF comment ID exists in database:
                PASS
            ELSE:
                Call redditor_data method to get redditor ID and insert redditor if needed
                Collect comment data (body, score, edited, removed, etc.)
                IF mask_pii is True:
                    Anonymize text content
                Insert comment data into database
        Return counts of inserted comments and redditors

    Method subreddit_submission(subreddits, sort_types, limit, mask_pii):
        FOR each subreddit and sort type:
            FOR each submission:
                Call submission_data method
                Increment counts of inserted submissions and redditors
        Print total counts of inserted submissions and redditors

    Method subreddit_comment(subreddits, sort_types, limit, level, mask_pii):
        FOR each subreddit and sort type:
            FOR each submission:
                IF submission ID is not in comment database:
                    Collect comments up to given level
                    Call comment_data method
                    Increment counts of inserted comments and redditors
        Print total counts of inserted comments and redditors

    Method subreddit_submission_and_comment(subreddits, sort_types, limit, level, mask_pii):
        FOR each subreddit and sort type:
            FOR each submission:
                Call submission_data method
                IF submission ID not in comment database:
                    Collect comments up to given level
                    Call comment_data method
                Increment counts of inserted submissions, comments, and redditors
        Print total counts of inserted submissions, comments, and redditors

    Method submission_from_user(user_names, sort_types, limit, mask_pii):
        FOR each user:
            Get redditor object based on user ID
            FOR each submission of redditor object and sort_type:
                Call submission_data method
                Increment count of inserted submissions
        Print total count of inserted submissions

    Method comment_from_user(user_names, sort_types, limit, mask_pii):
        FOR each user:
            Get redditor object based on user ID
            FOR each comment of redditor object and sort_type:
                Call comment_data method
                Increment count of inserted comments
        Print total count of inserted comments

    Method submission_by_keyword(subreddits, query, limit, mask_pii):
        FOR each subreddit collect submissions based on query:
            Call submission_data method
            Increment count of inserted submissions
        Print total count of inserted submissions

    Method comment_from_submission(submission_ids, level, mask_pii):
        FOR each submission ID:
            IF submission ID is not in comment database:
                Collect comments up to given level
                Call comment_data method
                Increment count of inserted comments
        Print total count of inserted comments