UPDATE (February 4, 2024): This is the discussion about this project on HN: here. Please specifically read @dang's comment regarding the core assumption of this project: here. On a personal note, the number of Stories removed yesterday (Saturday, February 3, 2024) was the lowest ever recorded by the service. This includes 2 duplicate Stories. As a side note, in the list always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
The purpose of this project is to try to understand the type and scale of the moderation of the Hacker News Front Page.
NOTE: I love Hacker News. I try to read it every day. In the case of OnnxStream (here for example), 95% of the comments were helpful and intelligent. I also understand that moderating a site with huge traffic and where users are basically anonymous must be a very difficult task.
Returning to the purpose of this project, from what I have been able to see, the "public" (i.e. observable from the outside) moderation of the Front Page consists of two main tools: modification of the title of a Story (voluntarily or involuntarily influencing its growth in terms of rank) or directly its removal.
Regarding the first type of moderation, an excellent site is already available that tracks changes to Story titles. Here instead I will focus on the second type.
For the reasons explained in the "Why?" section below, I have developed a small application that logs all the Stories that are removed from the Front Page, for personal use. I later discovered that there is no tool/website that provides this type of information and I decided to make it public here. It was a difficult decision but my rationale is: is it better to have more transparency or less transparency?
If you know of a tool/website similar to this, please let me know: I will archive this repo or set it to private.
A possible very positive outcome for this project could be to have a list similar to this, but available directly among the HN lists. Or even to notify a user when a Story is penalized on the Front Page, perhaps indicating the number of flags and/or the reason, for example.
Feel free to skip this part or click to expand
A friend of mine posted two Stories on Hacker News related to OnnxStream (31 days apart), the first related to SDXL Turbo support and the second related to TinyLlama and Mistral 7B support.
In the case of the first, the Story was among the first on the Front Page, until its title was changed from "Stable Diffusion Turbo on a Raspberry Pi Zero 2 generates an image in 29 minutes" to "OnnxStream: Stable Diffusion XL 1.0 Base on a Raspberry Pi Zero 2". This effectively "killed" the Story. One user pointed out that the new title didn't reflect the spirit of the Story (thanks @practice9).
In the case of the second, the Story was in third place on the Front Page, less than an hour after the submission. In this case it was simply removed from the Front Page.
Having discovered this, perplexed, I sent an email to the moderator. @dang, who was very kind and quick in his response, explained to me that the Story had been flagged by users even without being explicitly [flagged], and that he could therefore only hypothesize the causes of the flag. His hypothesis was that (some?) users might be fed up with news related to LLMs.
While I have no reason to doubt Daniel's good faith, it's hard to believe that HN users would be tired of LLM-related news.
So I decided to develop a small console application to determine the frequency of this phenomenon (actually I was also motivated by the prospect of writing some C# code, after more than 2 years of complete abstinence). I subsequently discovered that there were no tools/websites that monitored this specific phenomenon and I therefore decided to make it public here.
Using the official HN API, the service fetches 90 Top Stories every minute and makes a comparison with the first 30 Top Stories (i.e. the Front Page) fetched the previous minute. It logs all missing Stories here. The assumption is that a Story cannot go from the top 30 to a position greater than 90 in a single minute, without having been explicitly removed. If a Story reappears on the Front Page, it is removed from this log. All Stories present in the second-chance pool are excluded from the log. Title and URL are those from when the Story first appeared in the top 30. The number of points and comments and the rank are those from when the Story was removed from the Front Page. The ID points to the news.social-protocols.org page for that Story, which provides a graph of the Story's position on the Front Page over time.
NOTE: always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
- 44854623 #15 6 points 2 comments -> The 5 stages of SaaS Death
- 44855530 #29 18 points 10 comments -> NASA finds multi-billion-year-old 'coral' on Mars
- 44857135 #25 6 points 3 comments -> ICE Took Half Their Work Force. What Do They Do Now?
- 44857252 #7 7 points 0 comments -> The Drone and AI Delusion
- 44857205 #14 30 points 41 comments -> South Korea's military has shrunk by 20% in six years as male population drops
- 44858944 #26 12 points 0 comments -> Israel Killed the last journalist sanding In North Gaza
- 44858915 #29 -> Prominent Al Jazeera journalist killed in Israeli airstrike on Gaza
- 44819346 #13 4 points 3 comments -> Microsoft's Windows XP Crocs
- 44860589 #23 8 points 0 comments -> Five journalists killed in targeted Israeli airstrike on Gaza
- 44862263 #2 8 points 2 comments -> Cursor's Go-to-Market Playbook: How an AI Coding Assistant Hit $100M+ ARR
- 44827617 #22 7 points 6 comments -> Against the UUID
- 44863303 #6 28 points 40 comments -> Why Is Web Performance Undervalued?
- 44863568 #19 28 points 40 comments -> Justice Dept. Settles with Greystar to End Participation in Algorithmic Pricing
- 44864171 #14 10 points 0 comments -> Whistleblow HN: Ollama is being deceitful and everyone should know
- 44865319 #27 9 points 0 comments -> Hard Drive Shortage in Canada?
- 44866970 #17 23 points 9 comments -> Cloudflare Is Not a CDN
- 44863487 #1 618 points 203 comments -> Wikimedia Foundation Challenges UK Online Safety Act Regulations
- 44870502 #23 8 points 1 comments -> The GPT-5 rollout has been a big mess
- 44833223 #22 7 points 2 comments -> Dollar Street – Photos from families with different incomes
- 44838029 #16 14 points 12 comments -> You're Wrong About Dates – and Your Code Is Lying to You
- 44843774 #8 21 points 2 comments -> Reflecting on My Failure to Build a Billion-Dollar Company (2019)
- 44836560 #14 6 points 0 comments -> OOP is not Design Patterns
- 44876743 #20 6 points 1 comments -> Entry-Level Jobs Are Disappearing Fast Because of AI
- 44878744 #20 17 points 3 comments -> OpenAI Burns the Boats
- 44881440 #24 12 points 18 comments -> Show HN: I accidentally built a startup idea validation tool
- 44883585 #16 29 points 17 comments -> Please Don't Promote Wayland
- 44884107 #12 23 points 24 comments -> GPT-5 is a joke. Will it matter?
- 44885367 #18 22 points 8 comments -> Sheet0, a data agent transform webpages to structured spreadsheets
- 44887830 #5 38 points 9 comments -> The number of ICE flights is skyrocketing – but the planes are harder to track
- 44887941 #23 24 points 11 comments -> Just how much has DOGE exaggerated its numbers? Now we have receipts
- 44889038 #8 6 points 3 comments -> We caught companies making it harder to delete your personal data online
- 44889050 #4 8 points 12 comments -> The case for shorter .com domains
- 44891139 #15 18 points 2 comments -> Trump Just Militarized the Capital – What Comes Next?
- 44891107 #22 8 points 1 comments -> Can You Gerrymander Your Party to Power?
- 44891499 #15 4 points 0 comments -> What Is America, and for Whom?
- 44891655 #23 10 points 6 comments -> Tesla Diner Drops Most Menu Options and Cuts Hours Just Weeks After Opening
- 44891955 #25 10 points 0 comments -> Why israel must hold itself to account and how it can be made to do so
- 44891997 #20 27 points 42 comments -> Man develops rare condition after ChatGPT query over stopping eating salt
- 44892592 #13 10 points 3 comments -> The Poison Within Patriotism
- 44892580 #25 43 points 29 comments -> Fuse is 95% cheaper and 10x faster than NFS
- 44889073 #5 172 points 70 comments -> Pebble Time 2* Design Reveal
- 44892497 #25 46 points 6 comments -> DeepKit Story: how $160M company killed EU trademark for a small OSS project
- 44897802 #2 8 points 1 comments -> Optimizing Your Debian 13 Desktop
- 44897885 #26 5 points 1 comments -> Over 100 people dead in Gaza in 24 hours, Gaza officials claim
- 44869184 #7 84 points 88 comments -> Debian 13 arrives with major updates for Linux users – what's new in 'Trixie'
- 44858693 #22 32 points 13 comments -> A telephony agent for my parents. Should I turn it into a full-fledged service?
- 44897682 #29 6 points 8 comments -> AI Efficiency? Give Me a Break
- 44898283 #16 6 points 7 comments -> Social Media Apps Engineered Like Narcotics: The Dopamine Addiction Crisis
- 44898812 #24 2 points 0 comments -> Program the BBC Micro in C with VBCC for 6502 on macOS Using Docker
- 44900202 #14 7 points 0 comments -> Kubernetes Resource Optimization Strategies That Work in Production
- 44868353 #22 14 points 1 comments -> Great Myths #16: The Conflict Thesis
- 44903290 #2 59 points 16 comments -> Meta appoints anti-LGBTQ+ conspiracy theorist Robby Starbuck as AI bias advisor
- 44898955 #12 158 points 1 comments -> Org-social is a decentralized social network that runs on an Org Mode
- 44906397 #29 6 points 2 comments -> Palantir might be the most overvalued firm of all time
- 44911394 #3 9 points 0 comments -> The AI Job Swindle
- 44911325 #2 10 points 12 comments -> It is time to 'Correct the Map'
- 44911872 #30 4 points 0 comments -> Review: Dating Men in the Bay Area
- 44913726 #15 12 points 7 comments -> Have you ever noticed that people dressed better in the past?
- 44913606 #11 18 points 6 comments -> SC's proposed nuclear reboot: 'We're going to finish these reactors'
- 44914435 #13 43 points 28 comments -> German court declares Karl Marx's teachings unconstitutional
- 44915015 #15 34 points 12 comments -> Are We Creating Entrepreneurs or Just Privileged Risk-Takers?
- 44915090 #13 28 points 8 comments -> HTTP/1.1 must die: the desync endgame
- 44917011 #13 -> ICE Adds Random Person to Group Chat, Exposes Details of Manhunt in Real-Time
- 44894302 #22 7 points 1 comments -> Dicing an Onion, the Mathematically Optimal Way
- 44920245 #2 9 points 0 comments -> Single Sign on for Furries