GSoC'25 idea: IHR status page

[romain-fontugne]

Things breaks easily, so we need a status page that tells us which services are running as usual and which one are not. This page could also provide a machine-readable JSON object so that we can automate email alerts when something is not running as usual. It would also be nice to keep historical availability data.

We provide datasets in different formats (e.g., RUST API, files on an FTP server), which need to be monitored. Thus, there can be different liveness checks, e.g., that an API responds (and also gives results), that a file exists, etc.

The project should use, or build on top of, an existing monitoring solution, should be maintainable, and extendable if we add new datasets in the future.

Expected results:

• Find a suitable monitoring tool that can be used as a base for this project
• Implement a status page for all API endpoints and results, websites (ihr, ihr-archive, iyp), and the IYP database
• Add a machine-readable format of the page
• Automatic email notification when something is down

Knowledge Prerequisite:

• Overview of the IHR ecosystem, see Resources below
• Docker
• Web development experience

Resources:

• https://github.com/InternetHealthReport/
• IHR API
• IHR Archive
• IYP website
• IYP database

[Aryash2003]

To implement status page I would be using HTTPS request using python request library to process the running services on the website and write a python script generate the response email when there is a service failure and export JSON file for the same.
For API endpoints I will be using method GET and find all services running status and export them on the page which will get dynamically updated.
For frontend I will be integrating JavaScript to fetch JSON endpoints data and show them.
I think this would be a great project to contribute in.

[dpgiakatos]

@romain-fontugne are we considering using an open-source tool for this, such as Upptime (https://upptime.js.org/), or do we want to re-implement it? Since there are already existing open-source tools available, it might be more efficient to use one of them and modify it, rather than reinventing the wheel.

[Avr8]

Hi sorry to drop in directly, @dpgiakatos @romain-fontugne i feel like using an opensource tool like uptime-kuma can help do the same, Uptime Kuma is a self-hosted monitoring tool that can:

Monitor HTTP(s), TCP, DNS, and Ping endpoints.
Provide a real-time status page.
Send notifications via Slack, Teams, Discord, and more.

Essentially what we can do is divide the setup into two parts...one is end-user facing status page,and other is our centralised monitoring platform..this tool can hit all the end service endpoints, and update the status dynamically...we can also use visualization to make the status page more appeling for instance if our domain gives 5xx we can display red dot and if the url is timing out or response is slow yellow dot and then if everything is all right green. something like this..we can divide region wise maybe

We can have two different pages, looking at the architecture of ihr...havent dwell deep, happy to discuss but lets say there is issue with one of the atlas producer or there is lag in kafka we can get to know by hitting the service we can also configure escalation policies to ensure that critical alerts are addressed promptly and then this tool gives slack,teams alert notification right out of the box.

So essentially, two things with one project ..status page end user facing and centralised monitoring system.

I am a SRE engineer from india by the way :)
Have done something similar before using prometheus blackbox and grafana for visualization but yeah its doable,let me know if you guys interested to discuss more about it, we can maybe do a small poc for this and see.

[itz-rajshekhar18]

@romain-fontugne This is my pathway for the problem statement (Idea)
Project Details
Goal
Develop a system to monitor and display the status of:

APIs: IHR, IHR-Archive, and IYP
Websites: IHR, IHR-Archive, and IYP
Database: IYP database
This includes:

Real-time health checks.
JSON output for automation.
Automatic email notifications during downtimes.
Plan

1. API and Website Monitoring

Perform periodic health checks by:
Sending HTTP requests to API endpoints and websites.
Checking response status codes (e.g., 200 OK vs errors like 500 or 404).
Measuring response time to detect latency issues.
For the database, periodically execute a test query to ensure connectivity.
2. Machine-Readable Format

Create a JSON endpoint (/status.json) that provides real-time status in a structured format

3. Automated Alerts

Set up an email notification system:
Use libraries like NodeMailer (Node.js) or smtplib (Python) to send alerts.
Trigger alerts when:
A service goes down.
Latency exceeds a predefined threshold (e.g., >500ms).

Here’s how you can refine your proposal into a detailed and professional format:

Proposal for GSoC: Centralized Status Dashboard
Introduction
Hi, I'm Chayan Das, a 2nd-year undergraduate student at IIIT Gwalior, and I’m thrilled to contribute to this project. To address the need for real-time service monitoring, I propose developing a Centralized Status Dashboard to ensure the reliability and quick resolution of issues in critical services like APIs, websites, and databases.

The dashboard will provide a user-friendly interface displaying service statuses, a machine-readable JSON format for automation, and automated alerts for instant notifications during outages.

Project Details
Goal
Develop a system to monitor and display the status of:

APIs: IHR, IHR-Archive, and IYP
Websites: IHR, IHR-Archive, and IYP
Database: IYP database
This includes:

Real-time health checks.
JSON output for automation.
Automatic email notifications during downtimes.
Plan

1. API and Website Monitoring

Perform periodic health checks by:
Sending HTTP requests to API endpoints and websites.
Checking response status codes (e.g., 200 OK vs errors like 500 or 404).
Measuring response time to detect latency issues.
For the database, periodically execute a test query to ensure connectivity.
2. Machine-Readable Format

Create a JSON endpoint (/status.json) that provides real-time status in a structured format:
json
Copy
Edit
{
"services": [
{
"name": "IHR API",
"type": "API",
"status": "Running",
"response_time": "150ms",
"last_checked": "2025-01-22T10:00:00Z"
},
{
"name": "IYP Database",
"type": "Database",
"status": "Down",
"error": "Connection Timeout",
"last_checked": "2025-01-22T10:00:00Z"
}
]
}
3. Automated Alerts

Set up an email notification system:
Use libraries like NodeMailer (Node.js) or smtplib (Python) to send alerts.
Trigger alerts when:
A service goes down.
Latency exceeds a predefined threshold (e.g., >500ms).

4. Frontend Dashboard

Build a simple web interface to display:
Service names.
Current status (Running/Down).
Last checked time.
Response time.
Use technologies like HTML/CSS, Bootstrap, or React for a clean UI.

5. Deployment and Maintenance

Deploy the system on platforms like Heroku, AWS, or Vercel.
Use a task scheduler (e.g., cron jobs or Celery) to automate periodic health checks.

Benefits
Transparency: Provides a single source of truth for service status.
Efficiency: Automates incident detection and notification.
Reliability: Helps reduce downtime by enabling quick identification and resolution of issues.

[romain-fontugne]

Hi all,
This is not really the place to draft your proposal (and we don't really need ChatGPT opinion neither). Instead, let us know if you have specific questions about the idea.

[Abhsihekkaul]

@romain-fontugne
I want to work on this but i need some clarifications

1. Will I use (React or Vue.js) since the IHR website is build on top of Vue.js it could be better to make the UI using Vue to maintain consistancy
2. What sort of UI are we looking for the dashboard
3. Are there any existing monitoring tools that the team prefers (e.g., Prometheus, Grafana, UptimeRobot)? since I have the experience using grafana and i think it is one of the best bcuz of the following reasons
   -Grafana dashboard displaying real-time service status
   -Historical tracking of uptime/downtime
   -Automated alerts for failures
4. are we considering to having a bot there as well.
5. What defines a service as "down"? - Is it a 500 response, timeout, missing files, or something else?
6. and how exactly we want the reports
   like do we need to get all the reports like a weekly newsletter kind of thing!

And, is there anywhere we are going to implement it so that we could at least begin building it from now!
I understood GSOC has quite a good amount of time but since this project is only 90 hours long it could be better to begin some kind of development now as by that time we could reach to a certain level where we could make it so good

I am very excited to contribute to IHR. I'm also looking into the IHR-website project, but if that doesn't work out, I am fully committed to this one.

[dpgiakatos]

Hello,

I am answering below:

1-3: From the project's description, "The project should use, or build on top of, an existing monitoring solution, should be maintainable, and extendable if we add new datasets in the future," and from the expected results, "Find a suitable monitoring tool that can be used as a base for this project."
4. Not applicable in this phase
5. "Service down" does not only mean transition HTTP errors, but also instances where the API is working but does not return data, for example
6. The report system should report when a system is down

As for the development phase, we cannot rule out the possibility of developing a prototype, but keep in mind that developing a prototype does not guarantee selection (this applies to all our GSoC projects). Instead, focus on preparing a good proposal that can be completed during the GSoC period.

Thanks.

[trinetra110]

@romain-fontugne @dpgiakatos
Hi, I have been reading the docs of IHR since a few days, and I'm interested in this project. Currently, I'm analysing the steps for making the project. I will ask here later if I need more clarity.

Can you please tell me whether I can send a draft proposal to admin@ihr.live within a few days so that you guys can review it?

Thank you.

[dpgiakatos]

Hi, it's better to send your draft proposal to the emails listed below this project in the GSOC ideas file on our gsoc repo. Please keep in mind that we can only provide feedback on your proposal once (due to the large number of potential GSoC candidates), so make sure it's in good shape before sending it. Thanks.

[trinetra110]

Ok thanks.

[AayushmanChulet]

Hey @romain-fontugne @dpgiakatos ,
I have researched the problem statement and found some possible approaches. However, I still have some doubts, which I have listed below:

1. Do we need to create a custom status page layout, or can we use the existing template for the project?
   a) If we can use the default template with some necessary modifications, these changes can be made by updating the config file:
   ( https://upptime.js.org/docs/configuration#status-website )
   Here are some organizations using Upptime with the default template : ( https://status.forwardemail.net/ ) and ( https://status.chatwoot.com/ ).
   b) We can update the legacy code using this repository, which is built with the Svelte framework: ( https://github.com/upptime/status-page )
   ( https://github.com/orgs/upptime/discussions/257 ).
   c) Alternatively, we can embed it on the website. For further clarification, I have included an example JSON file:
   ( https://raw.githubusercontent.com/AayushmanChulet/upptime/master/history/summary.json )
   ( https://raw.githubusercontent.com/AayushmanChulet/upptime/master/graphs/iyp/response-time-week.png )

2. For IHR Archive Data, do we only need to make an HTTP request to https://archive.ihr.live/ihr/ ?

3. For the IYP database, do we have to make a TCP ping request to iyp-bolt.iijlab.net on port 443?

Additionally, for the IHR API page, we have to check whether the endpoint is live and sends a response containing some data.

I would also like to confirm if I am missing anything or if there are additional requirements for these endpoints and would like to know more about them.

[dpgiakatos]

Hello,

I am answering below:

1. It's already been answered on this discussion page.
2. Yes and no, getting a response from the root does not necessarily mean that the other endpoints will be working or will be able to retrieve the data. However, since it contains static files, I guess getting a response from the root means that the site is up. You have to decide what's more suitable for this case.
3. You have to discover it by yourself. The IYP offers both a WebSocket and an HTTP API. We prefer the HTTP API.
4. Yes, you have to check whether the endpoint is live and sends a response containing some data.

Thanks.

[kishorprajapati1212]

Hello,
I'm interested in contributing to the project for GSoC and would love to get more details.

1. Which parts of the codebase need refactoring the most?
2. Are there any existing discussions or issues related to refactoring?
3. Is there a specific coding style or design pattern preferred for the refactored components?
4. Are there any known challenges or blockers in refactoring the existing components?

Thank you

[dpgiakatos]

There isn't a codebase for this project. I don't understand your questions.

[kishorprajapati1212]

I am researching on this project "Improve the IHR Website Codebase" so which codebase wants to refctoring as per priority.

[dpgiakatos]

This discussion page is for "GSoC'25 idea: IHR status page".

[Keehinde678]

Hello @romain-fontugne!
I am writing this comment to show my interest in the Health API Status Monitor project. After reading through the project details, I would like to clarify a few aspects and express my focus on the automatic email notification system when an API or service goes down.
As mentioned earlier, I am particularly interested in this part of the project because I believe the notification feature will be important to ensuring the smooth operation of the monitoring system, most especially for users who rely heavily on real-time updates about the status of health-related APIs. For this project, I intend to identify API failures by using Python's requests library to send periodic HTTP requests to the monitored API. If the API fails (e.g., it returns a non-200 status code or does not respond at all), I will log the failure, and for the email notification integration, I would like to use Python's smtplib to send email notifications. I am considering using a service like SendGrid for handling the email sending, as it may simplify the process and improve reliability.
However, I would like to know if there are any specific libraries or tools to mention when drafting my project proposal.

[dpgiakatos]

For the email notifications, we will use our own email server. You need to identify libraries that can be configured to work with a self-hosted mail server. Additionally, we have explained multiple times that an endpoint may function but return empty data. This means we also need to implement a notification system to alert users when the service is down. Thanks.

[MaroB05]

@romain-fontugne @dpgiakatos
I would like to know exactly what are the services we have?
and what exactly would we be checking?
below is a list of the services which would be checked with the following format:

Service type:

• things to be checked
• • things I am not sure if it is needed to be checked or not, please let me know

there are:

RESTful APIs:

• Endpoints (up/down)
• Endpoint's responses validation (correct or not)

Databases:

• up/down
• • would we validate responses

FTP servers:

• responded or not

• • do specific files exist or not?
• • compare to a known list of files

• are those all of the services/checks?

• what else should be checked?

• Would historical data be important? (response time, response status)

thanks, and please validate the points I am not sure about.

[MaroB05]

I guess there is a a bit of confusion here😅
I would like to contribute to the project, I am not a mentor.

Good work, though, I wish you good luck.

[coderkhushal]

My bad 😅 . Thanks for your advice though.

[dpgiakatos]

Hello @MaroB05, I am a bit confused from your questions. If I miss something please follow up here.

I would like to know exactly what are the services we have?

We describe IHR architecture in out handbook which is available in github.

what exactly would we be checking?

I am not sure I understand your question.

are those all of the services/checks?
what else should be checked?

It's already answered in this page and in our handbook.

Would historical data be important? (response time, response status)

Can you elaborate on this?

Thanks.

[dpgiakatos]

Hello @coderkhushal, please avoid spamming your proposal here. This page is a place for questions. Thanks.

[Keehinde678]

Thank you all for the feedback. Also I would like to know how am going to submit my proposal. Am I submitting it together with application on GSOC or to one of mentors here?

[dpgiakatos]

Hello, please read gsoc official instructions.