Skip to content

reduce response times, improve caching #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 83 commits into
base: readme-test
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
f979145
add DB logging
martin-mfg Jun 21, 2025
9fc5b57
move DB initialization
martin-mfg Jun 21, 2025
87a79bf
fix DB initialization
martin-mfg Jun 21, 2025
7cc0217
add repeating old requests
martin-mfg Jun 22, 2025
76a9b83
fix repeating requests
martin-mfg Jun 22, 2025
a0060f0
add database bypass header
martin-mfg Jun 22, 2025
23eb3bf
add user_requested_at DB column
martin-mfg Jun 22, 2025
4a41426
try prerender function
martin-mfg Jun 22, 2025
fc9d927
add vercel config.json
martin-mfg Jun 22, 2025
31ca8a3
move core app to .vercel/output/..., adapt only some paths
martin-mfg Jun 23, 2025
03e15ec
adapt some paths
martin-mfg Jun 23, 2025
00787b2
try with 2 prerendering routes
martin-mfg Jun 23, 2025
4b23278
fix: longer refresh token
martin-mfg Jun 23, 2025
3db04a1
moving more files, adapting more paths
martin-mfg Jun 23, 2025
6ead107
adapt paths, try fix dependencies
martin-mfg Jun 24, 2025
22557f1
test with copied dependencies
martin-mfg Jun 24, 2025
bc8d831
test: add axios to router
martin-mfg Jun 24, 2025
effdf06
add outer package-lock.json
martin-mfg Jun 24, 2025
eba0267
add root-level package info
martin-mfg Jun 24, 2025
8a3a1bb
move files back and adapt most paths
martin-mfg Jun 24, 2025
793ceec
revert all remaining unnecessary changes
martin-mfg Jun 24, 2025
6e073e6
try custom build script with .vercel logic, npm install
martin-mfg Jun 24, 2025
497a2e0
fix mv not dealing with dot directory
martin-mfg Jun 24, 2025
9766626
try fix vercel error, use 'core-app' folder
martin-mfg Jun 24, 2025
e5188c8
do not use 'core-app' folder
martin-mfg Jun 24, 2025
51fc0b9
rename dot_vercel_copy, log requires req and res keys
martin-mfg Jun 25, 2025
47e8886
finish adapter for res
martin-mfg Jun 25, 2025
81d7efa
fix DB table creation, remove log line
martin-mfg Jun 25, 2025
f5d0708
make all endpoints available
martin-mfg Jun 25, 2025
d24f44e
try 'isr' subfolder
martin-mfg Jun 25, 2025
df52c0e
fix: adapt build paths for 'isr' folder
martin-mfg Jun 25, 2025
19013dd
debug: try different rewrite path
martin-mfg Jun 25, 2025
a256c73
debug: remove 'api/' from isr paths
martin-mfg Jun 25, 2025
7d69900
debug: fix symlink path
martin-mfg Jun 25, 2025
7fe2d43
back to non-debug commit
martin-mfg Jun 25, 2025
e157b60
try without '/api' folder
martin-mfg Jun 25, 2025
ac3563e
debug: remove 3 symlinks
martin-mfg Jun 25, 2025
a6d52f1
debug: remove trailing slash from symlink
martin-mfg Jun 25, 2025
f4e706b
try fully without trailing slashes
martin-mfg Jun 25, 2025
6311c1a
try different rewrite
martin-mfg Jun 25, 2025
c70abd6
try rewrite with 'api'
martin-mfg Jun 25, 2025
35d7b93
fix: don't mix up rewrite and redirect
martin-mfg Jun 25, 2025
3ceb0fb
try renaming './api'
martin-mfg Jun 25, 2025
a9a052a
adapt router imports
martin-mfg Jun 25, 2025
017db86
fix router import for pat-info
martin-mfg Jun 25, 2025
0dcdeac
try single concrete rewrite
martin-mfg Jun 25, 2025
fb39290
try without 404 from router
martin-mfg Jun 25, 2025
0217967
try without rewrite or 'isr/' prefix
martin-mfg Jun 25, 2025
bba5049
fix build command
martin-mfg Jun 25, 2025
0b69f00
rename back ./api-copy
martin-mfg Jun 25, 2025
a437887
rename ./api in build command
martin-mfg Jun 25, 2025
e87bc7a
debug: rename ./api _first_
martin-mfg Jun 25, 2025
26db144
debug: don't rename ./api twice
martin-mfg Jun 25, 2025
9e5c82e
rename ./api
martin-mfg Jun 25, 2025
ba11864
undo unnecessary .prettierignore change
martin-mfg Jun 26, 2025
2377ec7
fix 'status' endpoints
martin-mfg Jun 26, 2025
6f2773a
fix /up endpoint
martin-mfg Jun 26, 2025
909f11c
add repeat-recent config
martin-mfg Jun 26, 2025
dea0dca
fix repeat-recent code
martin-mfg Jun 26, 2025
3dafe0d
debug: logging in repeat-recent
martin-mfg Jun 26, 2025
0fde0b9
debug: make repeat-recent not ISR
martin-mfg Jun 26, 2025
669701a
make status/ endpoints not ISR
martin-mfg Jun 26, 2025
add8fd9
remove debug logging
martin-mfg Jun 26, 2025
3b9dc32
debug: make api not ISR
martin-mfg Jun 26, 2025
1d94b2e
debug: add rewrite
martin-mfg Jun 26, 2025
0241bfc
debug: re-add api prerender config
martin-mfg Jun 27, 2025
4106948
try fully with 'api-rewrite'
martin-mfg Jun 27, 2025
1a93e8d
debug: try again without api prerender config
martin-mfg Jun 27, 2025
c43fec4
try rewriting only non-ISR endpoints
martin-mfg Jun 27, 2025
f010c7c
add missing parts
martin-mfg Jun 27, 2025
92b2ba2
move api prerender config to right place
martin-mfg Jun 27, 2025
3dcf72c
debug: remove again api prerender config
martin-mfg Jun 27, 2025
6a72494
debug: 30s api prerender expiration
martin-mfg Jun 27, 2025
0e2f019
debug: disable api and pin cache header
martin-mfg Jun 27, 2025
8c95e88
revert test changes
martin-mfg Jun 27, 2025
8987a7f
revert more changes
martin-mfg Jun 28, 2025
57ceaa9
add Github Action with cron job
martin-mfg Jun 28, 2025
4925bc1
fix cron schedule
martin-mfg Jun 28, 2025
e577235
test: trigger every 5 min
martin-mfg Jun 28, 2025
3ead183
test: create copy of new workflow
martin-mfg Jun 28, 2025
bfb92fe
cron: improve names, undo test changes
martin-mfg Jun 28, 2025
233c47a
adapt cache/cron/re-request/ISR times
martin-mfg Jun 29, 2025
fbe6718
change timings a bit
martin-mfg Jun 29, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .github/workflows/repeat-recent-requests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: Trigger server to update cached data
on:
schedule:
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12 or JAN-DEC)
# │ │ │ │ ┌───────────── day of the week (0 - 6 or SUN-SAT)
# │ │ │ │ │
# │ │ │ │ │
# │ │ │ │ │
# * * * * *
- cron: "45 * * * *"
workflow_dispatch:

jobs:
triggerRepeatRecent:
name: Trigger server to update cached data.
runs-on: ubuntu-latest
steps:
- name: Make Request
id: myRequest
uses: fjogeleit/http-request-action@v1
with:
url: "https://github-readme-stats-git-caching-martin-mfgs-projects.vercel.app/api/repeat-recent"
method: "POST"
timeout: "62000"
- name: Show Response
run: |
echo ${{ steps.myRequest.outputs.response }}
echo ${{ steps.myRequest.outputs.status }}
2 changes: 0 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,3 @@ vercel_token
!.vscode/extensions.json
!.vscode/settings.json
*.code-workspace

.vercel
3 changes: 3 additions & 0 deletions _dot_vercel_copy/output/config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"version": 3
}
5 changes: 5 additions & 0 deletions _dot_vercel_copy/output/functions/api.func/.vc-config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"runtime": "nodejs22.x",
"handler": "router.js",
"launcherType": "Nodejs"
}
55 changes: 55 additions & 0 deletions _dot_vercel_copy/output/functions/api.func/router.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import { default as api } from "./api-renamed/index.js";
import { default as gist } from "./api-renamed/gist.js";
import { default as pin } from "./api-renamed/pin.js";
import { default as topLangs } from "./api-renamed/top-langs.js";
import { default as wakatime } from "./api-renamed/wakatime.js";
import { default as repeatRecent } from "./api-renamed/repeat-recent.js";
import { default as patInfo } from "./api-renamed/status/pat-info.js";
import { default as statusUp } from "./api-renamed/status/up.js";

export default async (req, res) => {
// remaining code expects express.js-like request and response objects
res.send = function (data) {
if (typeof data === "object") {
res.setHeader("Content-Type", "application/json");
res.end(JSON.stringify(data));
} else if (typeof data === "string") {
res.end(data);
} else {
res.end(String(data));
}
};
const url = new URL(req.url, "https://localhost");
req.query = Object.fromEntries(url.searchParams.entries());

switch (url.pathname) {
case "/api":
api(req, res);
break;
case "/api/gist":
gist(req, res);
break;
case "/api/pin":
pin(req, res);
break;
case "/api/top-langs":
topLangs(req, res);
break;
case "/api/wakatime":
wakatime(req, res);
break;
case "/api/repeat-recent":
repeatRecent(req, res);
break;
case "/api/status/pat-info":
patInfo(req, res);
break;
case "/api/status/up":
statusUp(req, res);
break;
default:
res.statusCode = 404;
res.end("Not Found");
break;
}
};
5 changes: 5 additions & 0 deletions _dot_vercel_copy/output/functions/api.prerender-config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"expiration": 39600,
"bypassToken": "r3fr3shT0k3n-r3fr3shT0k3n-r3fr3shT0k3n",
"passQuery": true
}
1 change: 1 addition & 0 deletions _dot_vercel_copy/output/functions/api/gist.func
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"expiration": 39600,
"bypassToken": "r3fr3shT0k3n-r3fr3shT0k3n-r3fr3shT0k3n",
"passQuery": true
}
1 change: 1 addition & 0 deletions _dot_vercel_copy/output/functions/api/pin.func
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"expiration": 39600,
"bypassToken": "r3fr3shT0k3n-r3fr3shT0k3n-r3fr3shT0k3n",
"passQuery": true
}
1 change: 1 addition & 0 deletions _dot_vercel_copy/output/functions/api/repeat-recent.func
1 change: 1 addition & 0 deletions _dot_vercel_copy/output/functions/api/status/pat-info.func
1 change: 1 addition & 0 deletions _dot_vercel_copy/output/functions/api/status/up.func
1 change: 1 addition & 0 deletions _dot_vercel_copy/output/functions/api/top-langs.func
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"expiration": 39600,
"bypassToken": "r3fr3shT0k3n-r3fr3shT0k3n-r3fr3shT0k3n",
"passQuery": true
}
1 change: 1 addition & 0 deletions _dot_vercel_copy/output/functions/api/wakatime.func
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"expiration": 39600,
"bypassToken": "r3fr3shT0k3n-r3fr3shT0k3n-r3fr3shT0k3n",
"passQuery": true
}
6 changes: 4 additions & 2 deletions api/gist.js → api-renamed/gist.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import {
import { isLocaleAvailable } from "../src/translations.js";
import { renderGistCard } from "../src/cards/gist-card.js";
import { fetchGist } from "../src/fetchers/gist-fetcher.js";
import { storeRequest } from "../src/common/database.js";

export default async (req, res) => {
const {
Expand Down Expand Up @@ -39,11 +40,12 @@ export default async (req, res) => {
}

try {
await storeRequest(req);
const gistData = await fetchGist(id);

let cacheSeconds = clampValue(
parseInt(cache_seconds || CONSTANTS.TWO_DAY, 10),
CONSTANTS.TWO_DAY,
parseInt(cache_seconds || CONSTANTS.TEN_HOURS, 10),
CONSTANTS.FOUR_HOURS,
CONSTANTS.SIX_DAY,
);
cacheSeconds = process.env.CACHE_SECONDS
Expand Down
4 changes: 3 additions & 1 deletion api/index.js → api-renamed/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import {
} from "../src/common/utils.js";
import { fetchStats } from "../src/fetchers/stats-fetcher.js";
import { isLocaleAvailable } from "../src/translations.js";
import { storeRequest } from "../src/common/database.js";

export default async (req, res) => {
const {
Expand Down Expand Up @@ -89,6 +90,7 @@ export default async (req, res) => {
}

try {
await storeRequest(req);
const showStats = parseArray(show);
const organizations = parseArray(owners);
let repositories = parseArray(repos);
Expand All @@ -115,7 +117,7 @@ export default async (req, res) => {

let cacheSeconds = clampValue(
parseInt(cache_seconds || CONSTANTS.CARD_CACHE_SECONDS, 10),
CONSTANTS.TWELVE_HOURS,
CONSTANTS.FOUR_HOURS,
CONSTANTS.TWO_DAY,
);
cacheSeconds = process.env.CACHE_SECONDS
Expand Down
4 changes: 3 additions & 1 deletion api/pin.js → api-renamed/pin.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import {
} from "../src/common/utils.js";
import { fetchRepo } from "../src/fetchers/repo-fetcher.js";
import { isLocaleAvailable } from "../src/translations.js";
import { storeRequest } from "../src/common/database.js";

export default async (req, res) => {
const {
Expand Down Expand Up @@ -81,6 +82,7 @@ export default async (req, res) => {
}

try {
await storeRequest(req);
const showStats = parseArray(show);
const repoData = await fetchRepo(
username,
Expand All @@ -94,7 +96,7 @@ export default async (req, res) => {

let cacheSeconds = clampValue(
parseInt(cache_seconds || CONSTANTS.PIN_CARD_CACHE_SECONDS, 10),
CONSTANTS.ONE_DAY,
CONSTANTS.FOUR_HOURS,
CONSTANTS.TEN_DAY,
);
cacheSeconds = process.env.CACHE_SECONDS
Expand Down
18 changes: 18 additions & 0 deletions api-renamed/repeat-recent.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import { repeatRecentRequests } from "../src/common/database.js";

export default async (req, res) => {
if (req.method !== "POST") {
res.statusCode = 405;
res.send({ error: "Method Not Allowed" });
return;
}
try {
await repeatRecentRequests();
res.statusCode = 200;
res.send({ message: "Recent requests repeated successfully." });
} catch (error) {
console.error("Error repeating recent requests:", error);
res.statusCode = 500;
res.send({ error: error.message || "Internal Server Error" });
}
};
4 changes: 2 additions & 2 deletions api/status/pat-info.js → api-renamed/status/pat-info.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@
* @file Contains a simple cloud function that can be used to check which PATs are no
* longer working. It returns a list of valid PATs, expired PATs and PATs with errors.
*
* @description This function is currently rate limited to 1 request per 5 minutes.
* @description This function is currently rate limited to 1 request per 3 minutes.
*/

import { logger, request, dateDiff } from "../../src/common/utils.js";
export const RATE_LIMIT_SECONDS = 60 * 5; // 1 request per 5 minutes
export const RATE_LIMIT_SECONDS = 60 * 3; // 1 request per 3 minutes

/**
* @typedef {import('axios').AxiosRequestHeaders} AxiosRequestHeaders Axios request headers.
Expand Down
4 changes: 2 additions & 2 deletions api/status/up.js → api-renamed/status/up.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@
* @file Contains a simple cloud function that can be used to check if the PATs are still
* functional.
*
* @description This function is currently rate limited to 1 request per 5 minutes.
* @description This function is currently rate limited to 1 request per 3 minutes.
*/

import retryer from "../../src/common/retryer.js";
import { logger, request } from "../../src/common/utils.js";

export const RATE_LIMIT_SECONDS = 60 * 5; // 1 request per 5 minutes
export const RATE_LIMIT_SECONDS = 60 * 3; // 1 request per 3 minutes

/**
* @typedef {import('axios').AxiosRequestHeaders} AxiosRequestHeaders Axios request headers.
Expand Down
2 changes: 2 additions & 0 deletions api/top-langs.js → api-renamed/top-langs.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import {
} from "../src/common/utils.js";
import { fetchTopLanguages } from "../src/fetchers/top-languages-fetcher.js";
import { isLocaleAvailable } from "../src/translations.js";
import { storeRequest } from "../src/common/database.js";

export default async (req, res) => {
const {
Expand Down Expand Up @@ -62,6 +63,7 @@ export default async (req, res) => {
}

try {
await storeRequest(req);
const topLangs = await fetchTopLanguages(
username,
parseArray(exclude_repo),
Expand Down
4 changes: 3 additions & 1 deletion api/wakatime.js → api-renamed/wakatime.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import {
} from "../src/common/utils.js";
import { fetchWakatimeStats } from "../src/fetchers/wakatime-fetcher.js";
import { isLocaleAvailable } from "../src/translations.js";
import { storeRequest } from "../src/common/database.js";

export default async (req, res) => {
const {
Expand Down Expand Up @@ -49,11 +50,12 @@ export default async (req, res) => {
}

try {
await storeRequest(req);
const stats = await fetchWakatimeStats({ username, api_domain });

let cacheSeconds = clampValue(
parseInt(cache_seconds || CONSTANTS.CARD_CACHE_SECONDS, 10),
CONSTANTS.SIX_HOURS,
CONSTANTS.FOUR_HOURS,
CONSTANTS.TWO_DAY,
);
cacheSeconds = process.env.CACHE_SECONDS
Expand Down
1 change: 1 addition & 0 deletions jest.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ export default {
transform: {},
testEnvironment: "jsdom",
coverageProvider: "v8",
setupFiles: ["<rootDir>/tests/setup.jest.js"],
testPathIgnorePatterns: ["<rootDir>/node_modules/", "<rootDir>/tests/e2e/"],
modulePathIgnorePatterns: ["<rootDir>/node_modules/", "<rootDir>/tests/e2e/"],
coveragePathIgnorePatterns: [
Expand Down
Loading