Skip to content

Larger updates, fixes and changes to the a/b testing #2980

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Larger updates, fixes and changes to the a/b testing #2980

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e201cdd
Larger updates, fixes and changes to the a/b testing
marcusljf Oct 17, 2025
0d1fa49
Quick change back on the tooltip
marcusljf Oct 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
141 changes: 120 additions & 21 deletions apps/web/lib/api/links/complete-ab-tests.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { recordLink } from "@/lib/tinybird";
import { sendWorkspaceWebhook } from "@/lib/webhook/publish";
import { ABTestVariantsSchema, linkEventSchema } from "@/lib/zod/schemas/links";
import { prisma } from "@dub/prisma";
import { normalizeUrl } from "@dub/utils";
import { Link } from "@prisma/client";
import { waitUntil } from "@vercel/functions";
import { linkCache } from "./cache";
Expand All @@ -15,43 +16,141 @@ export async function completeABTests(link: Link) {

const testVariants = ABTestVariantsSchema.parse(link.testVariants);

const analytics: { url: string; leads: number }[] = await getAnalytics({
event: "leads",
// Fetch composite analytics (clicks, leads, sales, saleAmount) grouped by URLs
const analytics: {
url: string;
clicks: number;
leads: number;
sales: number;
saleAmount: number;
}[] = await getAnalytics({
event: "composite",
groupBy: "top_urls",
linkId: link.id,
workspaceId: link.projectId,
start: link.testStartedAt ? new Date(link.testStartedAt) : undefined,
end: link.testCompletedAt,
});

const max = Math.max(
...testVariants.map(
(test) => analytics.find(({ url }) => url === test.url)?.leads || 0,
),
);
// Aggregate analytics by normalized URL for stable matching with variants
const analyticsByNormalizedUrl = new Map<
string,
{ clicks: number; leads: number; sales: number; saleAmount: number }
>();

// There are no leads generated for any test variant, do nothing
if (max === 0) {
console.log(
`AB Test completed but all results are zero for ${link.id}, doing nothing.`,
);
return;
for (const row of analytics || []) {
const key = normalizeUrl(row.url);
const existing = analyticsByNormalizedUrl.get(key);
if (existing) {
analyticsByNormalizedUrl.set(key, {
clicks: (existing.clicks || 0) + (row.clicks || 0),
leads: (existing.leads || 0) + (row.leads || 0),
sales: (existing.sales || 0) + (row.sales || 0),
saleAmount: (existing.saleAmount || 0) + (row.saleAmount || 0),
});
} else {
analyticsByNormalizedUrl.set(key, {
clicks: row.clicks || 0,
leads: row.leads || 0,
sales: row.sales || 0,
saleAmount: row.saleAmount || 0,
});
}
}

const winners = testVariants.filter(
(test) =>
(analytics.find(({ url }) => url === test.url)?.leads || 0) === max,
);
type VariantMetrics = {
url: string;
clicks: number;
leads: number;
conversions: number; // sales
};

const variants: VariantMetrics[] = testVariants.map((tv) => {
const key = normalizeUrl(tv.url);
const a = analyticsByNormalizedUrl.get(key);
return {
url: tv.url,
clicks: a?.clicks ?? 0,
leads: a?.leads ?? 0,
conversions: a?.sales ?? 0,
};
});

const safeRate = (num: number, den: number) => (den > 0 ? num / den : 0);

const maxConversions = Math.max(0, ...variants.map((v) => v.conversions));

let candidateUrls: string[] = [];

if (maxConversions > 0) {
// Primary path: conversions -> conversion rate -> clicks
const maxConv = maxConversions;
const convCandidates = variants.filter((v) => v.conversions === maxConv);
if (convCandidates.length === 1) {
candidateUrls = [convCandidates[0].url];
} else {
const maxConvRate = Math.max(
0,
...convCandidates.map((v) => safeRate(v.conversions, v.clicks)),
);
const rateCandidates = convCandidates.filter(
(v) => safeRate(v.conversions, v.clicks) === maxConvRate,
);
if (rateCandidates.length === 1) {
candidateUrls = [rateCandidates[0].url];
} else {
const maxClicks = Math.max(0, ...rateCandidates.map((v) => v.clicks));
const clickCandidates = rateCandidates.filter(
(v) => v.clicks === maxClicks,
);
if (clickCandidates.length === 1) {
candidateUrls = [clickCandidates[0].url];
} else {
// Still tied after all tie-breakers: do not pick randomly
candidateUrls = [];
}
}
}
} else {
// Fallback path: all variants have zero conversions
// leads -> lead rate -> clicks
const maxLeads = Math.max(0, ...variants.map((v) => v.leads));
const leadCandidates = variants.filter((v) => v.leads === maxLeads);
if (leadCandidates.length === 1) {
candidateUrls = [leadCandidates[0].url];
} else {
const maxLeadRate = Math.max(
0,
...leadCandidates.map((v) => safeRate(v.leads, v.clicks)),
);
const rateCandidates = leadCandidates.filter(
(v) => safeRate(v.leads, v.clicks) === maxLeadRate,
);
if (rateCandidates.length === 1) {
candidateUrls = [rateCandidates[0].url];
} else {
const maxClicks = Math.max(0, ...rateCandidates.map((v) => v.clicks));
const clickCandidates = rateCandidates.filter(
(v) => v.clicks === maxClicks,
);
if (clickCandidates.length === 1) {
candidateUrls = [clickCandidates[0].url];
} else {
// Still tied after all tie-breakers: do not pick randomly
candidateUrls = [];
}
}
}
}

// this should NEVER happen, but just in case
if (winners.length === 0) {
if (candidateUrls.length !== 1) {
console.log(
`AB Test completed but failed to find winners based on max leads for link ${link.id}, doing nothing.`,
`AB Test completed for ${link.id} but no deterministic winner after tie-breakers. Keeping original destination URL.`,
);
return;
}

const winner = winners[Math.floor(Math.random() * winners.length)];
const winner = { url: candidateUrls[0] };

if (winner.url === link.url) {
return;
Expand Down
15 changes: 14 additions & 1 deletion apps/web/ui/links/destination-url-input.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,20 @@ export const DestinationUrlInput = forwardRef<
const url = getUrlFromString(e.target.value);
if (url) {
// remove trailing slash and set the https:// prefix
formContext.setValue("url", url.replace(/\/$/, ""));
const normalizedUrl = url.replace(/\/$/, "");
formContext.setValue("url", normalizedUrl, {
shouldDirty: true,
});

// If an A/B test exists, keep the first variant in sync
const testVariants = formContext.getValues("testVariants");
if (Array.isArray(testVariants) && testVariants.length > 0) {
formContext.setValue(
"testVariants.0.url" as any,
normalizedUrl,
{ shouldDirty: true },
);
}
}
}}
/>
Expand Down
57 changes: 47 additions & 10 deletions apps/web/ui/links/link-tests.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import useWorkspace from "@/lib/swr/use-workspace";
import { ABTestVariantsSchema } from "@/lib/zod/schemas/links";
import { fetcher } from "@dub/utils";
import { fetcher, normalizeUrl } from "@dub/utils";
import { motion } from "motion/react";
import { memo, useMemo } from "react";
import useSWR from "swr";
Expand Down Expand Up @@ -54,6 +54,42 @@ export const LinkTests = memo(({ link }: { link: ResponseLink }) => {
},
);

const analyticsByNormalizedUrl = useMemo(() => {
if (!data) return null;

const map = new Map<
string,
{
clicks: number;
leads: number;
sales: number;
saleAmount: number;
}
>();

for (const row of data) {
const key = normalizeUrl(row.url);
const existing = map.get(key);
if (existing) {
map.set(key, {
clicks: existing.clicks + (row.clicks ?? 0),
leads: existing.leads + (row.leads ?? 0),
sales: existing.sales + (row.sales ?? 0),
saleAmount: existing.saleAmount + (row.saleAmount ?? 0),
});
} else {
map.set(key, {
clicks: row.clicks ?? 0,
leads: row.leads ?? 0,
sales: row.sales ?? 0,
saleAmount: row.saleAmount ?? 0,
});
}
}

return map;
}, [data]);

if (!testVariants || !testVariants.length) return null;

return (
Expand All @@ -65,7 +101,8 @@ export const LinkTests = memo(({ link }: { link: ResponseLink }) => {
>
<ul className="flex flex-col gap-2.5 border-t border-neutral-200 bg-neutral-100 p-3">
{testVariants.map((test, idx) => {
const analytics = data?.find(({ url }) => url === test.url);
const normalizedTestUrl = normalizeUrl(test.url);
const analytics = analyticsByNormalizedUrl?.get(normalizedTestUrl);

return (
<li
Expand All @@ -86,14 +123,7 @@ export const LinkTests = memo(({ link }: { link: ResponseLink }) => {
</span>
</div>

<div className="flex items-center gap-5">
{/* Test percentage */}
<div className="h-7 shrink-0 select-none rounded-[6px] border border-neutral-200/50 p-px">
<div className="flex size-full items-center justify-center rounded-[5px] bg-gradient-to-t from-neutral-950/5 px-1.5 text-xs font-semibold tabular-nums text-neutral-800">
{Math.round(test.percentage)}%
</div>
</div>

<div className="flex items-center gap-3">
{/* Analytics badge */}
<div className="flex justify-end sm:min-w-48">
{isLoading ? (
Expand All @@ -112,6 +142,13 @@ export const LinkTests = memo(({ link }: { link: ResponseLink }) => {
/>
)}
</div>

{/* Test percentage */}
<div className="h-7 shrink-0 select-none rounded-[6px] border border-neutral-200/50 p-px">
<div className="flex size-full items-center justify-center rounded-[5px] bg-gradient-to-t from-neutral-950/5 px-1.5 text-xs font-semibold tabular-nums text-neutral-800">
{Math.round(test.percentage)}%
</div>
</div>
</div>
</li>
);
Expand Down
31 changes: 28 additions & 3 deletions apps/web/ui/links/tests-badge.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"use client";

import { Flask } from "@dub/ui/icons";
import { cn } from "@dub/utils";
import { cn, formatDateTime } from "@dub/utils";
import * as HoverCard from "@radix-ui/react-hover-card";
import { useLinkCardContext } from "./link-card";
import { ResponseLink } from "./links-container";
Expand All @@ -13,16 +13,41 @@ export function TestsBadge({
}) {
const { showTests, setShowTests } = useLinkCardContext();

const completedAtDate = link.testCompletedAt
? new Date(link.testCompletedAt)
: null;
const formattedDate = completedAtDate
? formatDateTime(completedAtDate, {
month: "short",
day: "numeric",
year: "numeric",
})
: undefined;

return (
<div className="hidden sm:block">
<HoverCard.Root openDelay={100}>
<HoverCard.Portal>
<HoverCard.Content
side="bottom"
sideOffset={8}
className="animate-slide-up-fade z-[99] items-center overflow-hidden rounded-xl border border-neutral-200 bg-white p-2 text-sm text-neutral-700 shadow-sm"
className="animate-slide-up-fade z-[99] overflow-hidden rounded-xl border border-neutral-200 bg-white p-3 text-sm text-neutral-700 shadow-sm"
>
A/B tests
{formattedDate ? (
<div className="text-center">
<p className="font-semibold text-neutral-900">
A/B test is running
</p>
<p className="mt-1 text-neutral-600">
Scheduled completion date is{" "}
<span className="font-medium text-neutral-800">
{formattedDate}
</span>
</p>
</div>
) : (
<p className="text-center">A/B tests</p>
)}
Comment on lines +36 to +50
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Missing feature: 6-week completion date warning.

The conditional rendering correctly shows the test status and completion date. However, according to the PR objectives, "Completion date visual warning: if completion date is set beyond six weeks, the note turns red." This logic is not implemented.

Apply this diff to add the 6-week warning:

  const formattedDate = completedAtDate
    ? formatDateTime(completedAtDate, {
        month: "short",
        day: "numeric",
        year: "numeric",
      })
    : undefined;
+  const isBeyondSixWeeks = completedAtDate
+    ? completedAtDate.getTime() > Date.now() + 6 * 7 * 24 * 60 * 60 * 1000
+    : false;

  return (
    <div className="hidden sm:block">
      <HoverCard.Root openDelay={100}>
        <HoverCard.Portal>
          <HoverCard.Content
            side="bottom"
            sideOffset={8}
            className="animate-slide-up-fade z-[99] overflow-hidden rounded-xl border border-neutral-200 bg-white p-3 text-sm text-neutral-700 shadow-sm"
          >
            {formattedDate ? (
              <div className="text-center">
                <p className="font-semibold text-neutral-900">
                  A/B test is running
                </p>
-                <p className="mt-1 text-neutral-600">
+                <p className={cn("mt-1", isBeyondSixWeeks ? "text-red-600" : "text-neutral-600")}>
                  Scheduled completion date is{" "}
-                  <span className="font-medium text-neutral-800">
+                  <span className={cn("font-medium", isBeyondSixWeeks ? "text-red-800" : "text-neutral-800")}>
                    {formattedDate}
                  </span>
                </p>
              </div>
            ) : (
              <p className="text-center">A/B tests</p>
            )}
          </HoverCard.Content>

Committable suggestion skipped: line range outside the PR's diff.

</HoverCard.Content>
</HoverCard.Portal>
<HoverCard.Trigger>
Expand Down
Loading
Loading