Batch Trigger upgrades (#1502)

* WIP batch trigger v2

* Fix for the DateField being one month out… getUTCMonth() is zero indexed 🤦‍♂️

* Added a custom date range filter

* Deal with closing the custom date range

* Child runs filter

* Fix for the clear button untoggling the child runs

* WIP batchTriggerV2

* Finished removing rate limit from the webapp

* Added an index TaskRun to make useRealtimeBatch performant

* Renamed the period filter labels to be “Last X mins”

* Denormalize background worker columns into TaskRun

* Use the runTags column on TaskRun

* Add TaskRun ("projectId", "id" DESC) index

* Improved the v2 batch trigger endpoint to process items in parallel and also added a threshold, below which the processing of items is async

* Added a runId filter, and WIP for batchId filter

* WIP triggerAll

* Add new batch methods for triggering multiple different tasks in a single batch

* Disabled switch styling

* Batch filtering, force child runs to show if filtering by batch/run

* Added schedule ID filtering

* Force child runs to show when filtering by scheduleId, for consistency

* realtime: allow setting enabled: false on useApiClient

* Batches page

* Always complete batches, not only batchTriggerAndWait in deployed tasks

* Add batch.retrieve and allow filtering by batch in runs.list

* Renamed pending to “In progress”

* Tidied up the table a bit

* Deal with old batches: “Legacy batch”

* Added the Batch to the run inspector

* Fixed the migration that created the new idempotency key index on BatchTaskRun

* Fixed the name of the idempotencyKeyExpiresAt option and now default idempotency key TTL is 30 days, not 24 hours

* Timezone fix: wrong month in Usage page dropdown

* The DateField now defaults to local time, but can be overriden to use utc with an option

* Don’t allow the task icon to get squished

* BatchFilters removed unused imports

* In the batch filtering, use `id` instead of `batchId` in the URL

* BatchFilters: we don’t need a child tasks hidden input field

* Creates some common filter components/functions

* Fix for batchVersion check when filtering by batch status

* Add additional logging around telemetry and more attributes for trigger spans

* Show clear button for specific id filters

* Batch list: only allow environments that are part of this project

* Unnecessary optional chain

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Add JSDocs

---------

Co-authored-by: Matt Aitken <matt@mattaitken.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

Author: 3 people

.changeset/perfect-onions-call.md

@@ -0,0 +1,80 @@
+---
+"@trigger.dev/sdk": patch
+"trigger.dev": patch
+"@trigger.dev/core": patch
+---
+
+Added new batch.trigger and batch.triggerByTask methods that allows triggering multiple different tasks in a single batch:
+
+```ts
+import { batch } from '@trigger.dev/sdk/v3';
+import type { myTask1, myTask2 } from './trigger/tasks';
+
+// Somewhere in your backend code
+const response = await batch.trigger<typeof myTask1 | typeof myTask2>([
+  { id: 'task1', payload: { foo: 'bar' } },
+  { id: 'task2', payload: { baz: 'qux' } },
+]);
+
+for (const run of response.runs) {
+  if (run.ok) {
+    console.log(run.output);
+  } else {
+    console.error(run.error);
+  }
+}
+```
+
+Or if you are inside of a task, you can use `triggerByTask`:
+
+```ts
+import { batch, task, runs } from '@trigger.dev/sdk/v3';
+
+export const myParentTask = task({
+  id: 'myParentTask',
+  run: async () => {
+    const response = await batch.triggerByTask([
+      { task: myTask1, payload: { foo: 'bar' } },
+      { task: myTask2, payload: { baz: 'qux' } },
+    ]);
+
+    const run1 = await runs.retrieve(response.runs[0]);
+    console.log(run1.output) // typed as { foo: string }
+
+    const run2 = await runs.retrieve(response.runs[1]);
+    console.log(run2.output) // typed as { baz: string }
+
+    const response2 = await batch.triggerByTaskAndWait([
+      { task: myTask1, payload: { foo: 'bar' } },
+      { task: myTask2, payload: { baz: 'qux' } },
+    ]);
+
+    if (response2.runs[0].ok) {
+      console.log(response2.runs[0].output) // typed as { foo: string }
+    }
+
+    if (response2.runs[1].ok) {
+      console.log(response2.runs[1].output) // typed as { baz: string }
+    }
+  }
+});
+
+export const myTask1 = task({
+  id: 'myTask1',
+  run: async () => {
+    return {
+      foo: 'bar'
+    }
+  }
+});
+
+export const myTask2 = task({
+  id: 'myTask2',
+  run: async () => {
+    return {
+      baz: 'qux'
+    }
+  }
+});
+
+```

.changeset/ten-pans-itch.md

@@ -0,0 +1,25 @@
+---
+"@trigger.dev/react-hooks": minor
+"@trigger.dev/sdk": minor
+"@trigger.dev/core": minor
+---
+
+Improved Batch Triggering:
+
+- The new Batch Trigger endpoint is now asynchronous and supports up to 500 runs per request.
+- The new endpoint also supports triggering multiple different tasks in a single batch request (support in the SDK coming soon).
+- The existing `batchTrigger` method now supports the new endpoint, and shouldn't require any changes to your code.
+
+- Idempotency keys now expire after 24 hours, and you can customize the expiration time when creating a new key by using the `idempotencyKeyTTL` parameter:
+
+```ts
+await myTask.batchTrigger([{ payload: { foo: "bar" }}], { idempotencyKey: "my-key", idempotencyKeyTTL: "60s" })
+// Works for individual items as well:
+await myTask.batchTrigger([{ payload: { foo: "bar" }, options: { idempotencyKey: "my-key", idempotencyKeyTTL: "60s" }}])
+// And `trigger`:
+await myTask.trigger({ foo: "bar" }, { idempotencyKey: "my-key", idempotencyKeyTTL: "60s" });
+```
+
+### Breaking Changes
+
+- We've removed the `idempotencyKey` option from `triggerAndWait` and `batchTriggerAndWait`, because it can lead to permanently frozen runs in deployed tasks. We're working on upgrading our entire system to support idempotency keys on these methods, and we'll re-add the option once that's complete.

.changeset/wild-needles-hunt.md

@@ -0,0 +1,7 @@
+---
+"@trigger.dev/react-hooks": patch
+"@trigger.dev/sdk": patch
+"@trigger.dev/core": patch
+---
+
+Added ability to subscribe to a batch of runs using runs.subscribeToBatch

.github/copilot-instructions.md

@@ -0,0 +1 @@
+This is the repo for Trigger.dev, a background jobs platform written in TypeScript. Our webapp at apps/webapp is a Remix 2.1 app that uses Node.js v20. Our SDK is an isomorphic TypeScript SDK at packages/trigger-sdk. Always prefer using isomorphic code like fetch, ReadableStream, etc. instead of Node.js specific code. Our tests are all vitest. We use prisma in internal-packages/database for our database interactions using PostgreSQL. For TypeScript, we usually use types over interfaces. We use zod a lot in packages/core and in the webapp. Avoid enums. Use strict mode. No default exports, use function declarations.

apps/webapp/app/components/navigation/SideMenu.tsx

@@ -12,6 +12,7 @@ import {
   RectangleStackIcon,
   ServerStackIcon,
   ShieldCheckIcon,
+  Squares2X2Icon,
 } from "@heroicons/react/20/solid";
 import { UserGroupIcon, UserPlusIcon } from "@heroicons/react/24/solid";
 import { useNavigation } from "@remix-run/react";
@@ -45,6 +46,7 @@ import {
   projectSetupPath,
   projectTriggersPath,
   v3ApiKeysPath,
+  v3BatchesPath,
   v3BillingPath,
   v3ConcurrencyPath,
   v3DeploymentsPath,
@@ -475,6 +477,13 @@ function V3ProjectSideMenu({
         activeIconColor="text-teal-500"
         to={v3RunsPath(organization, project)}
       />
+      <SideMenuItem
+        name="Batches"
+        icon={Squares2X2Icon}
+        activeIconColor="text-blue-500"
+        to={v3BatchesPath(organization, project)}
+        data-action="batches"
+      />
       <SideMenuItem
         name="Test"
         icon={BeakerIcon}

apps/webapp/app/components/primitives/DateField.tsx

@@ -1,4 +1,4 @@
-import { BellAlertIcon } from "@heroicons/react/20/solid";
+import { BellAlertIcon, XMarkIcon } from "@heroicons/react/20/solid";
 import { CalendarDateTime, createCalendar } from "@internationalized/date";
 import { useDateField, useDateSegment } from "@react-aria/datepicker";
 import type { DateFieldState, DateSegment } from "@react-stately/datepicker";
@@ -12,7 +12,7 @@ const variants = {
   small: {
     fieldStyles: "h-5 text-sm rounded-sm px-0.5",
     nowButtonVariant: "tertiary/small" as const,
-    clearButtonVariant: "minimal/small" as const,
+    clearButtonVariant: "tertiary/small" as const,
   },
   medium: {
     fieldStyles: "h-7 text-base rounded px-1",
@@ -35,9 +35,12 @@ type DateFieldProps = {
   showNowButton?: boolean;
   showClearButton?: boolean;
   onValueChange?: (value: Date | undefined) => void;
+  utc?: boolean;
   variant?: Variant;
 };
 
+const deviceTimezone = Intl.DateTimeFormat().resolvedOptions().timeZone;
+
 export function DateField({
   label,
   defaultValue,
@@ -50,22 +53,23 @@ export function DateField({
   showGuide = false,
   showNowButton = false,
   showClearButton = false,
+  utc = false,
   variant = "small",
 }: DateFieldProps) {
   const [value, setValue] = useState<undefined | CalendarDateTime>(
-    utcDateToCalendarDate(defaultValue)
+    utc ? utcDateToCalendarDate(defaultValue) : dateToCalendarDate(defaultValue)
   );
 
   const state = useDateFieldState({
     value: value,
     onChange: (value) => {
       if (value) {
         setValue(value);
-        onValueChange?.(value.toDate("utc"));
+        onValueChange?.(value.toDate(utc ? "utc" : deviceTimezone));
       }
     },
-    minValue: utcDateToCalendarDate(minValue),
-    maxValue: utcDateToCalendarDate(maxValue),
+    minValue: utc ? utcDateToCalendarDate(minValue) : dateToCalendarDate(minValue),
+    maxValue: utc ? utcDateToCalendarDate(maxValue) : dateToCalendarDate(maxValue),
     shouldForceLeadingZeros: true,
     granularity,
     locale: "en-US",
@@ -78,7 +82,9 @@ export function DateField({
   useEffect(() => {
     if (state.value === undefined && defaultValue === undefined) return;
 
-    const calendarDate = utcDateToCalendarDate(defaultValue);
+    const calendarDate = utc
+      ? utcDateToCalendarDate(defaultValue)
+      : dateToCalendarDate(defaultValue);
     //unchanged
     if (state.value?.toDate("utc").getTime() === defaultValue?.getTime()) {
       return;
@@ -134,23 +140,19 @@ export function DateField({
           <Button
             type="button"
             variant={variants[variant].nowButtonVariant}
-            LeadingIcon={BellAlertIcon}
-            leadingIconClassName="text-text-dimmed group-hover:text-text-bright"
             onClick={() => {
               const now = new Date();
-              setValue(utcDateToCalendarDate(new Date()));
+              setValue(utc ? utcDateToCalendarDate(now) : dateToCalendarDate(now));
               onValueChange?.(now);
             }}
           >
-            <span className="text-text-dimmed transition group-hover:text-text-bright">Now</span>
+            Now
           </Button>
         )}
         {showClearButton && (
           <Button
             type="button"
             variant={variants[variant].clearButtonVariant}
-            LeadingIcon={"close"}
-            leadingIconClassName="-mr-2"
             onClick={() => {
               setValue(undefined);
               onValueChange?.(undefined);
@@ -181,7 +183,7 @@ function utcDateToCalendarDate(date?: Date) {
   return date
     ? new CalendarDateTime(
         date.getUTCFullYear(),
-        date.getUTCMonth(),
+        date.getUTCMonth() + 1,
         date.getUTCDate(),
         date.getUTCHours(),
         date.getUTCMinutes(),
@@ -190,6 +192,19 @@ function utcDateToCalendarDate(date?: Date) {
     : undefined;
 }
 
+function dateToCalendarDate(date?: Date) {
+  return date
+    ? new CalendarDateTime(
+        date.getFullYear(),
+        date.getMonth() + 1,
+        date.getDate(),
+        date.getHours(),
+        date.getMinutes(),
+        date.getSeconds()
+      )
+    : undefined;
+}
+
 type DateSegmentProps = {
   segment: DateSegment;
   state: DateFieldState;

apps/webapp/app/components/primitives/Select.tsx

@@ -440,7 +440,7 @@ export interface SelectItemProps extends Ariakit.SelectItemProps {
 }
 
 const selectItemClasses =
-  "group cursor-pointer px-1 pt-1 text-sm text-text-dimmed focus-custom last:pb-1";
+  "group cursor-pointer px-1 pt-1 text-2sm text-text-dimmed focus-custom last:pb-1";
 
 export function SelectItem({
   icon,
@@ -613,7 +613,7 @@ export function SelectPopover({
         "z-50 flex flex-col overflow-clip rounded border border-charcoal-700 bg-background-bright shadow-md outline-none animate-in fade-in-40",
         "min-w-[max(180px,calc(var(--popover-anchor-width)+0.5rem))]",
         "max-w-[min(480px,var(--popover-available-width))]",
-        "max-h-[min(520px,var(--popover-available-height))]",
+        "max-h-[min(600px,var(--popover-available-height))]",
         "origin-[var(--popover-transform-origin)]",
         className
       )}

apps/webapp/app/components/primitives/Switch.tsx

@@ -13,10 +13,10 @@ const variations = {
   },
   small: {
     container:
-      "flex items-center h-[1.5rem] gap-x-1.5 rounded hover:bg-tertiary pr-1 py-[0.1rem] pl-1.5 transition focus-custom",
+      "flex items-center h-[1.5rem] gap-x-1.5 rounded hover:bg-tertiary disabled:hover:bg-transparent pr-1 py-[0.1rem] pl-1.5 transition focus-custom disabled:hover:text-charcoal-400 disabled:opacity-50 text-charcoal-400 hover:text-charcoal-200 disabled:hover:cursor-not-allowed hover:cursor-pointer",
     root: "h-3 w-6",
     thumb: "h-2.5 w-2.5 data-[state=checked]:translate-x-2.5 data-[state=unchecked]:translate-x-0",
-    text: "text-xs text-charcoal-400 group-hover:text-charcoal-200 hover:cursor-pointer transition",
+    text: "text-xs transition",
   },
 };
 

apps/webapp/app/components/runs/TimeFrameFilter.tsx

@@ -214,6 +214,7 @@ export function AbsoluteTimeFrame({
             granularity="second"
             showNowButton
             showClearButton
+            utc
           />
         </div>
         <div className="space-y-1">
@@ -227,6 +228,7 @@ export function AbsoluteTimeFrame({
             granularity="second"
             showNowButton
             showClearButton
+            utc
           />
         </div>
       </div>