Give a way to access the `escapedWorkerSchema` or get a job by its key

[GerkinDev]

Feature description

Facilitate retrieval of jobs in graphile, by either giving ways to query the schema or exposing new helpers to do so in a more durable way.

Motivating example

Loosely inspired by https://github.com/politics-rewired/graphile-worker-rate-limiter, I have written a reusable configuration factory that allows a task C to be executed only once a task A and B have been executed. It behaves like so:

const compositeTask = new CompositeTask({
  name: 'C',
  dependencies: {
    A : (payload) => payload.id,
    B : (payload) => payload.id
  }
});
// install composite task somehow, omitting for brievety
workerUtils.addJob('A', {foo: 'bar', id: 1}) // Nothing happens
workerUtils.addJob('B', {baz: 'qux', id: 2}) // Nothing happens
workerUtils.addJob('B', {hello: 'world', id: 1}) // Task C is ran with `{A: {foo: 'bar', id: 1}, B: {hello: 'world', id: 1}}`

My implementation works flawlessly. However, to work, each dependency task (A & B) need to alter the payload and retrieve the flags, and I can't find a way to do it than the brittle way of querying directly the DB but this requires knowing the graphile schema during initial configuration. I could get away without mutating the payload but the flags is the core behavior to allow a task C to be ran.

I am aware that reading from the jobs table is discouraged but the total impossibility to do so even in the cases where it is needed pushes users towards hacking around.

Breaking changes

Supporting development

I [tick all that apply]:

• am interested in building this feature myself
• am interested in collaborating on building this feature
• am willing to help testing this feature before it's released
• am willing to write a test-driven test suite for this feature (before it exists)
• am a Graphile sponsor ❤️
• have an active support or consultancy contract with Graphile

[benjie]

I would strongly advise that you do this negotiation in a different table, one you control, and then file the final jobs (the ones to execute) only when they are ready to execute. Worker strongly discourages reading from jobs because a) it reduces performance by introducing new locks on the table, b) by the time you’ve read the row and decided how to manipulate it, it may not exist any more since worker may have executed it, and c) if the job isn’t ready to execute then sitting in the jobs table bloats that table and impacts performance.

Handling these concerns outside of worker lets you build the most efficient schema to achieve your goal, lets you evolve it as you see fit, and keeps these concerns outside of worker’s main responsibilities making it easier to iterate and optimize.

[GerkinDev]

Let's say I did it the way you suggest (and I'm willing to do so), but as a library author, I want to distribute this package. Have I to also force the user to explicitly specify a schema in configuration and potentially pollute the user namespace for something that should behave like a graphile addition ? Getting access to the schema name is still relevant to keep what is tied to graphile within its schema

[benjie]

escapedWorkerSchema is just the worker schema setting the user passes then escaped (`"${schema.replace(/"/g, '""')}"`); where exactly are you trying to access it from?

[GerkinDev]

Well following your suggestion I would ideally like to have a hook I could attach on to wait for Graphile to migrate itself, then add my own tables in that schema (assuming that I have access to it), before actually starting the Graphile runner. Requiring to pass the schema both to graphile and my custom task is redundant. Think of it as a plugin initialization process.

For now, I don't use custom tables so I try to use it from within the tasks, but I am entirely willing to get away from it.

In a broader sense, I feel like it is hard to extend graphile and plug new behaviors on. Even looking at the sample given in the docs for rate limiting, it uses an external redis database, while pg could perfectly do the job.

In the end, I'm building a set of objects to build DAGs tasks on top of graphile, to manage very expensive and long processes. Given that I will have multiple of such DAG tasks, I need to somehow have reusable components, and eventually isolate them between DAG tasks as I would for graphile I would install in multiple schemas/databases.

[benjie]

I feel like it is hard to extend graphile and plug new behaviors on.

Have you tried the plugin system?

import { run } from "graphile-worker";

const MyMigrationPlugin: GraphileConfig.Plugin = {
  name: "MyMigrationPlugin",
  worker: {
    middleware: {
      async migrate(next, event) {
        const result = await next();

        console.log(event.ctx.escapedWorkerSchema);
        // Do your stuff here.

        return result;
      },
    },
  },
};
const preset: GraphileConfig.Preset = { plugins: [MyMigrationPlugin] };
const worker = await run({ preset, taskList: { /* ... */ });

(Note: this example requires graphile-worker@canary; for the currently released version you can use plugin.worker.hooks instead, but that system is deprecated because middleware is more powerful.)

[benjie]

You can read more about the Graphile Config plugin/preset system (in the process of being integrated into all of our projects) here: https://star.graphile.org/graphile-config/ - it was extracted from the plugin system in PostGraphile V5.

See also:

• https://worker.graphile.org/docs/config#preset
• https://worker.graphile.org/news/2023-12-11-016-release#graphile-config-and-the-new-plugin-system

[GerkinDev]

This looks very promising, I'll take a look at it. Despite having read the documentation multiple times I did not see that there was a plugin system. It may be a mistake on my side, or it could be made more visible in the docs (even mentioning this is experimental). Anyway I'll try this out and come back to you :)

[benjie]

Contributions to the docs appreciated 👍