Using Redis incrBy and getDel in job data

[alko89]

Hi, I have a somewhat specific question and I'm sorry I created an issue for this, but I couldn't found any chat or forum to be able to do so.

I want to use Bull to rate limit creation of usage records on Stripe (100/s). I'm receiving these usage records and sending the sum of them within the API limits.
For this I'm using Redis INCRBY and creating a job, I'm using job ID so that the job is only created once and removed after being processed:

  private async createUsageQueueJob(
    quantity: number,
    usage: UsageCreateJobDto,
  ) {
    this.logger.debug(
      `Added ${quantity} usage for ${usage.addon} to org ${usage.orgId}`,
    );
    await this.redisClient.incrBy(usage.key, quantity);
    await this.usageQueue.add(BullQueueJobs.USAGE_CREATE, usage, {
      jobId: usage.key,
      removeOnComplete: true,
    });
  }

When processing the queue I'm using GETDEL to retrieve the usage data and send it to Stripe

  @Process(BullQueueJobs.USAGE_CREATE)
  async handleUsageCreate(job: Job<UsageCreateJobDto>) {
    const quantity = Number(await this.redisClient.getDel(job.data.key));
    if (quantity) {
      await this.createUsage(quantity, job.data);
      await this.checkUsage(job.data);
    }
  }

What I'm wondering is, if its possible to use INCRBY and GETDEL for job data instead of relying on additional keys to be stored elsewhere?

I'm encountering cases when a usage arrives while the job is being processed, which causes the newly arrived usage to be processed the next time a new usage request arrives. The reason being a new job will not be created (because of the same id) while the old one is still being processed.

Being able to increase usage quantity in job data would potentially solve this issue, I'm also open to other suggestions.

[manast]

You are aware that Bull supports Rate limit right? https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queue

[alko89]

Yes I am, I have setup the queue to rate limit:

    limiter: {
        max: 100,
        duration: 1000,
    },

[manast]

I am having trouble trying to comprehend your question, could you rephrase it please?

[alko89]

Basically what I'm wondering is if its possible to use Redis commands (like INCRBY and GETDEL) to manipulate job data, instead of having to store the job and quantity for the usage as two separate keys in Redis?