Pagination for findItems & findInstances Api calls

[kvdinker]

Hello :)
We are currently working with your great workflow server tool to handle some business processes. Due to some long running instances we get more and more running processes which we want to display in a suitable way. For this we use the bpmn-client to retrieve all active and completed data and display it via a Next UI. Due to the amount of data, the query takes more and more time.
A great approach here would be to determine the amount of data ourselves using server-side pagination. This would allow you to limit the query time and process the data step by step.

Have you ever thought about such a feature?
It would be a huge help for us and probably many others. I look forward to hearing your thoughts on server-side pagination.

Thanks a lot

[ralphie2025]

Hello @kvdinker ,
Thanks for your feedback, I agree about your request, but like to get more details.
We can definitely add the feature of server-side pagination, but you still have to call the server to get the next page.
How do you want to sort the instances?
Once I get your feedback, a new release should be coming soon.

[kvdinker]

@ralphie2025 - thanks for your quick reply and great that you also like the proposal.
I know that every page has to be requested from the server again. However, if we can determine the page size ourselves, we are technically much more flexible.
Good question about sorting, I hadn't thought about that in advance. Currently, we mainly use the values from the data object and also display a few standard values such as name, creation date or due date (in case of tasks). For this reason, these fields would be important for sorting. However, I'm not sure if you can make it possible to sort by the custom fields from the data object. Otherwise, I would suggest using the creation date by default. In addition to the field, it would be great if we could specify the direction (asc, desc).

I hope the additional details help. If you have any further questions, I will try to answer them as soon as possible.

Thanks for your effort :)

[ralphhanna]

Now available in Release 2.3.2
https://bpmnserver.github.io/docs/data_find/

[kvdinker]

Thank you :) It looks like a huge benefit for us.
I started to add the new feature for us, but unfortunately the new bpmn-client version (2.3.2) is not available with npm install.
@ralphhanna could you check or publish the version too ?

[ralphhanna]

Sorry, my bad, all done.

[kvdinker]

Hey @ralphhanna,
Do you know how I can get all items in one instance with the new find call (like InstaceData Object)? Currently, the items object contains only one item and due to that I got multiple data instances with the same IDs. So the pagination is a bit difficult for instances only, because if I set items: 0 in the propagation, the result looks like this: {data: Array(0), nextCursor: null, totalCount: 0}
Thanks :)

[ralphhanna]

what is the condition? and the sort?
Are you trying to display all items for Instance, even though they don't meet the condition?

[kvdinker]

filter: {status: 'running'}, sort: { _id: 1 }, projection: {name: 1, id: 1, status: 1, data: 1 // also items: 0 has the same result}

no, currently I´m starting with the basic filter, sort, projection stuff and try to get a similar object like the InstanceData, only with a smaller result set :D

[ralphhanna]

I see the problem, since we flattened the results, you can't have projection of 'items': 0 ,it will not return any results (may fix this later)

Current Release

Try this as a workaround:

filter:			{ name: 'Buy Used Car',"items.seq":1 },
sort:			{"name":1},
projection:	{ id: 1, data: 1, name: 1, _id:1,startedAt:1,status:1, "items.dueDate":1, "items.type":1,"items.name":1,"items.status":1	items.seq":1}

The trick here is to restrict only one item (seq==1)

New Release 2.3.4 you can do this:

filter:		{ name: 'Buy Used Car'},
sort:			{"name":1},
projection:	{ id: 1, data: 1, name: 1, _id:1,startedAt:1,status:1}

Will return only instances as long as you don't reference "items" in your filter, sort or projection

[kvdinker]

Hey @ralphhanna,
Thanks for the update :)
I had some time to update our code as well. Pagination with instances works great. Also, I took a deeper look into filtering and found a good way to add a “contains” filter due to aggregation. Here I could add a regex expression like below:

filter:		{ name: { $regex: 'Used' }},
projection:	{ id: 1, data: 1, name: 1, _id:1,startedAt:1,status:1}

In combination with a custom sort ( data.customState: 1 or name: -1), I noticed that I get strange values from page 2 onwards. Several entries are repeated, others are no longer displayed and sometimes the number of entries does not match the limit. I have seen that the nextCursor looks different

• with sorting (Verfahren|679b2ca8e09cf450b4be4c9a or [object Object]|67d1674eb74edf56bc412d7d)
• without (67f392fc67356225a8b5615c)

This changes the aggregation internally (as far as I can see).
Can you explain what is going on there and know how to fix it?

Example
FindParameters:

after: "Verfahren|679b2ca8e09cf450b4be4c9a"
filter: {name: 'Verfahren', status: 'end'}
getTotalCount: true
limit: 5
sort: {name: -1}
projection: { id: 1, data: 1, name: 1, _id:1,startedAt:1,status:1}

Converts to:

  [
    {
      "$match": {
        "name": "Verfahren",
        "status": "end",
        "$and": [
          {
            "$or": [
              {
                "name": {
                  "$lt": "Verfahren"
                },
                "_id": {
                  "$ne": "679b2ca8e09cf450b4be4c9a"
                }
              },
              {
                "name": "Verfahren",
                "_id": {
                  "$lt": "679b2ca8e09cf450b4be4c9a"
                }
              }
            ]
          }
        ]
      }
    },
    {
      "$project": {
        "name": 1,
        "id": 1,
        "status": 1,
        "data": 1,
        "startedAt": 1
      }
    },
    {
      "$sort": {
        "name": -1
      }
    },
    {
      "$limit": 5
    }
 ]

Results in (Page 2 Entries 6-10):

{
  "data": [
      {
          "_id": "679b2c3ce09cf450b4be4c96",
          "id": "0118ed9e-7c15-46b7-bb08-0ad694beb85d",
          "name": "Verfahren",
          "status": "end",
          "data": {},
          "startedAt": "2025-01-30T07:37:30.996Z"
      },
      {
          "_id": "679b2c95e09cf450b4be4c98",
          "id": "4b27677e-2106-4e80-b953-c3366ecfe79b",
          "name": "Verfahren",
          "status": "end",
          "data": {},
          "startedAt": "2025-01-30T07:38:59.849Z"
      }
  ],
  "nextCursor": "Verfahren|679b2c95e09cf450b4be4c98",
  "totalCount": 26
}

[bpmn-ts]

Pagination requires sorting on a unique field, so we need to add '_id' to the sort field nextCursor should be; - with sorting (Verfahren|679b2ca8e09cf450b4be4c9a or [object Object]|67d1674eb74edf56bc412d7d) - without (67f392fc67356225a8b5615c) But `[object Object]|67d1674eb74edf56bc412d7d)` is a problem.. possible invalid sort field... I can't see what the problem in your scenario, I suggest to examine the problem further that you run it first with very large limit, say 500, and see if they are sorted OK,

…

On Wed, Apr 23, 2025 at 8:35 AM kvdinker ***@***.***> wrote: Hey @ralphhanna <https://github.com/ralphhanna>, Thanks for the update :) I had some time to update our code as well. Pagination with instances works great. Also, I took a deeper look into filtering and found a good way to add a “contains” filter due to aggregation. Here I could add a regex expression like below: filter: { name: { $regex: 'Used' }}, projection: { id: 1, data: 1, name: 1, _id:1,startedAt:1,status:1} In combination with a custom sort ( data.customState: 1 or name: -1), I noticed that I get strange values from page 2 onwards. Several entries are repeated, others are no longer displayed and sometimes the number of entries does not match the limit. I have seen that the nextCursor looks different - with sorting (Verfahren|679b2ca8e09cf450b4be4c9a or [object Object]|67d1674eb74edf56bc412d7d) - without (67f392fc67356225a8b5615c) This changes the aggregation internally (as far as I can see). Can you explain what is going on there and know how to fix it? *Example* FindParameters: after: "Verfahren|679b2ca8e09cf450b4be4c9a" filter: {name: 'Verfahren', status: 'end'} getTotalCount: true limit: 5 sort: {name: -1} projection: { id: 1, data: 1, name: 1, _id:1,startedAt:1,status:1} Converts to: [ { "$match": { "name": "Verfahren", "status": "end", "$and": [ { "$or": [ { "name": { "$lt": "Verfahren" }, "_id": { "$ne": "679b2ca8e09cf450b4be4c9a" } }, { "name": "Verfahren", "_id": { "$lt": "679b2ca8e09cf450b4be4c9a" } } ] } ] } }, { "$project": { "name": 1, "id": 1, "status": 1, "data": 1, "startedAt": 1 } }, { "$sort": { "name": -1 } }, { "$limit": 5 } ] Results in (Page 2 Entries 6-10): { "data": [ { "_id": "679b2c3ce09cf450b4be4c96", "id": "0118ed9e-7c15-46b7-bb08-0ad694beb85d", "name": "Verfahren", "status": "end", "data": {}, "startedAt": "2025-01-30T07:37:30.996Z" }, { "_id": "679b2c95e09cf450b4be4c98", "id": "4b27677e-2106-4e80-b953-c3366ecfe79b", "name": "Verfahren", "status": "end", "data": {}, "startedAt": "2025-01-30T07:38:59.849Z" } ], "nextCursor": "Verfahren|679b2c95e09cf450b4be4c98", "totalCount": 26 } — Reply to this email directly, view it on GitHub <#247 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQL5NQ5F4IOUTMRCCFZVUI3226CEFAVCNFSM6AAAAABZPJF72KVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOJSGI2TSMA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>