Dora processing is too slow to complete in a few hours

[AnkeshThakur]

We have been using devlake for more than 6 months and now it is hitting us in performance. Earlier it used to take 2-3 hours to complete.

Over the course of time, I am observing that this is taking extremely long time (~22 hours) to process DORA metrics. Is there any flag or configuration which can set it run tasks in parallel or in bulk?

[GIN] 2025/07/22 - 14:00:32 | 200 |    3.305575ms |   10.220.48.179 | GET      "/pipelines/128/tasks"
time="2025-07-22 14:00:37" level=info msg=" [pipeline service] [pipeline #128] [task #2926] [calculateChangeLeadTime] finished records: 5151(not exactly)"
[GIN] 2025/07/22 - 14:00:37 | 200 |         650ns |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:00:37 | 200 |        2.71µs |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:00:37 | 200 |    4.331136ms |   10.220.48.179 | GET      "/pipelines/128"
[GIN] 2025/07/22 - 14:00:37 | 200 |     3.71054ms |   10.220.48.179 | GET      "/pipelines/128/tasks"
[GIN] 2025/07/22 - 14:00:42 | 200 |         2.5µs |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:00:42 | 200 |        2.45µs |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:00:42 | 200 |    9.670353ms |   10.220.48.179 | GET      "/pipelines/128"
[GIN] 2025/07/22 - 14:00:42 | 200 |    3.103823ms |   10.220.48.179 | GET      "/pipelines/128/tasks"
[GIN] 2025/07/22 - 14:00:47 | 200 |        3.36µs |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:00:47 | 200 |        2.88µs |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:00:47 | 200 |    5.273676ms |   10.220.48.179 | GET      "/pipelines/128"
[GIN] 2025/07/22 - 14:00:47 | 200 |    3.411076ms |   10.220.48.179 | GET      "/pipelines/128/tasks"
[GIN] 2025/07/22 - 14:00:52 | 200 |         510ns |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:00:52 | 200 |        2.82µs |   10.220.45.224 | GET      "/ping"
time="2025-07-22 14:00:52" level=info msg=" [pipeline service] [pipeline #128] [task #2926] [calculateChangeLeadTime] finished records: 5152(not exactly)"
[GIN] 2025/07/22 - 14:00:57 | 200 |        2.43µs |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:00:57 | 200 |        1.92µs |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:01:02 | 200 |        2.51µs |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:01:02 | 200 |         880ns |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:01:07 | 200 |        2.64µs |   10.220.45.224 | GET      "/ping"
[GIN] 2025/07/22 - 14:01:07 | 200 |        2.64µs |   10.220.45.224 | GET      "/ping"
time="2025-07-22 14:01:08" level=info msg=" [pipeline service] [pipeline #128] [task #2926] [calculateChangeLeadTime] finished records: 5153(not exactly)"

Single record takes around 15 seconds which too slow.

[dosubot[bot]]

There isn't a direct flag or config to enable parallel or bulk DORA metrics processing in DevLake. The main architectural bottleneck is that tasks are processed sequentially by a single runner, so true parallelism for DORA subtasks isn't natively supported. However, you can try a few things to improve performance:

• Increase the global worker concurrency by setting the PIPELINE_MAX_PARALLEL environment variable, which allows more pipelines to run in parallel (though this won't parallelize subtasks within a single pipeline)
• Enable slow query logging in your database and add indexes to the tables used in DORA calculations—users have reported dramatic speedups (from hours to minutes) after optimizing indexes and queries for DORA tables like pull_requests_* and related foreign keys (see discussion)
• Make sure your hardware (CPU, memory) and database resources are sufficient; increasing memory has resolved similar issues for others (see example)
• Disable external monitoring tools like Dynatrace if you use them, as they've been shown to cause severe slowdowns (see example)

If you want help finding specific batch size or concurrency settings in the code, let me know!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}