Skip to content

Snapshot creation parallelization #2818

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Apr 16, 2025
Merged

Conversation

andrejtonev
Copy link
Contributor

@andrejtonev andrejtonev commented Mar 21, 2025

New flags:
--storage-parallel-snapshot-creation <- switch between single and multi threaded creation
--storage-snapshot-thread-count <- define number of threads to use (default to all)

@andrejtonev andrejtonev added feature feature CI -build=coverage -test=core Run coverage build and core tests on push CI -build=debug -test=core Run debug build and core tests on push CI -build=release -test=e2e Run release build and e2e tests on push labels Mar 21, 2025
@andrejtonev andrejtonev added this to the mg-v3.2.0 milestone Mar 21, 2025
@andrejtonev andrejtonev self-assigned this Mar 21, 2025
@andrejtonev
Copy link
Contributor Author

andrejtonev commented Mar 21, 2025

Git commit description, explain the changes you made here


Leave above in PR description, copy the below into a comment


Tracking

  • [Link to Epic/Issue]

Standard development

CI Testing Labels

  • Select the appropriate CI test labels (CI -build=build-name -test=test-suite)

Documentation checklist

  • Add the documentation label
  • Add the bug / feature label
  • Add the milestone for which this feature is intended
    • If not known, set for a later milestone
  • Write a release note, including added/changed clauses
    • Snapshot creation has been parallelized. Edges and vertices are batches (as defined via flag), these batches are processed in parallel on number of threads. Users can now create snapshots in less time, freeing up resources for other tasks. 2818
  • Create snapshot parallelization documentation#1243
    • Is back linked to this development PR
  • @katarinasupe

@andrejtonev andrejtonev force-pushed the prallel_snapshot_create branch from 9ac798c to d56f5b1 Compare April 9, 2025 10:42
@andrejtonev andrejtonev requested a review from as51340 April 9, 2025 12:23
@Ignition Ignition self-requested a review April 10, 2025 15:24
@andrejtonev andrejtonev force-pushed the prallel_snapshot_create branch from d56f5b1 to 1e54139 Compare April 10, 2025 15:41
@as51340
Copy link
Contributor

as51340 commented Apr 11, 2025

also, I read that std::hardware_concurrency can return 0 on some machines and that it makes sense to have "back-up default". This is example from the c++ concurrency in action:

auto max_threads = user flag thread count
auto hardware_threads = std::thread::hardware_concurrency();
auto num_threads = std::min(hardware_threads != 0 ? hardware_threads : 2, max_threads);

Maybe would be good that we change this at all places in the code and provide a utility function for obtaining the number of threads since std::thread::hardware_concurrency is only a hint https://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency

@Ignition Ignition dismissed their stale review April 14, 2025 10:00

Was explained that the approach was different than I initially thought. All corner cases I could think up were resolved.

@as51340 as51340 self-requested a review April 14, 2025 13:05
Copy link
Contributor

@as51340 as51340 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andrejtonev andrejtonev added Docs needed Docs needed and removed CI -build=coverage -test=core Run coverage build and core tests on push CI -build=debug -test=core Run debug build and core tests on push CI -build=release -test=e2e Run release build and e2e tests on push labels Apr 15, 2025
@andrejtonev andrejtonev enabled auto-merge April 15, 2025 10:58
@andrejtonev andrejtonev added this pull request to the merge queue Apr 15, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 15, 2025
@andrejtonev andrejtonev enabled auto-merge April 15, 2025 18:07
@andrejtonev andrejtonev added this pull request to the merge queue Apr 15, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 15, 2025
@andrejtonev andrejtonev enabled auto-merge April 16, 2025 08:54
@andrejtonev andrejtonev added this pull request to the merge queue Apr 16, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 16, 2025
@andrejtonev andrejtonev enabled auto-merge April 16, 2025 10:56
@andrejtonev andrejtonev added this pull request to the merge queue Apr 16, 2025
Copy link

Merged via the queue into master with commit 08a342a Apr 16, 2025
17 checks passed
@andrejtonev andrejtonev deleted the prallel_snapshot_create branch April 16, 2025 12:31
@gitbuda gitbuda mentioned this pull request Apr 16, 2025
53 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Docs needed Docs needed feature feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants