Priority inversion with CKSyncEngine #1781
Replies: 4 comments 1 reply
-
Hello @Jason-Abbott,
The quality of service of GRDB connections is controlled by By default GRDB connections execute with the
That would be my understanding as well. CKSyncEngine calls your app, which wants to perform a read and thus to acquire an available connection from That's why I'd put a breakpoint and look at all threads and queues (mandatory
We probably need to figure out what's wrong, exactly, before modifying your code, or GRDB code. |
Beta Was this translation helpful? Give feedback.
-
Hi @groue. Thank you for that clarification and those ideas. I’ve added the runtime issue breakpoints. I’ll comment again when I’ve gathered more actionable details. |
Beta Was this translation helpful? Give feedback.
-
I captured the backtrace. There seem to be only a couple queries in play, to produce a CKRecord from a model within the Although these should be happening in the same I think I have enough information now to figure it out but in case something quickly catches your eye, here is that trace: bt all
thread #1, queue = 'com.apple.main-thread'
frame #0: 0x000000019b615570 AppKit`___nonOverridableViewHitTest_block_invoke + 80
frame #1: 0x0000000196e491d4 CoreFoundation`__NSARRAY_IS_CALLING_OUT_TO_A_BLOCK__ + 24
frame #2: 0x0000000196e490b0 CoreFoundation`-[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 436
frame #3: 0x000000019b6156ac AppKit`___nonOverridableViewHitTest_block_invoke + 396
frame #4: 0x0000000196e491d4 CoreFoundation`__NSARRAY_IS_CALLING_OUT_TO_A_BLOCK__ + 24
frame #5: 0x0000000196e490b0 CoreFoundation`-[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 436
frame #6: 0x000000019b6156ac AppKit`___nonOverridableViewHitTest_block_invoke + 396
frame #7: 0x0000000196e491d4 CoreFoundation`__NSARRAY_IS_CALLING_OUT_TO_A_BLOCK__ + 24
frame #8: 0x0000000196e490b0 CoreFoundation`-[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 436
frame #9: 0x000000019b6156ac AppKit`___nonOverridableViewHitTest_block_invoke + 396
frame #10: 0x0000000196e491d4 CoreFoundation`__NSARRAY_IS_CALLING_OUT_TO_A_BLOCK__ + 24
frame #11: 0x0000000196e490b0 CoreFoundation`-[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 436
frame #12: 0x000000019b6156ac AppKit`___nonOverridableViewHitTest_block_invoke + 396
frame #13: 0x0000000196e491d4 CoreFoundation`__NSARRAY_IS_CALLING_OUT_TO_A_BLOCK__ + 24
frame #14: 0x0000000196e490b0 CoreFoundation`-[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 436
frame #15: 0x000000019b6156ac AppKit`___nonOverridableViewHitTest_block_invoke + 396
frame #16: 0x0000000196e491d4 CoreFoundation`__NSARRAY_IS_CALLING_OUT_TO_A_BLOCK__ + 24
frame #17: 0x0000000196e490b0 CoreFoundation`-[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 436
frame #18: 0x000000019b6156ac AppKit`___nonOverridableViewHitTest_block_invoke + 396
frame #19: 0x0000000196e491d4 CoreFoundation`__NSARRAY_IS_CALLING_OUT_TO_A_BLOCK__ + 24
frame #20: 0x0000000196e490b0 CoreFoundation`-[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 436
frame #21: 0x000000019b6156ac AppKit`___nonOverridableViewHitTest_block_invoke + 396
frame #22: 0x0000000196e491d4 CoreFoundation`__NSARRAY_IS_CALLING_OUT_TO_A_BLOCK__ + 24
frame #23: 0x0000000196e490b0 CoreFoundation`-[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 436
frame #24: 0x000000019b6156ac AppKit`___nonOverridableViewHitTest_block_invoke + 396
frame #25: 0x0000000196e491d4 CoreFoundation`__NSARRAY_IS_CALLING_OUT_TO_A_BLOCK__ + 24
frame #26: 0x0000000196e490b0 CoreFoundation`-[__NSArrayM enumerateObjectsWithOptions:usingBlock:] + 436
frame #27: 0x000000019b60e3e8 AppKit`-[_NSTrackingAreaAKManager _updateActiveTrackingAreasForWindowLocation:modifierFlags:] + 624
frame #28: 0x000000019b610d0c AppKit`-[_NSTrackingAreaAKManager routeEnterExitEvent:] + 28
frame #29: 0x000000019aea1b70 AppKit`-[NSWindow(NSEventRouting) _reallySendEvent:isDelayedEvent:] + 152
frame #30: 0x000000019aea1910 AppKit`-[NSWindow(NSEventRouting) sendEvent:] + 288
frame #31: 0x000000019b6089f0 AppKit`+[_NSTrackingAreaAKManager routeEnterExitEvent:] + 80
frame #32: 0x000000019b5c8edc AppKit`+[_NSTrackingAreaManager routeEnterExitEvent:] + 244
frame #33: 0x000000019b718ec0 AppKit`-[NSApplication(NSEventRouting) sendEvent:] + 368
frame #34: 0x000000019b31842c AppKit`-[NSApplication _handleEvent:] + 60
frame #35: 0x000000019ad6ec8c AppKit`-[NSApplication run] + 520
frame #36: 0x000000019ad4535c AppKit`NSApplicationMain + 880
frame #37: 0x00000001c70a3878 SwiftUI`merged generic specialization of function signature specialization of SwiftUI.runApp(__C.NSResponder & __C.NSApplicationDelegate) -> Swift.Never + 160
frame #38: 0x00000001c7501658 SwiftUI`SwiftUI.runApp<τ_0_0 where τ_0_0: SwiftUI.App>(τ_0_0) -> Swift.Never + 108
frame #39: 0x00000001c78625c0 SwiftUI`static SwiftUI.App.main() -> () + 224
frame #40: 0x0000000103ed445c ElidedApp (debug).debug.dylib`static TriageApp.main() at ElidedApp.swift:89:82
frame #42: 0x0000000103ed4be0 ElidedApp (debug).debug.dylib`main at ElidedApp.swift:0
frame #43: 0x00000001969cab98 dyld`start + 6076
thread #9, name = 'com.apple.NSEventThread'
frame #0: 0x0000000196d28c34 libsystem_kernel.dylib`mach_msg2_trap + 8
frame #1: 0x0000000196d3b3a0 libsystem_kernel.dylib`mach_msg2_internal + 76
frame #2: 0x0000000196d31764 libsystem_kernel.dylib`mach_msg_overwrite + 484
frame #3: 0x0000000196d28fa8 libsystem_kernel.dylib`mach_msg + 24
frame #4: 0x0000000196e55e7c CoreFoundation`__CFRunLoopServiceMachPort + 160
frame #5: 0x0000000196e54798 CoreFoundation`__CFRunLoopRun + 1208
frame #6: 0x0000000196e53c58 CoreFoundation`CFRunLoopRunSpecific + 572
frame #7: 0x000000019ae9f7fc AppKit`_NSEventThread + 140
frame #8: 0x000000010107e558 libclang_rt.asan_osx_dynamic.dylib`asan_thread_start(void*) + 80
frame #9: 0x000000010099d08c libsystem_pthread.dylib`_pthread_start + 136
* thread #19, queue = 'RPAC issue generation workloop', stop reason = breakpoint 1.1
* frame #0: 0x0000000196ababb0 libsystem_trace.dylib`os_log_fault_default_callback
frame #1: 0x0000000196ac5674 libsystem_trace.dylib`_os_log_invoke_callback + 200
frame #2: 0x0000000196ab57b4 libsystem_trace.dylib`_os_log_impl_flatten_and_send + 7684
frame #3: 0x0000000196ab3998 libsystem_trace.dylib`_os_log + 168
frame #4: 0x0000000196ab9ff4 libsystem_trace.dylib`_os_log_fault_impl + 28
frame #5: 0x0000000101bac528 libRPAC.dylib`culledOsLogFault + 1352
frame #6: 0x0000000101bad85c libRPAC.dylib`__generateCulledBacktrace_block_invoke_2 + 4436
frame #7: 0x0000000100bf52dc libdispatch.dylib`_dispatch_client_callout + 16
frame #8: 0x0000000100bdcd30 libdispatch.dylib`_dispatch_block_invoke_direct + 312
frame #9: 0x0000000101080510 libclang_rt.asan_osx_dynamic.dylib`__wrap_dispatch_async_block_invoke + 196
frame #10: 0x0000000100bd8514 libdispatch.dylib`_dispatch_call_block_and_release + 32
frame #11: 0x0000000100bf52dc libdispatch.dylib`_dispatch_client_callout + 16
frame #12: 0x0000000100be3e10 libdispatch.dylib`_dispatch_workloop_invoke + 2840
frame #13: 0x0000000100beffbc libdispatch.dylib`_dispatch_root_queue_drain_deferred_wlh + 664
frame #14: 0x0000000100bef414 libdispatch.dylib`_dispatch_workloop_worker_thread + 580
frame #15: 0x000000010099f7a4 libsystem_pthread.dylib`_pthread_wqthread + 292
thread #21, queue = 'actual.reader.1'
frame #0: 0x00000001a88b19d8 libswiftCore.dylib`std::__1::enable_if const* const>::value, __swift::__runtime::llvm::hash_code>::type __swift::__runtime::llvm::hashing::detail::hash_combine_range_impl const* const>(swift::TargetMetadata const* const*, swift::TargetMetadata const* const*) + 36
frame #1: 0x00000001a88b179c libswiftCore.dylib`std::__1::pair*, unsigned int> swift::ConcurrentReadableHashMap, swift::Mutex>::find<(anonymous namespace)::TupleCacheEntry::Key>((anonymous namespace)::TupleCacheEntry::Key const&, swift::ConcurrentReadableHashMap, swift::Mutex>::IndexStorage, unsigned long, swift::HashMapElementWrapper<(anonymous namespace)::TupleCacheEntry>*) + 72
frame #2: 0x00000001a8897500 libswiftCore.dylib`std::__1::pair<(anonymous namespace)::TupleCacheEntry*, swift::MetadataResponse> swift::LockingConcurrentMap<(anonymous namespace)::TupleCacheEntry, (anonymous namespace)::TupleCacheStorage>::getOrInsert<(anonymous namespace)::TupleCacheEntry::Key, swift::MetadataRequest&, swift::TargetValueWitnessTable const*&>((anonymous namespace)::TupleCacheEntry::Key, swift::MetadataRequest&, swift::TargetValueWitnessTable const*&) + 336
frame #3: 0x00000001a88971ac libswiftCore.dylib`swift_getTupleTypeMetadata + 96
frame #4: 0x00000001a8898334 libswiftCore.dylib`swift_getTupleTypeMetadata2 + 32
frame #5: 0x0000000105727740 ElidedKit`static CKRecord.fetch(id=(single = 09A65E5A-B025-482F-BF00-685FA47FC30C, compound = (Foundation.UUID, ElidedKit.Nothing) @ 0x000000016f89ccc0), storage=ElidedKit.TableAndKeys @ 0x000000012e567c20, db=0x000061500000f300) at CKRecord+SQL.swift:107:63
frame #6: 0x000000010573d530 ElidedKit`static CloudSyncable.fetchAsCloudRecord(id=(single = 09A65E5A-B025-482F-BF00-685FA47FC30C, compound = (Foundation.UUID, ElidedKit.Nothing) @ 0x000000016f89ccc0), db=0x000061500000f300) at CloudSyncable.swift:120:39
frame #8: 0x000000010573c9dc ElidedKit`static CloudSyncable.fetchAsCloudRecord(unparsedID=1 value, db=0x000061500000f300) at CloudSyncable.swift:92:13
frame #9: 0x000000010576dd20 ElidedKit`closure #1 in closure #2 in CloudSyncDelegate.nextRecordZoneChangeBatch($0=0x000061500000f300, model=ElidedKit.Node, unparsedID=1 value) at CloudSyncDelegate+outgoing.swift:39:35
frame #11: 0x00000001075f1acc GRDB`closure #1 in closure #1 in closure #1 in closure #1 in DatabasePool.read(db=0x000061500000f300, value=0x0000000105781fc4 ElidedKit`partial apply forwarder for closure #1 @Sendable (GRDB.Database) throws -> Swift.Optional<__C.CKRecord> in closure #2 @Sendable (__C.CKRecordID) async -> Swift.Optional<__C.CKRecord> in ElidedKit.CloudSyncDelegate.nextRecordZoneChangeBatch(_: CloudKit.CKSyncEngine.SendChangesContext, syncEngine: CloudKit.CKSyncEngine) async -> Swift.Optional at ) at DatabasePool.swift:378:44
frame #13: 0x00000001075e0238 GRDB`closure #2 in Mutex.inDatabase(work=0x0000000107603efc GRDB`partial apply forwarder for closure #1 () throws -> A in closure #1 @Sendable (GRDB.Database) -> () in closure #1 @Sendable (Swift.Result<(element: GRDB.SerializedDatabase, release: @Sendable (GRDB.PoolCompletion) -> ()), Swift.Error>) -> () in closure #1 (Swift.UnsafeContinuation) -> () in GRDB.DatabasePool.read(@Sendable (GRDB.Database) throws -> A) async throws -> A at ) at SerializedDatabase.swift:396:17
frame #15: 0x00000001078dd560 GRDB`throwingFirstError(execute=0x00000001075e0324 GRDB`partial apply forwarder for closure #2 () throws -> A1 in GRDB.Mutex.inDatabase(_: GRDB.Database, execute: () throws -> A1) throws -> A1 at , finally=0x00000001075e0638 GRDB`partial apply forwarder for closure #3 () throws -> () in GRDB.Mutex.inDatabase(_: GRDB.Database, execute: () throws -> A1) throws -> A1 at ) at Utils.swift:107:22
frame #16: 0x00000001075dbf98 GRDB`Mutex.inDatabase(db=0x000061500000f300, work=0x0000000107603efc GRDB`partial apply forwarder for closure #1 () throws -> A in closure #1 @Sendable (GRDB.Database) -> () in closure #1 @Sendable (Swift.Result<(element: GRDB.SerializedDatabase, release: @Sendable (GRDB.PoolCompletion) -> ()), Swift.Error>) -> () in closure #1 (Swift.UnsafeContinuation) -> () in GRDB.DatabasePool.read(@Sendable (GRDB.Database) throws -> A) async throws -> A at ) at SerializedDatabase.swift:395:20
frame #17: 0x00000001075f14c0 GRDB`closure #1 in closure #1 in closure #1 in DatabasePool.read(db=0x000061500000f300, releaseReader=0x0000000107603e6c GRDB`partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed @Sendable (@in_guaranteed GRDB.PoolCompletion) -> (@out ()) to @escaping @callee_guaranteed @Sendable (@unowned GRDB.PoolCompletion) -> () with unmangled suffix ".81" at , dbAccess=0x000060300058e4c0, value=0x0000000105781fc4 ElidedKit`partial apply forwarder for closure #1 @Sendable (GRDB.Database) throws -> Swift.Optional<__C.CKRecord> in closure #2 @Sendable (__C.CKRecordID) async -> Swift.Optional<__C.CKRecord> in ElidedKit.CloudSyncDelegate.nextRecordZoneChangeBatch(_: CloudKit.CKSyncEngine.SendChangesContext, syncEngine: CloudKit.CKSyncEngine) async -> Swift.Optional at , continuation=Swift.UnsafeContinuation, Swift.Error> @ 0x000000012e4ff0c0) at DatabasePool.swift:374:55
frame #18: 0x00000001075d90f4 GRDB`closure #1 in SerializedDatabase.async(block=0x0000000107603ed0 GRDB`partial apply forwarder for closure #1 @Sendable (GRDB.Database) -> () in closure #1 @Sendable (Swift.Result<(element: GRDB.SerializedDatabase, release: @Sendable (GRDB.PoolCompletion) -> ()), Swift.Error>) -> () in closure #1 (Swift.UnsafeContinuation) -> () in GRDB.DatabasePool.read(@Sendable (GRDB.Database) throws -> A) async throws -> A at ) at SerializedDatabase.swift:228:13
frame #20: 0x0000000101080510 libclang_rt.asan_osx_dynamic.dylib`__wrap_dispatch_async_block_invoke + 196
frame #21: 0x0000000100bd8514 libdispatch.dylib`_dispatch_call_block_and_release + 32
frame #22: 0x0000000100bf52dc libdispatch.dylib`_dispatch_client_callout + 16
frame #23: 0x0000000100be1594 libdispatch.dylib`_dispatch_lane_serial_drain + 828
frame #24: 0x0000000100be2438 libdispatch.dylib`_dispatch_lane_invoke + 440
frame #25: 0x0000000100beffbc libdispatch.dylib`_dispatch_root_queue_drain_deferred_wlh + 664
frame #26: 0x0000000100bef414 libdispatch.dylib`_dispatch_workloop_worker_thread + 580
frame #27: 0x000000010099f7a4 libsystem_pthread.dylib`_pthread_wqthread + 292
thread #22, queue = 'com.apple.cloudkit.operation-50B732E4BB736853.callback'
frame #0: 0x0000000196d2aa8c libsystem_kernel.dylib`__ulock_wake + 8
frame #1: 0x0000000196da2420 libsystem_platform.dylib`_os_unfair_lock_unlock_slow + 56
frame #2: 0x000000019697a0cc libobjc.A.dylib`lookUpImpOrForward + 612
frame #3: 0x0000000196979b84 libobjc.A.dylib`_objc_msgSend_uncached + 68
frame #4: 0x00000001a092b4f4 CloudKit`___lldb_unnamed_symbol13651 + 204
frame #5: 0x00000001a07dd148 CloudKit`___lldb_unnamed_symbol5181 + 712
frame #6: 0x00000001a090c4f0 CloudKit`___lldb_unnamed_symbol13048 + 876
frame #7: 0x00000001a07dbcfc CloudKit`___lldb_unnamed_symbol5160 + 612
frame #8: 0x00000001a09ca350 CloudKit`___lldb_unnamed_symbol16879 + 180
frame #9: 0x0000000196e364d4 CoreFoundation`__invoking___ + 148
frame #10: 0x0000000196e36364 CoreFoundation`-[NSInvocation invoke] + 424
frame #11: 0x0000000196e6a678 CoreFoundation`-[NSInvocation invokeWithTarget:] + 64
frame #12: 0x00000001a092bd40 CloudKit`___lldb_unnamed_symbol13654 + 276
frame #13: 0x0000000100bf52dc libdispatch.dylib`_dispatch_client_callout + 16
frame #14: 0x0000000100bdcd30 libdispatch.dylib`_dispatch_block_invoke_direct + 312
frame #15: 0x00000001a094ea28 CloudKit`ck_call_or_dispatch_async_if_not_key + 84
frame #16: 0x00000001a092b950 CloudKit`___lldb_unnamed_symbol13652 + 964
frame #17: 0x0000000196e34bbc CoreFoundation`___forwarding___ + 956
frame #18: 0x0000000196e34740 CoreFoundation`_CF_forwarding_prep_0 + 96
frame #19: 0x000000019842bd70 Foundation`__NSXPCCONNECTION_IS_CALLING_OUT_TO_EXPORTED_OBJECT_S2__ + 16
frame #20: 0x0000000198d9b698 Foundation`-[NSXPCConnection _decodeAndInvokeMessageWithEvent:reply:flags:] + 1664
frame #21: 0x0000000198d9ccd8 Foundation`message_handler_message + 88
frame #22: 0x00000001983e8a94 Foundation`message_handler + 152
frame #23: 0x0000000196a72830 libxpc.dylib`_xpc_connection_call_event_handler + 112
frame #24: 0x0000000196a7112c libxpc.dylib`_xpc_connection_mach_event + 1196
frame #25: 0x0000000100bf5324 libdispatch.dylib`_dispatch_client_callout4 + 16
frame #26: 0x0000000100bf7fec libdispatch.dylib`_dispatch_mach_msg_invoke + 608
frame #27: 0x0000000100be13a8 libdispatch.dylib`_dispatch_lane_serial_drain + 336
frame #28: 0x0000000100bf9738 libdispatch.dylib`_dispatch_mach_invoke + 516
frame #29: 0x0000000100be13a8 libdispatch.dylib`_dispatch_lane_serial_drain + 336
frame #30: 0x0000000100be246c libdispatch.dylib`_dispatch_lane_invoke + 492
frame #31: 0x0000000100beffbc libdispatch.dylib`_dispatch_root_queue_drain_deferred_wlh + 664
frame #32: 0x0000000100bef414 libdispatch.dylib`_dispatch_workloop_worker_thread + 580
frame #33: 0x000000010099f7a4 libsystem_pthread.dylib`_pthread_wqthread + 292
thread #23, stop reason = thread-creation
frame #0: 0x00000001009a6f40 libsystem_pthread.dylib`start_wqthread
thread #24, stop reason = thread-creation
frame #0: 0x00000001009a6f40 libsystem_pthread.dylib`start_wqthread
thread #25
frame #0: 0x000000010099f684 libsystem_pthread.dylib`_pthread_wqthread + 4
|
Beta Was this translation helpful? Give feedback.
-
This code path isn’t overly complex so old-school debugging won’t be too hard: comment things out, see what changes. Just in case it matters, this is all kicked off by a Then in Those operations are all complete by the time I’ll comment again when I’ve solved it or have something clearer to act on. Thank you, as always! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Priority inversion has been well discussed here:
I am hoping for a quick sanity check before applying the advice there. The background is:
The thread checker is flagging an inversion at GRDB Pool.get() when
CKSyncEngine.Event.sentRecordZoneChanges
fires. If I logTask.currentPriority
I see these events are basically allmedium
.The
CKSyncEngine
delegate methods are synchronous so my GRDB calls within the event handlers are also synchronous. If I understand the previous discussion correctly, that is likely the source of the inversion.Some of these events should be a lower priority but if I want to resolve the inversion, will I need to do something like wrapping the GRDB calls in
Task {}
so I can use the async methods?Beta Was this translation helpful? Give feedback.
All reactions