ReplayKit and audio DispatchQueue choke

[levs42]

There's an issue where mixer.audioIO.lockQueue keeps accumulating buffers until the memory is over. With some queue size reporting:

private var audioQueueSize: Atomic<Int> = .init(0)

public func append(_ sampleBuffer: CMSampleBuffer, track: UInt8 = 0) {
    switch sampleBuffer.formatDescription?.mediaType {
    case .audio?:
        audioQueueSize.mutate { $0 += 1 }
        if audioQueueSize.value > 10 {
            print("audioQueueSize: \(audioQueueSize.value)")
        }
        mixer.audioIO.lockQueue.async {
            self.mixer.audioIO.append(track, buffer: sampleBuffer)
            self.audioQueueSize.mutate { $0 -= 1 }
        }
    case .video?:
        ...
    }
}

audioQueueSize increases until there is no memory. This happens sometimes when I switch between the apps. However, I can't create a reproducible example; using 11 Pro Max + iOS 17.4.1. Before I had the same issue while switching to BE headphones on iOS 15; and it probably was fixed with #1384.

Would it be reasonable to add some kind of capacity limiter to the lockQueue?

[levs42]

I managed to reproduce the issue on 11 Pro Max + iOS 17.4.1 while switching between multiple heavy apps: Asphalt 9, Bad Piggies, Angry Birds 2, Fruit Ninja. videoIO starts choking too and it results to OOM. Queue limiter helps to avoid the crash. It looks like running a heavy app reduces available CPU/GPU for couple of seconds and the video encoder starts lagging. Same about the audio, skip in IOAudioRingBuffer starts growing (2031; 8192; 7168; 7168). Here's a fix proposal to mitigate queue accumulation:

if videoQueueSize.value > 5 { // some number
    return
}
videoQueueSize.mutate { $0 += 1 }
mixer.videoIO.lockQueue.async {
    self.videoQueueSize.mutate { $0 -= 1 }
    ...

[shogo4405]

If you incorporate it into HaishinKit, I think the right approach is to calculate the processing delay of the VideoCodec and drop CMSampleBuffer frames accordingly. I plan to find some time to attempt this.

If you focus on ReplayKit, you can use os_proc_available_memory() to get the current amount of memory available to the process. Using this value to drop CMSampleBuffer frames is another possible method.

This way, you don't have to consider how many frames can be queued, and you don't have to modify HaishinKit, so I think it’s an optimal workaround.

[levs42]

Thank you for pointing the right direction. I tried the following and it works fine:

private var maxMemory = os_proc_available_memory()

override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
    let availableMemory = os_proc_available_memory()
    maxMemory = max(maxMemory, availableMemory)
    guard availableMemory > maxMemory / 10 else {
        return
    }
}

I'm curious to look at the implementation of the processing delay. Would you drop it if difference between last processed and the current buffer more than 1 / fps?

[shogo4405]

There is a risk of dropping data even during temporary stalls, so I think it would be better to use something like AsyncStream to store N items in the execution queue and delete the older ones.

https://developer.apple.com/documentation/swift/asyncstream/init(_:bufferingpolicy:_:)#discussion

[shogo4405]

[FYI] Starting from version 2.0.0, you will be able to specify a buffer limit like this.
https://github.com/shogo4405/HaishinKit.swift/blob/7bf05519cfe9b08bb37072061257340b739ab9d3/Examples/iOS/Screencast/SampleHandler.swift#L58