Thread issue?

[stellanhaglund]

I'm trying to run the denoiser on microphone input from ios.

And I have a problem.

The rust module.

pub unsafe extern fn detect(size: usize, array: *const i16, detector: *mut c_void) {

    let mut out_buf = [0.0; DenoiseState::FRAME_SIZE];
    let mut denoise = Box::from_raw(detector as *mut DenoiseState);
    
    let audio: &[i16] = std::slice::from_raw_parts(array as *const i16, size as usize);
    let mut real: Vec<f32> = vec![]; 

    for frame in audio.iter() {
        real.push(*frame as f32);    
    }

    for _chunk in real.chunks_exact(DenoiseState::FRAME_SIZE){
        denoise.process_frame(&mut out_buf[..], _chunk);        
    }

    println!("got audio");
    Box::leak(denoise);

}

#[no_mangle]
pub extern fn detect_init() -> *mut c_void {
    let denoise = DenoiseState::new();
    Box::into_raw(Box::new(denoise)) as *mut c_void
} 

and the swift part

   let detector = detect_init()

    inputNode.installTap(onBus: 0, bufferSize: AVAudioFrameCount(bufferSize), format: inputFormat) {
      buffer, time in
        
        //buffer.frameLength = self.mySampleRate

      self.conversionQueue.async {
        // An AVAudioConverter is used to convert the microphone input to the format required
        // for the model.(pcm 16)
        guard let pcmBuffer = AVAudioPCMBuffer(
          pcmFormat: recordingFormat,
          frameCapacity: AVAudioFrameCount(recordingFormat.sampleRate * 2.0)
        ) else { return }

        var error: NSError?
        let inputBlock: AVAudioConverterInputBlock = { _, outStatus in
          outStatus.pointee = AVAudioConverterInputStatus.haveData
          return buffer
        }

        formatConverter.convert(to: pcmBuffer, error: &error, withInputFrom: inputBlock)

        if let error = error {
          print(error.localizedDescription)
          return
        }
        if let channelData = pcmBuffer.int16ChannelData {
          let channelDataValue = channelData.pointee
          let channelDataValueArray = stride(
            from: 0,
            to: Int(pcmBuffer.frameLength),
            by: buffer.stride
          ).map { channelDataValue[$0] }
            
            
            detect(channelDataValueArray.count, channelDataValueArray, self.detector)

          // Converted pcm 16 values are delegated to the controller.
//          self.delegate?.audioInputManager(self, didCaptureChannelData: channelDataValueArray)
        }
      }
    }

When I run the code without the denoise.process_frame() and just print the frames there is no problem but as soon as I use it I get memory errors.
I tried wrapping the whole denoiser in a mutex but that didn't help. any suggestions?

[jneem]

Hm, I don't see anything obviously wrong. Can you elaborate about the memory errors? Like, if it's a use-after-free then presumably something is getting dropped when it shouldn't be. I'd be surprised if it were a memory error from within nnnoiseless, because it literally has 2 unsafe blocks and they're very simple.

Is it possible for detect to be called simultaneously from multiple threads? If so, I think the line let mut denoise = ... is already UB even if you don't use it.

[stellanhaglund]

Yes it’s a use after free.
I can send the error when I’m back at the computer.

My guess is that it’s multithreaded because it works in an asynchronous way, you install the tap and it gets called every time there’s a new chunk.

not sure what you mean by UB?

[jneem]

By UB I meant undefined behavior: I'm pretty sure it's undefined behavior to create two Boxes pointing to the same location.

Anyway, it would be definitely be useful to know which memory was freed, and where.

[stellanhaglund]

Maybe it will be better to get the input data from cpal instead, that will make the code more portable as well, I will try that.
If you want we could add it to some sort of examples