Planning for the new Audio Graph

[SolarLiner]

With audio work gaining traction in bevy, I thought it would be appropriate to create a roadmap of sorts laying out what needs to be done, but before, a little recap of what this new audio graph is:

The Audio Graph wants to provide a unified abstraction over audio on bevy, much like bevy_math is a unified abstraction over CG linear algebra in bevy, or bevy_render is a unified abstraction over 3D graphics. This is because, as the ecosystem is currently fragmented, a lot of features end up getting written multiple times over to target each backend/framework/library.

Whether it's by reusing an existing a library, or building our own, the Audio Graph aims to be flexible enough to incorporate all use cases while being easy enough to work on, and efficient enough to run within the tight constraint of real-time audio processing. It's a "want all three, choose only two" kind of deal, and to help get this off the ground, I personally think we should focus on efficiency and simplicity at the detriment of flexibility. Flexibility is something that is earned through the hard work of iterating on the API surface, and so we need to bump into the limits of what we design to better understand it and evolve it.

With that in mind, I propose the following roadmap of "events", where the timeline is neither linear nor matching any kind of bevy-wide milestone (so as to not set any expectation):

Milestone 1: The Audio Graph

• Set up an audio graph abstraction that allows ECS systems to mutate the graph, and run it on the audio thread
  • Define traits for the various kinds of nodes (audio source/sink and filters)
  • Evaluate current libraries (rodio/oddio/knyst/cpal/other) or begin our own
• Provide a minimal Proof of Concept set of nodes into a "standard library" - audio player, hardware sources and sinks, basic effects (reverb, panning, filtering)
• High-resolution clock from the audio thread
• Update examples & documentation

Milestone 2: Audio events

• Sample-accurate scheduling of audio events (ie. playing audio/sending MIDI/parameter changes) into the future, for scheduling audio events in time with graphical events
• Depending on the performance of the "naive" runner which traverses the graph directly to process audio, add a "compilation" step to linearize the process
• Audio events from the audio thread

Milestone 3: Spatial audio

• Source-to-sink propagation delay/Doppler effect
• Reverb boxes
• HRTF convolution

Milestone 4: Bigger files

• Integrate file streaming for large files (> 1GB, configurable?) (need collaboration with assets dev)

Milestone 5: User feedback

• Reflect and reprioritize audio features based on demand volume
• <more ambitious?>

This is obviously not set in stone or possess any authority whatsoever, but I feel centralizing information about this is important to help get the effort started.

Happy bleep bloop bevying!

EDIT 2022/11/19: Inserted Milestone 3 for Spatial audio
EDIT 2022/11/20: Inserted Milestone 4 for file streaming

[alice-i-cecile]

This is obviously not set in stone or possess any authority whatsoever, but I feel centralizing information about this is important to help get the effort started.

As someone with a moderate amount of authority here, I'm on board with this plan. It's clear, user-focused and made of useful, actionable steps.

[SolarLiner]

Spliced in a new Milestone 3 with spatial audio, which is a popular feature which I had completely forgotten about somehow.

[harudagondi]

I have now a working minimal viable prototype (it plays audio at least) for this: https://github.com/harudagondi/bevy_prototype_audio_graph

This is a Entity/Component centric prototype, forgoing asset handling and resource based API.

The current example for the API is like this:

use bevy::{
    prelude::{App, Commands, Entity, Resource},
    DefaultPlugins,
};
use bevy_prototype_audio_graph::{Audio, AudioPlugin, SineWave};

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_plugin(AudioPlugin)
        .add_startup_system(play_sound)
        .run()
}

#[derive(Resource)]
struct SoundEntity(Entity);

fn play_sound(mut commands: Commands) {
    let entity = commands
        .spawn(Audio::new(SineWave {
            frequency_hz: 440.0,
            phase: 0.0,
        }))
        .id();
    commands.insert_resource(SoundEntity(entity));
}

Current questions:

• How should audio files be handled? This is hard since asset loading is asynchronous and the current prototype expects the entire audio data to be loaded.
• How should audio sinks/custom audio controls work?
  • Should we utilize oddio::Controlled to create a AudioControl component?
• How to communicate from audio graph to the ECS?
• What's the best API to manipulate the audio graph assuming we have currently playing sound? This will solve the audio filters story.
• How do we handle entities with both Audio and Transform components? How about entities that don't have the Transform component
• Can we integrate other libraries (rodio, kira, oddio) into this easily?
  • oddio can probably be integrated similar to the fundsp_encapsulation example in knyst.
• Insert other questions here that I haven't thought of yet.

[SolarLiner]

Nice job!

For files, for 99% of use-cases (one-shot samples, loops and background music/soundscapes), audio files will be small enough that holding them in memory is going to be fine; so I think that not worrying too much about that (especially for milestone 1) is okay, and we'll kick the can down to future us.

For manipulating the graph we'll need to have stable node references similar to how it's done in the render graph, so that systems can connect to known points. Bar that, we'll need to have a query system that can return a best-matching node connection for the new audio node to connect to (e.g. "give me the first mono hardware input" or "if it's a 5.1 context, give me the output LFE channel").

Manipulation of the audio graph while playing is going to be done with a communication layer coming up soon into knyst, that will be able to schedule control changes.

Integrating other libraries is going to be a secondary objective, and the likes of rodio which are by design overarching are going to he hard or impossible anyway. Instead we should provide a best-effort solution for users to do this, should they really want to (but why, because this audio graph will have everything they need already and be perfect beyond imagination? 😝 ). oddio is a different story because it's purely DSP, it's similar conceptually to having hierarchical/nested audio graphs.