Improving existing Animation System to improve usability for adaptive/dynamic Music

[betalars]

Describe the project you are working on

A game that has an adaptive Soundtrack.

Describe the problem or limitation you are having in your project

Syncing Audio Players, using Animation-Trees to manage Music Clips, doing transitions/intros/endings ... it's really awkward right now. I have not even found a native way to keep a piece of music playing when an animation that started it ends, because even when using a PropertyTrack to set an AudioPlayer to be plying, Godot will be smart and treat it as an AudioPlaybackTrack.
I may be severely overlooking something, but I also have not found great answers in online resources.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

I propose several small updates to the AnimationPlayer and AnimationTree to accommodate adaptive Music without breaking a sweat.

The smallest change, that would make adaptive Music possible:

• Allow an AudioStream to be qued by an Animation, that will continue playing after the animation is done
  • also allow an Animation to continue playing a stream if it is already running

Some further worthwhile, but larger and arguably optional changes:

• Allow Animating on Beats (and fractions of beats) as an alternative to Frame-Based Animation (this will require some tweaks in both AnimationPlayer and AnimationTree)
  • next to FPS and Seconds, add a "Beat" Option, that allows you to configure a beat (so 3 over 4 120 BPM)
  • Options to AnimationTreeStateMashines so
    • Transitions may take an amount of beats
    • Transitions may only start every 5th major or minor beat for instance
• add an automagic Button, that sets the length of an Animation to be the length of an audio file

The changes to the tree might also require the Tree to have a metronome, but I am not quite sure on this. Setting a BPM/beats when recording an Audio track and selecting the timestamp of the first beat may also be a good idea if beat-based animations were to become a thing.

I suspect there may be some remaining tweaks I currently don't see, but I currently can't think of much else I'd need.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Will work on this.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Bodgy Music Players are pretty easy to script, but it gets really difficult really fast.

Is there a reason why this should be core and not an add-on in the asset library?

Frankly this is already trying to be addressed here and I think this is the much simpler Solution.

[betalars]

So I've just realized this whole thing might become a whole much bigger beast with rhythm based games, like for instance with Remember Me.

Having a core utility I would call a "conductor" that allows you to have a rythm in your Scene, to cue sound effects based on that rythm, to handle input based on rythm while having koyote-frames (so it can handle inputs even when they are slightly off beat) sounds like an incredible utility, that will make developing a lot of games a lot easier.

But also ... the scope of this would be substantial.

[Mickeon]

Not all games, not even most, may necessarily benefit from this. Therefore it may be better off as an addon.

[bend-n]

I disagree, I think this is quite a useful feature, and should be in core.
As godot already has many audio utilities (generator, bus, doppler, attentuation) i think adding some more is okay.
However, this proposal isnt very well fleshed out, and the issue seems to be being unable to keep a stream playing with a animation.

[betalars]

Not all games, not even most, may necessarily benefit from this. Therefore it may be better off as an addon.

The thing is: implementing it on an engine level by improving the animation systems to work better with rythm is imo a much better way to go about it, and choosing this approach would be really clunky to do as an AddOn.

And like ... most games made in Godot will not use localisation, but there's a reason it's natively supported and I think this falls into that category.

[AThousandShips]

And like ... most games made in Godot will don't use localisation

While it is arguably off topic I would like to see where you get this from? Also the editor uses localization.

How does this godotengine/godot#64488 fit this proposal?

[bend-n]

Quite well imo! i already have a thumbup on that pr.

[AThousandShips]

Be aware that an addon can be built as a module an be fully integrated, though requiring you to rebuild the engine, so performance doesn't really come into it

What does come into it however is the workload in maintaining any feature, the size taken up by it, etc.

[betalars]

And like ... most games made in Godot will don't use localisation

While it is arguably off topic I would like to see where you get this from? Also the editor uses localization.

How does this godotengine/godot#64488 fit this proposal?

Quite well looking at it broadly, I only checked for adaptive music. Will probably close this issue once I've had a closer look at it.

[betalars]

How does this godotengine/godot#64488 fit this proposal?

I've looked at it an I'm not going to close the issue right now, as I'm not sure if the Merge is going to go trough and if it is the best approach. I feel like retro-fitting an animation-system to an audio player is really redundant when we have an animation system already, that we could just make more compatible with music.

[AThousandShips]

Then the issue of feature bloat and maintainability remains, will quote myself from another proposal:
As this looks like a relatively niche feature, most projects are likely to use pre-processed audio as far as I know, there's many arguments against making it part of the engine, it adds to the size of the source and the executable and exports, and is harder to maintain, as a dedicated plugin or module maintained by people focused on it makes this much easier and smoother without complicating the situation for those who don't need it or use it.

[betalars]

Then the issue of feature bloat and maintainability remains

I don't think so honestly. I've looked further into AnimationTree and AnimationPlayer and I honestly think these components in conjunction are at 90% of what we'd need for a great adaptive Music System.

I really need to update my initial comment, but I think it's kind of wasteful to not fix the remaining issues with these systems, that prevent them from being useful to play music instead of building a new Animation System into the AudioStreamPlayer, witch seems to be what the Godot Devs are currently trying to do to accommodate adaptive music.

[betalars]

I have changed the Proposal to be a lot more clear now and will soon provide some UI Mockups.

[betalars]

@Riteo

it applies everything but there's no easy way of syncing everything

So in a professional setting, you would sync your audio before putting it trough the mixer, witch is also what I'd suggest for this.

transition types and the actual audiobus orchestration (eg. when to lower what) kinda concern me

Okay, you can adjust levels for each AudioStreamPlayer right now, so that would be fine, but orchestrating the effects you'd put on a bus could be difficult, as the Audio-Busses are global state iirc and I don't know how well Godot
Handles Animating these from Editor.

This would mean taking keyframe-based animations and add audio-specific handling, which is again kinda sus.

I mean on what measure you are animating is purely a UX-Question. You can put framerate to whatever FPS you like, you can speed up or slow down animations, the Engine doesn't really care about it.

You'd have to do some amount of rework to the UI, but I think it's far less substantial than the currently open branch.

[deralmas]

@betalars

I mean on what measure you are animating is purely a UX-Question. You can put framerate to whatever FPS you like, you can speed up or slow down animations, the Engine doesn't really care about it.

I'm pretty sure that animation handling revolves around keyframes and that adding something like BPM handling would be a separate new feature, not just an UX question.

BTW note that I'm not at all against an audio animation system and I do indeed see similarities, but I really don't like the idea of mixing animations and audio in the same place, instead preferring the idea of having a common framework for such things, if that even makes sense.

[reduz]

The audio playback code on the animation side has been greatly improved recently. I think setting a BPM for snapping on AnimationPlayer could be done easily, but audio would not start exactly synchronized and this may be a problem, this needs to be worked out on the audio server side.