Add Note-On-Sustained as source modulator

[RobsonFBP]

add note-on sustain as a modulator source

Could someone please add note-on-sustain as a modulator source?
I mean note-on-sustain when the key is released, but the note remains sustained by the depressed sustain pedal.

According to the specification, these are the non-CC sources:

0 No Controller
2 Note-On Velocity
3 Note-On Key Number
10 Poly Pressure
13 Channel Pressure
14 Pitch Wheel
16 Pitch Wheel Sensitivity

I propose to create this additional number 1:

0 No Controller
1 Note-On Sustained
2 Note-On Velocity
3 Note-On Key Number
10 Poly Pressure
13 Channel Pressure
14 Pitch Wheel
16 Pitch Wheel Sensitivity

1 Note-On Sustained = The controller source to be used is the event value key_off is sent from the MIDI note-on command which generated the given sound width CC64 depressed.

This way, a volume envelope could be applied when the key is released and is sustained by CC64=1

I found that in "fluid_voice.c" there is the function: "fluid_voice_noteoff(fluid_voice_t *voice)" that changes the voice status to Sustained if the pedal is already pressed.

I believe that this is where this modulator should be triggered if it is activated.

[derselbst]

I fail to comprehend this approach, sry.

If you want this to be taken seriously, you should once again explain your use-case or problem you're trying to solve. I remember that we've already been in touch some time ago, so feel free to share a link to where you have explained this before.

The controller source to be used is the event value key_off is sent from the MIDI note-on command which generated the given sound width CC64 depressed.

I fail to understand what you mean by this sentence. Pls. try again. It would also be helpful if you provided a full modulator example. So far, you have only suggested to introduce a new modulator source, but you have not explained how this is supposed to influence any particular destination generator, which is the main task of a modulator.

[RobsonFBP]

Hello from @derselbst . Thank you for your attention.

I fail to understand what you mean by this sentence. Pls. try again. It would also be helpful if you provided a full modulator example.

Sorry for my bad English.

The controller source to be used is the event value key_off is sent from the MIDI note-on command which generated the given sound width CC64 depressed.

Actually I meant: "The controller source uses the value of the key_off event sent from the MIDI command with the sustain pedal pressed".

I would like to be able to modify the volume envelope when the voice status changes to sustained: (voice->status = FLUID_VOICE_SUSTAINED).

I have a flute preset that sounds indefinitely while the key is pressed and also when the key is released while the sustain pedal is pressed.

This is the initial volume envelope: Delay = 0.001s, Attack = 0.025s, Hold = 0.001s, Sustain = 0.0 dB, Release = 0.250s, Decay= 9.998s

With modulator 1 Note-On Sustained = 1, amout 600 mod dest volume envelope sustain.

When the modulator is activated there should be a smooth decay in about 10 seconds.

The existing modulation sources do not give us access to the key release event (key_off) when the sustain pedal is pressed, so I propose creating a modulation source (Note-On Sustained) to be triggered when the voice changes status to sustained (voice->status = FLUID_VOICE_SUSTAINED).

Note that this smooth decay must occur before volume envelope release.

[derselbst]

In your screenshot, modulator source1 has an unipolar positive switch. That means, if the key is less than 64, the effect of the modulator would be zero, i.e. it wouldn't work I'm afraid.

---

The key of a noteOff event is the same key of the noteOn event. So why can't you just specify a modulator with two sources, like so?

src1: 3 - Note-On Key Number, bipolar switch
src2: CC64 - Sustain, unipolar positive switch
amount: 600
absolute transform

In this case, if sustain pedal is not pressed, the effect of the modulator would be zero. If sustain pedal is pressed, the modulator would evaluate to +-600 cB, the absolute transform will turn it into +600 cB. Wouldn't that solve your problem, or am I missing smth.?

[RobsonFBP]

The key of a noteOff event is the same key of the noteOn event.

The intended source value here is not the key but whether the event (sustain) occurred.

In this case, if sustain pedal is not pressed, the effect of the modulator would be zero. If sustain pedal is pressed, the modulator would evaluate to +-600 cB, the absolute transform will turn it into +600 cB. Wouldn't that solve your problem, or am I missing smth.?

The effect should only occur if the key is released. If this doesn't happen, when the sustain pedal is released, the volume envelope returns to its previous state, increasing the volume.
With the key released, when the pedal is released, the volume envelope goes to the release state.

We need to catch the KeyOff event.
I'm differentiating between the KeyOff and NoteOff events.
KeyOff occurs when the key is released,
NoteOff occurs when the sound ends.

Usually, these would happen simultaneously, but if the sustain pedal is pressed and the key is released, the keyOff event occurs. However, the noteOff event will only occur when the pedal is released.

So why can't you just specify a modulator with two sources, like so?

Considering your proposal to use two sources, we could create the HasKeyOff source that would catch the key release event.

You might not even need to create the new source, but consider the keyOff event when using the CC64 source with the volume envelope sustain destination.

In fact, the sustain pedal's function only kicks in when the key is released.

[derselbst]

NoteOff occurs when the sound ends.

Let's not override the meaning of existing terms. NoteOff refers to the MIDI event. The corresponding sound will end when the voice has finished its release phase (e.g. let's call it voiceOff, or soundOff).

But I think I'm starting to understand it. What you need is a way for the modulator to only become effective once the noteOff event has been received. So let's consider my example from above again:

src1: 3 - Note-On Key Number, bipolar switch
src2: CC64 - Sustain, unipolar positive switch
amount: 600
absolute transform

If you were able to not only specify an "absolute transform" but also an additional transform, which evaluates to:

• either 1, when a noteOff event has been received, or
• 0 otherwise,

from my understanding, this would satisfy your use-case, wouldn't it?

I'm suggesting to use an additional transform because, whether or not a noteOff event has been received is a binary option, i.e. it wouldn't make sense to waste a modulator source for this, as you cannot map it in the range [0;1].

(I'm trying to find more general solution to your specific use-case.)

[RobsonFBP]

Let's not override the meaning of existing terms. NoteOff refers to the MIDI event.

OK.

What you need is a way for the modulator to only become effective once the noteOff event has been received.

That's right, you got it.

If you were able to not only specify an "absolute transform" but also an additional transform, which evaluates to:
either 1, when a noteOff event has been received, or
0 otherwise,

Yes, we can add an additional transform.

.

which evaluates to:
either 1, when a noteOff event has been received, or
0 otherwise,

I believe it could also be evaluated like this:
either 127, when a noteOff event was received, or
0 otherwise.
In this case, no additional transformations would be necessary.
The noteOff event is a binary option as is CC64, both of which can receive the values either 0 or 127, so you can map it in the range [0;1].

.
If there is a way for the modulator to only become effective after the noteOff event is received, the problem will be solved.

[derselbst]

The noteOff event is a binary option as is CC64, both of which can receive the values either 0 or 127, so you can map it in the range [0;1].

Not quite. CC64 has values from 0 up to 127. It is the user's decision to use a unipolar positive switch mapping which results in a binary 0 or 1 mapping. The "note off received" source however is binary right from the beginning, making the mapping you've chosen pretty much useless. You would be wasting a modulator source for this.

That's why I prefer the transform approach.

So, here's the next step: I will create an enhancement ticket and ping some community members to hear their opinions.

In the meantime, you could go ahead and create a test case:

a) A MIDI file which does the following:

1. NoteOn key=60, vel=127
2. CC64=127 to enable sustain
3. NoteOff key=60
4. (... the sound of the flute in your case will now play in sustain phase...)
5. Use any other CC to modulate the VolEnvSustain. Eg. CC65 that you ramp from 0 to 127.

b) A corresponding test soundfont will contain a single instrument with a single modulator:

src1: CC65, unipolar positive linear mapping
amount: 600
dest: VolEnvSustain

This simple test case will reveal, whether fluidsynth is actually able to manipulate the VolEnvSustain while the note is playing in sustain phase and has already received a noteOff. Looking at the code I'm not quite sure if this would actually work. That's why having such a test case would be helpful to ensure your use-case would be supported later on.

[RobsonFBP]

I ran the test as you suggested to verify that FluidSynth is actually capable of manipulating VolEnvSustain while the note is playing in the sustain phase and has already received a noteOff.

Yes! FluidSynth can do this!

a) A MIDI file which does the following:

I recorded the MIDI file.

b) A corresponding test soundfont will contain a single instrument with a single modulator:

I created the soundfont:
source1: CC65
amout: 600
dest: VolEnvSustain

Note that I entered a volume decay value of 15 seconds for the volume envelope, which is responsible for the smooth decay.

Below are the test.mid and test.sf2 files.

test.zip

[derselbst]

I'm afraid it does not work as expected. At least from my testing. And that's for two reasons:

• CC65 is unfortunately among those CCs that are not allowed to modulate. Fluidsynth still considers the value of the CC for modulators at the time when spawning a voice. But when the CC changes afterwards, it will have no effect on the playing voice.
• So I picked CC75 instead (which fluidsynth initializes to 64 by default - good to keep in mind).
• But still, I hear a continuous flute sound even though CC75 suddenly jumps to 127. I would have expected for the volume to decrease equally quickly, but this doesn't happen.

Below is my test, also including two renderings. Maybe I missed something. However unfortunately when I ran into the first issue, CC64 Sustain is among those that is "not be used as sources for the Modulator implementation" according to the SF2 spec. So technically it cannot be used to apply modulators that become only effective when sustain is enabled. Ofc, one might lift this requirement, but it would be yet another non-standard adjustment.

In addition - assuming that modulating the volEnvSustain in realtime would work as expected - Davy has made a valid point in the ticket I've opened: Basically, if the note plays longer than the decay time (i.e. causing the noteOff event to arrive after 15s) the volume would suddenly jump to 60dB of attenuation causing an audible break in the sound. Not sure whether you would consider this to be a problem though, it's just something one should be aware of.

test_cc75.zip

[RobsonFBP]

But still, I hear a continuous flute sound even though CC75 suddenly jumps to 127. I would have expected for the volume to decrease equally quickly, but this doesn't happen.

Maybe the final result will be 30db on this one, because of the range[-63:64].

In addition - assuming that modulating the volEnvSustain in realtime would work as expected - Davy has made a valid point in the ticket I've opened: Basically, if the note plays longer than the decay time (i.e. causing the noteOff event to arrive after 15s) the volume would suddenly jump to 60dB of attenuation causing an audible break in the sound.

I didn't pay attention to this. I thought I could keep volEnvHold while noteOff didn't happen, but that wasn't possible.

the volume would suddenly jump to 60dB.

I've noticed this a few times, but I haven't related it to the volEnvDecay time.

In addition - assuming that modulating the volEnvSustain in realtime would work as expected - Davy has made a valid point in the ticket.

I enjoyed seeing the high level conversation. I think the solution presented by @Davi7125 excellent and simple to implement, in my opinion: create another playback mode.

Another playback mode would be used, which is like the loop mode ON but not taking into account the state of the sustain pedal. As soon as the key is released, the voice enters in the release phase regardless CC64.

I tested it using CC69. It looked similar to @Davi7125's chart.

[RobsonFBP]

I tested it using CC69. It looked similar to @Davi7125's chart.

I tested it using CC69. It looked similar to @Davi7125's chart, but not quite the same.
Changing the release at runtime causes an abrupt drop when releasing the sustain pedal.

I'm thinking of another solution without modulators.
I believe the envelope initially consisted of attack, decay, sustain, and release (ADSR).

Delay and hold must have been introduced later.

I propose creating a DECAY2 phase in the envelope.

The decay2 phase could be introduced into the envelope after the sustain phase and before the release phase, with a drop for inaudible audio like the release phase.

,

The decay2 phase would only trigger if these two criteria were met:
1. NoteOff with the hold pedal ON.
2. Decay2 time greater than the release time.
If either of these is not met, the decay2 phase does not trigger.

1. This way, there is no need to modify the release at runtime.
2. Does not create new sources or destinations for modulators.
3. There should be no abrupt drop in sound.

What do you think of this?

[derselbst]

Adding a new phase into the AHDSR would be a pretty intrusive and complex change to the synthesis model, making it less likely to be adopted by other SF2 synths. Also, the resulting effect of your proposal would be pretty similar to what Davy has suggested in this comment.

[RobsonFBP]

Hi, @derselbst . Could you open this pull request?

//src/synth/fluid_voice.h add ignore_sustain to struct _fluid_voice_t at line 80

	unsigned char ignore_sustain;

//src/synth/fluid_voice.c init value to ignore_sustain at new_fluid_voice at line 249

	voice->ignore_sustain = 0;

// src/synth/fluid_voice.c evaluate ignore_sustain in fluid_voice_calculate_runtime_synthesis_parameters inside loop structure at line 616

	/*      fluid_dump_modulator(mod); */
	if (mod->src1 == SUSTAIN_SWITCH && mod->src2 == FLUID_MOD_NONE && mod->dest == GEN_VOLENVRELEASE)
	{
		voice->ignore_sustain = 1;
	}

// src/synth/fluid_voice.c add to try force new release if voice already is in release fase in fluid_voice_update_param in case GEN_VOLENVRELEASE at line 1106

	    //old 
	break;
	    //new
	if(fluid_adsr_env_get_section(&voice->envlfo.volenv) == FLUID_VOICE_ENVRELEASE)
	{
		fluid_voice_release(voice);
	}
    break;

// src/synth/fluid_voice.c change that line to ignore default sustain at fluid_voice_noteoff at line 1365

	//old
else if(fluid_channel_sustained(channel))
	//new
else if(fluid_channel_sustained(channel) && !voice->ignore_sustain)

With these few lines, perhaps you could modify the release without adding a new source. We only ignore channel_sustained if CC64 wants to modulate volEnvRelease.
If the voice is already in the release phase, we send a new release to try to force the time recount, avoiding an abrupt drop.

[derselbst]

if(fluid_adsr_env_get_section(&voice->envlfo.volenv) == FLUID_VOICE_ENVRELEASE)
{
fluid_voice_release(voice);
}
break;

fluid_voice_release causes the volenv to be advance to FLUID_VOICE_ENVRELEASE. That's it. The additional call to fluid_voice_release would effectively be a no-op.

Apart from that your change wouldn't even compile: fluid_voice_t has no member envlfo. Only the rvoice has. But accessing the rvoice in this context is forbidden.

[RobsonFBP]

your change wouldn't even compile: fluid_voice_t has no member envlfo. Only the rvoice has. But accessing the rvoice in this context is forbidden.

Wow. I didn't notice that either.

I wanted to make sure there would only be a release if it was already in progress, so the count would start at the time of the event.

(If the release was set to 5 seconds and 2 seconds have already passed by the time it's changed to 1 second, there will be a sudden drop because 2 seconds have already passed since the start of the count. Therefore, I believe that if there's a new release, the count will start over.)

The evaluation needs to be done before changing the release time.

Let's change it to this in line 1101:

/* old */

case GEN_VOLENVRELEASE:             /* SF2.01 section 8.1.3 # 38 */
    fluid_clip(x, FLUID_MIN_VOLENVRELEASE, 8000.0f);
    count = 1 + NUM_BUFFERS_RELEASE(x);
    fluid_voice_update_volenv(voice, TRUE, FLUID_VOICE_ENVRELEASE,
                              count, 1.0f, -1.0f / count, 0.0f, 1.0f);
    break;

/* new */

case GEN_VOLENVRELEASE:             /* SF2.01 section 8.1.3 # 38 */
    if(voice->has_noteoff && voice->ignore_sustain)
    {
        fluid_voice_release(voice);
    }
    fluid_clip(x, FLUID_MIN_VOLENVRELEASE, 8000.0f);
    count = 1 + NUM_BUFFERS_RELEASE(x);
    fluid_voice_update_volenv(voice, TRUE, FLUID_VOICE_ENVRELEASE,
                              count, 1.0f, -1.0f / count, 0.0f, 1.0f);
    break;

[derselbst]

Therefore, I believe that if there's a new release, the count will start over.)

But this is not how it works. As I said "fluid_voice_release causes the volenv to be advanced to FLUID_VOICE_ENVRELEASE". Releasing it again wouldn't have any effect.