JSBSim's randomseed impact on turbulence generation #1250
-
Hello JSBSim Team, I use the JSBSim python library version 1.2.0 on a Ubuntu 22.04 machine.
As you can see, the wind speed figures for both experiments are not identical when looking at the same seeds. The profile of the curves looks the same, but there are offsets. I would expect identical seeds to produce the exact same turbulent wind across experiments. Could you please provide an explanation ? In case you need it, here is the python script I wrote, I hope it's minimal enough for you to have a look. import numpy as np
import jsbsim
import matplotlib.pyplot as plt
from fw_jsbgym.trim.trim_point import TrimPoint
def set_wind(fdm):
# Setting up the wind: 5kph constant wind and severe turbulence
fdm["atmosphere/wind-north-fps"] = 5 * 0.3048
fdm["atmosphere/wind-east-fps"] = 0
fdm["atmosphere/wind-down-fps"] = 0
fdm["atmosphere/turb-type"] = 3
fdm["atmosphere/turbulence/milspec/windspeed_at_20ft_AGL-fps"] = 75
fdm["atmosphere/turbulence/milspec/severity"] = 6
def main():
# Setting up the JSBSim FDM
fdm = jsbsim.FGFDMExec("fdm_descriptions")
fdm.set_debug_level(0)
fdm.load_model("x8")
fdm.load_ic("initial_conditions/x8_basic_ic.xml", False)
fdm.run_ic()
# 2 episodes of 2000 timesteps each
episode_max_tsteps = 2000
n_episodes = 2
max_tsteps = episode_max_tsteps * n_episodes
# One seed per episode
seeds = [1, 2] # [1, 2] for 1st plot, [2, 1] for 2nd plot
fdm["simulation/randomseed"] = seeds[0]
set_wind(fdm)
# Setting up the commands to be at trim
trim_cmd = TrimPoint("x8")
# Create storing buffers
roll = np.zeros((n_episodes, episode_max_tsteps))
pitch = np.zeros((n_episodes, episode_max_tsteps))
yaw = np.zeros((n_episodes, episode_max_tsteps))
wind_n = np.zeros((n_episodes, episode_max_tsteps))
wind_e = np.zeros((n_episodes, episode_max_tsteps))
wind_d = np.zeros((n_episodes, episode_max_tsteps))
total_t = 0
ep_cnt = 0
ep_t = 0
while total_t < max_tsteps:
# input commands
fdm["fcs/aileron-cmd-norm"] = trim_cmd.aileron # -4.288e-15 / 0.0
fdm["fcs/elevator-cmd-norm"] = trim_cmd.elevator # 0.0406
fdm["fcs/throttle-cmd-norm"] = trim_cmd.throttle # 0.1295
# print(f"fdm['simulation/randomseed'] = {fdm['simulation/randomseed']}")
# get the attitude of the aircraft
roll[ep_cnt, ep_t] = fdm["attitude/roll-rad"]
pitch[ep_cnt, ep_t] = fdm["attitude/pitch-rad"]
yaw[ep_cnt, ep_t] = fdm["attitude/psi-rad"]
# get the wind speeds
wind_n[ep_cnt, ep_t] = fdm["atmosphere/total-wind-north-fps"]
wind_e[ep_cnt, ep_t] = fdm["atmosphere/total-wind-east-fps"]
wind_d[ep_cnt, ep_t] = fdm["atmosphere/total-wind-down-fps"]
fdm.run()
total_t += 1
ep_t += 1
if (ep_t == episode_max_tsteps) and (ep_cnt < n_episodes - 1):
ep_cnt += 1
ep_t = 0
fdm.reset_to_initial_conditions(0)
fdm["simulation/randomseed"] = seeds[ep_cnt]
set_wind(fdm)
print(f"Resetting to IC with seed {seeds[ep_cnt]}")
# Plot attitude and windspeed, episode 0 and 1 are side by side
fig, axs = plt.subplots(2, 2)
# Episode 0
axs[0, 0].plot(roll[0, :], label="roll")
axs[0, 0].plot(pitch[0, :], label="pitch")
axs[0, 0].plot(yaw[0, :], label="yaw")
axs[0, 0].legend()
axs[0, 0].grid()
axs[0, 0].set_title(f"Attitude Angles [rad] - EP1 seed {seeds[0]}")
axs[1, 0].plot(wind_n[0, :], label="wind_n")
axs[1, 0].plot(wind_e[0, :], label="wind_e")
axs[1, 0].plot(wind_d[0, :], label="wind_d")
axs[1, 0].legend()
axs[1, 0].grid()
axs[1, 0].set_title(f"Wind Speeds[rad] - EP1 seed {seeds[0]}")
# Episode 1
axs[0, 1].plot(roll[1, :], label="roll")
axs[0, 1].plot(pitch[1, :], label="pitch")
axs[0, 1].plot(yaw[1, :], label="yaw")
axs[0, 1].legend()
axs[0, 1].grid()
axs[0, 1].set_title(f"Attitude Angles[rad] - EP2 seed {seeds[1]}")
axs[1, 1].plot(wind_n[1, :], label="wind_n")
axs[1, 1].plot(wind_e[1, :], label="wind_e")
axs[1, 1].plot(wind_d[1, :], label="wind_d")
axs[1, 1].legend()
axs[1, 1].grid()
axs[1, 1].set_title(f"Wind Speeds[rad] - EP2 seed {seeds[1]}")
# plt.savefig('diff_seeds.png')
plt.show()
if __name__ == '__main__':
main() Thank you in advance! |
Beta Was this translation helpful? Give feedback.
Replies: 17 comments 9 replies
-
Quick initial look. Line 170 in e1ea4e8 Lines 1372 to 1376 in e1ea4e8 Lines 80 to 81 in e1ea4e8 Lines 692 to 693 in e1ea4e8 Line 630 in e1ea4e8 jsbsim/src/models/atmosphere/FGWinds.cpp Lines 75 to 76 in e1ea4e8 jsbsim/src/models/atmosphere/FGWinds.h Line 397 in e1ea4e8 jsbsim/src/models/atmosphere/FGWinds.cpp Lines 304 to 307 in e1ea4e8 So it appears as if But even if a pointer to the original |
Beta Was this translation helpful? Give feedback.
-
So I tried a very quick test. Used the 737 cruise script and set the <use aircraft="737" initialize="cruise_init"/>
<run start="0" end="100" dt="0.008333">
<property value="0"> simulation/notify-time-trigger </property>
<property value="1"> simulation/randomseed </property> Then set some breakpoints in JSBSim to take a look at the random number generator, and confirmed that Then added this test call in bool FGWinds::Run(bool Holding)
{
if (FGModel::Run(Holding)) return true;
if (Holding) return false;
// Testing
double ran = generator->GetNormalRandomNumber(); And confirmed that I get consistent random numbers based on the seed, running each seed case a couple of times. Seed 1:
-1.4028727091279207
-0.54974617895544953
-1.0451468104420230
Seed 2:
0.66244780319854357
-0.022564746291165897
-0.46089239739463156 So at least in this very limited test case the random seed functionality of JSBSim appears to be working as expected. |
Beta Was this translation helpful? Give feedback.
-
@Dobid so I was able to get consistent winds based on a randomseed of 1 and 2 using this code. Running it multiple times while changing the seed and comparing the outputs. import jsbsim
import matplotlib.pyplot as plt
fdm = jsbsim.FGFDMExec('..\\')
fdm["simulation/randomseed"] = 1
fdm.load_model('737') # Load the aircraft 737
# Set engines running
fdm['propulsion/engine[0]/set-running'] = 1
fdm['propulsion/engine[1]/set-running'] = 1
fdm['ic/h-sl-ft'] = 20000
fdm['ic/vc-kts'] = 250
fdm['ic/gamma-deg'] = 0
fdm.run_ic()
fdm['simulation/do_simple_trim'] = 1
fdm["atmosphere/wind-north-fps"] = 5 * 0.3048
fdm["atmosphere/wind-east-fps"] = 0
fdm["atmosphere/wind-down-fps"] = 0
fdm["atmosphere/turb-type"] = 3
fdm["atmosphere/turbulence/milspec/windspeed_at_20ft_AGL-fps"] = 75
fdm["atmosphere/turbulence/milspec/severity"] = 6
wind_n = []
wind_e = []
wind_d = []
while fdm.get_sim_time() < 10:
fdm.run()
wind_n.append(fdm["atmosphere/total-wind-north-fps"])
wind_e.append(fdm["atmosphere/total-wind-east-fps"])
wind_d.append(fdm["atmosphere/total-wind-down-fps"])
plt.figure()
plt.plot(wind_n)
plt.plot(wind_e)
plt.plot(wind_d)
plt.show() Seed = 1 Seed = 2 |
Beta Was this translation helpful? Give feedback.
-
@bcoconni I'm wondering whether we shouldn't provide an option to specify an independent seed for Let's say I'm wanting to test an autoland autopilot and how well it performs with turbulence, and so I need to compare how different versions of it perform against the exact same set of turbulence. Now let's assume different versions of the autoland autopilot make use of a different number of Given that jsbsim/src/models/flight_control/FGSensor.cpp Lines 54 to 55 in e1ea4e8 jsbsim/src/models/flight_control/FGSensor.cpp Line 167 in e1ea4e8 jsbsim/src/models/flight_control/FGSensor.cpp Lines 185 to 192 in e1ea4e8 |
Beta Was this translation helpful? Give feedback.
-
Yep, the idea would be remain backwards compatible. So if no wind specific seed is specified the winds will continue to use the random number generator and it's seed from |
Beta Was this translation helpful? Give feedback.
-
@Dobid it looks there is some side-effect of enabling and then disabling the turbulence. If I remove the turbulence code then the wind values behave as expected. So will need to dig into the turbulence code to figure out what is going wrong when you try and disable the turbulence. fdm["atmosphere/wind-north-fps"] = 5 * 0.3048
fdm["atmosphere/wind-east-fps"] = 3 * 0.3048
fdm["atmosphere/wind-down-fps"] = 2 * 0.3048
#fdm["atmosphere/turb-type"] = 3
#fdm["atmosphere/turbulence/milspec/windspeed_at_20ft_AGL-fps"] = 75
#fdm["atmosphere/turbulence/milspec/severity"] = 6
....
if fdm.get_sim_time() > 5.0:
fdm["atmosphere/wind-north-fps"] = 1
fdm["atmosphere/wind-east-fps"] = 2
fdm["atmosphere/wind-down-fps"] = 3 |
Beta Was this translation helpful? Give feedback.
-
At first glance, I'll double-check later, my guess is that when the turbulence is switched off, the jsbsim/src/models/atmosphere/FGWinds.cpp Lines 141 to 149 in e1ea4e8 Will need to double-check if similar issues may affect |
Beta Was this translation helpful? Give feedback.
-
I've checked the source code for |
Beta Was this translation helpful? Give feedback.
-
Hmm, I think if you want to immediately update your jsbsim python package you would need to build it locally after the PR is submitted. Otherwise wait for the next jsbsim release - https://github.com/JSBSim-Team/jsbsim/releases When a new release comes out the python package at PyPi and Conda Forge are updated: |
Beta Was this translation helpful? Give feedback.
-
@Dobid yep it could be a while before the next release. I've just submitted a pull request - #1252 to fix the issue where the turbulence effect isn't removed completely when turbulence is disabled. I'll submit a separate pull request for separate random number generator work. |
Beta Was this translation helpful? Give feedback.
-
I've written some code to allow a separate random seed to be specified for However, I'm battling to write a test case for it using our python test framework. Here is my attempt so far. For now I'm testing with from JSBSim_utils import JSBSimTestCase, CreateFDM, RunTest
class TestTurbulenceRandomSeed(JSBSimTestCase):
def testTurbulenceRandomSeed(self):
# Test that the wind turbulence random seed is reproducible
wind_random_seed = 2
wn1, we1, wd1 = self.captureTurbulence(wind_random_seed, 5)
wn2, we2, wd2 = self.captureTurbulence(wind_random_seed, 5)
for i in range(len(wn1)):
self.assertAlmostEqual(wn1[i], wn2[i], delta=1E-8)
self.assertAlmostEqual(we1[i], we2[i], delta=1E-8)
self.assertAlmostEqual(wd1[i], wd2[i], delta=1E-8)
def captureTurbulence(self, wind_seed, exec_seed):
fdm = self.create_fdm()
# Set random seeds for FGFDMExec and FGWinds
fdm["simulation/randomseed"] = exec_seed
fdm["atmosphere/randomseed"] = wind_seed
fdm.load_model('A4')
# Set engine running
fdm['propulsion/engine[0]/set-running'] = 1
fdm['ic/h-sl-ft'] = 20000
fdm['ic/vc-kts'] = 250
fdm['ic/gamma-deg'] = 0
fdm.run_ic()
fdm['simulation/do_simple_trim'] = 1
# Setup turbulence
fdm["atmosphere/turb-type"] = 3
fdm["atmosphere/turbulence/milspec/windspeed_at_20ft_AGL-fps"] = 75
fdm["atmosphere/turbulence/milspec/severity"] = 6
wn = []
we = []
wd = []
while fdm.get_sim_time() < 0.1:
fdm.run()
wn.append(fdm['atmosphere/total-wind-north-fps'])
we.append(fdm['atmosphere/total-wind-east-fps'])
wd.append(fdm['atmosphere/total-wind-down-fps'])
# Attempted cleanup to force new FDM for next execution, not helping...
self.delete_fdm()
del fdm
fdm = None
print('')
print(wn)
return (wn, we, wd)
RunTest(TestTurbulenceRandomSeed) However, what I'm seeing is that creating 2 separate instances of the FDM via calls to (base) C:\source\jsbsim\tests>python .\TestTurbulenceRandomSeed.py
testTurbulenceRandomSeed (__main__.TestTurbulenceRandomSeed) ...
[0.047354487544379376, 3.460354080430806, 4.636633740213811, -0.4374137958205208, -0.06377215234821773, -1.5456835409458707, -3.4411298921520492, -6.057946773083572, -5.709936753084856, -5.628235304626574, -6.480877261842454, -5.210906810642065, -2.2376367571469964]
[-2.1842398652974198, 1.2417689243655539, 2.426390607718044, -2.652099515681857, -2.2717034356215526, -3.750613502555642, -5.643749459388144, -8.259702814605022, -7.904921571924209, -7.81701415925393, -8.665327778342736, -7.3868098829003275, -4.401647923547139]
FAIL
======================================================================
FAIL: testTurbulenceRandomSeed (__main__.TestTurbulenceRandomSeed)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\source\jsbsim\tests\TestTurbulenceRandomSeed.py", line 32, in testTurbulenceRandomSeed
self.assertAlmostEqual(wn1[i], wn2[i], delta=1E-8)
AssertionError: 0.047354487544379376 != -2.1842398652974198 within 1e-08 delta (2.231594352841799 difference)
----------------------------------------------------------------------
Ran 1 test in 0.021s
FAILED (failures=1)
(base) C:\source\jsbsim\tests>
ANOTHER RUN:
(base) C:\source\jsbsim\tests>python .\TestTurbulenceRandomSeed.py
testTurbulenceRandomSeed (__main__.TestTurbulenceRandomSeed) ...
[0.047354487544379376, 3.460354080430806, 4.636633740213811, -0.4374137958205208, -0.06377215234821773, -1.5456835409458707, -3.4411298921520492, -6.057946773083572, -5.709936753084856, -5.628235304626574, -6.480877261842454, -5.210906810642065, -2.2376367571469964]
[-2.1842398652974198, 1.2417689243655539, 2.426390607718044, -2.652099515681857, -2.2717034356215526, -3.750613502555642, -5.643749459388144, -8.259702814605022, -7.904921571924209, -7.81701415925393, -8.665327778342736, -7.3868098829003275, -4.401647923547139]
FAIL
======================================================================
FAIL: testTurbulenceRandomSeed (__main__.TestTurbulenceRandomSeed)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\source\jsbsim\tests\TestTurbulenceRandomSeed.py", line 32, in testTurbulenceRandomSeed
self.assertAlmostEqual(wn1[i], wn2[i], delta=1E-8)
AssertionError: 0.047354487544379376 != -2.1842398652974198 within 1e-08 delta (2.231594352841799 difference)
----------------------------------------------------------------------
Ran 1 test in 0.023s
FAILED (failures=1)
(base) C:\source\jsbsim\tests> My guess is that given we're creating 2 instances of the FDM within the same process there is some state in the random number generation process that isn't being reset completely. /// Specify a new seed and reinitialize the random generation process.
void seed(unsigned int value) {
generator.seed(value);
uniform_random.reset();
normal_random.reset();
} |
Beta Was this translation helpful? Give feedback.
-
Hmm, ignore my comment/guess about it being a single process issue. Here is a simple single process example where the random number generation does work as expected. #include <iostream>
#include <random>
int main()
{
//std::mt19937 gen;
std::default_random_engine gen;
std::normal_distribution<double> normal_random;
for(int i=0; i<3; i++)
{
// Seed the engine with an unsigned int
gen.seed(1);
std::cout << "after seed by 1: " << gen() << '\n';
normal_random.reset();
for(int j=0; j<3; j++)
{
std::cout << normal_random(gen) << '\n';
}
}
} after seed by 1: 16807
0.207042
1.61506
0.118772
after seed by 1: 16807
0.207042
1.61506
0.118772
after seed by 1: 16807
0.207042
1.61506
0.118772 |
Beta Was this translation helpful? Give feedback.
-
@Dobid I've submitted a pull request - #1253 to allow a user to specify a separate random seed for wind turbulence. |
Beta Was this translation helpful? Give feedback.
-
I've been able to reproduce the issue with 2 instances of void run(char* script)
{
JSBSim::FGFDMExec* FDMExec = new JSBSim::FGFDMExec();
FDMExec->LoadScript(SGPath(script));
FDMExec->RunIC();
bool result = true;
while (result) result = FDMExec->Run();
delete FDMExec;
}
int main(int argc, char* argv[])
{
run("scripts\\random1.xml");
run("scripts\\random2.xml");
return 0;
} With scripts <run start="0" end="15" dt="0.008333">
<property value="1"> simulation/randomseed </property>
<property value="3"> atmosphere/randomseed </property>
<property value="3"> atmosphere/turb-type </property>
<property value="75"> atmosphere/turbulence/milspec/windspeed_at_20ft_AGL-fps </property>
<property value="3"> atmosphere/turbulence/milspec/severity </property> And script <run start="0" end="15" dt="0.008333">
<property value="2"> simulation/randomseed </property>
<property value="3"> atmosphere/randomseed </property>
<property value="3"> atmosphere/turb-type </property>
<property value="75"> atmosphere/turbulence/milspec/windspeed_at_20ft_AGL-fps </property>
<property value="3"> atmosphere/turbulence/milspec/severity </property>
|
Beta Was this translation helpful? Give feedback.
-
@bcoconni it looks like we've been bitten by jsbsim/src/models/atmosphere/FGWinds.cpp Lines 286 to 293 in 614a5b7 |
Beta Was this translation helpful? Give feedback.
-
I've pushed some more commits to PR - #1253 to fix the static local variables issue and also added the python test program that I couldn't originally get to work due to this static local variable issue. |
Beta Was this translation helpful? Give feedback.
-
@Dobid see pull request - #1254 which now also allows a user to specify individual randomseeds for individual sensors. So you have control over the randomseed for turbulence and sensors now. |
Beta Was this translation helpful? Give feedback.
At first glance, I'll double-check later, my guess is that when the turbulence is switched off, the
Turbulence()
method on line 146 is no longer called, but the vectorvTurbulenceNED
isn't zeroed out, rather it's left at it's last value it was assigned by the last call toTurbulence()
. In which case it'll be a simple fix.jsbsim/src/models/atmosphere/FGWinds.cpp
Lines 141 to 149 in e1ea4e8