Bnb/obs scaling

[bnb32]

Added parameter for scaling obs data in obs rasterizer.
Small refactor of arguments in ExoDataHandler.
Enabled aggregation of 3d data in ExoRasterizer

[grantbuster]

Minor comment but generally i'm a bit nervous about this. Have we figured out why the MADIS obs are negatively biased? How are you determining a scale factor?

[grantbuster]

why the extra L?

[bnb32]

oops, bad string replace

[grantbuster]

recommend adding docstring describing output shapes/dimensions here.

[bnb32]

Minor comment but generally i'm a bit nervous about this. Have we figured out why the MADIS obs are negatively biased? How are you determining a scale factor?

Don't have a complete explanation for the bias of MADIS vs WTK but that's how I'm determining the scale factor (average of wtk(at 10m) / madis over multiple sites). I haven't been using this right now but I think something like this makes sense so that the relationship between near surface and hub height learned by the model is preserved during inference.

[grantbuster]

Minor comment but generally i'm a bit nervous about this. Have we figured out why the MADIS obs are negatively biased? How are you determining a scale factor?

Don't have a complete explanation for the bias of MADIS vs WTK but that's how I'm determining the scale factor (average of wtk(at 10m) / madis over multiple sites). I haven't been using this right now but I think something like this makes sense so that the relationship between near surface and hub height learned by the model is preserved during inference.

Well the model will preserve those relationships whether we like it or not, just a matter of bias at surface and consequently at hub height, right? The model should learn to respect and assimilate the obs we give it, but if we don't like the obs then the question is what do we do? Does the WTK have tons of bias at 10m? This fudge factor makes sense but is it scientifically defensible? Would the fudge factor really be constant for all locations and times?

[bnb32]

Minor comment but generally i'm a bit nervous about this. Have we figured out why the MADIS obs are negatively biased? How are you determining a scale factor?

Don't have a complete explanation for the bias of MADIS vs WTK but that's how I'm determining the scale factor (average of wtk(at 10m) / madis over multiple sites). I haven't been using this right now but I think something like this makes sense so that the relationship between near surface and hub height learned by the model is preserved during inference.

Well the model will preserve those relationships whether we like it or not, just a matter of bias at surface and consequently at hub height, right? The model should learn to respect and assimilate the obs we give it, but if we don't like the obs then the question is what do we do? Does the WTK have tons of bias at 10m? This fudge factor makes sense but is it scientifically defensible? Would the fudge factor really be constant for all locations and times?

The value of near surface ws vs hub height ws is part of that relationship. Either way though, this is just following your suggestion to scale madis to wtk that came up in the discussion on what is the true measurement height for madis. Whether wtk is biased at 10m is unknown, since the comparison would be to madis... Of course the global scale factor is the simplest iteration and it would be better to apply some location / time dependent correction, but that isn't feasible right now.

[grantbuster]

Minor comment but generally i'm a bit nervous about this. Have we figured out why the MADIS obs are negatively biased? How are you determining a scale factor?

Don't have a complete explanation for the bias of MADIS vs WTK but that's how I'm determining the scale factor (average of wtk(at 10m) / madis over multiple sites). I haven't been using this right now but I think something like this makes sense so that the relationship between near surface and hub height learned by the model is preserved during inference.

Well the model will preserve those relationships whether we like it or not, just a matter of bias at surface and consequently at hub height, right? The model should learn to respect and assimilate the obs we give it, but if we don't like the obs then the question is what do we do? Does the WTK have tons of bias at 10m? This fudge factor makes sense but is it scientifically defensible? Would the fudge factor really be constant for all locations and times?

The value of near surface ws vs hub height ws is part of that relationship. Either way though, this is just following your suggestion to scale madis to wtk that came up in the discussion on what is the true measurement height for madis. Whether wtk is biased at 10m is unknown, since the comparison would be to madis... Of course the global scale factor is the simplest iteration and it would be better to apply some location / time dependent correction, but that isn't feasible right now.

I hear ya i'm just realizing how hacky this is. Do you know what scale factor you'd use and what kind of delta h that would imply using the power law? If you use this in published data or a manuscript just be really careful, will be hard to defend i think.

[bnb32]

Minor comment but generally i'm a bit nervous about this. Have we figured out why the MADIS obs are negatively biased? How are you determining a scale factor?

Don't have a complete explanation for the bias of MADIS vs WTK but that's how I'm determining the scale factor (average of wtk(at 10m) / madis over multiple sites). I haven't been using this right now but I think something like this makes sense so that the relationship between near surface and hub height learned by the model is preserved during inference.

Well the model will preserve those relationships whether we like it or not, just a matter of bias at surface and consequently at hub height, right? The model should learn to respect and assimilate the obs we give it, but if we don't like the obs then the question is what do we do? Does the WTK have tons of bias at 10m? This fudge factor makes sense but is it scientifically defensible? Would the fudge factor really be constant for all locations and times?

The value of near surface ws vs hub height ws is part of that relationship. Either way though, this is just following your suggestion to scale madis to wtk that came up in the discussion on what is the true measurement height for madis. Whether wtk is biased at 10m is unknown, since the comparison would be to madis... Of course the global scale factor is the simplest iteration and it would be better to apply some location / time dependent correction, but that isn't feasible right now.

I hear ya i'm just realizing how hacky this is. Do you know what scale factor you'd use and what kind of delta h that would imply using the power law? If you use this in published data or a manuscript just be really careful, will be hard to defend i think.

Yeah it's a fair point. Scale factor would be around 1.1 which puts the delta h at ~5 meters.