StateSpace Module Hurricane Case Study ~~WIP~~ Ready for Review #763
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
View / edit / reply to this conversation on ReviewNB fonnesbeck commented on 2025-01-05T01:33:47Z Not sure if its my setup, but there are some stray $ characters in the intro. Dekermanjian commented on 2025-01-05T11:37:14Z Hi @fonnesneck! Thank you for taking the time to review this notebook. On ReviewNB it seems that the double $ that should generate an equation on its own line doesn't render, but it does on the PyMC examples page. That is why there are stray $ characters. |
|
View / edit / reply to this conversation on ReviewNB fonnesbeck commented on 2025-01-05T01:33:48Z We don't currently use plotly in any of the other examples, so need to update the requirements. Dekermanjian commented on 2025-01-05T11:41:24Z Yes, you are correct. I will move the comment on line 6 to the very top of this cell. I have Plotly included in the extra_installs in the myst.md file. Again this does not render in ReviewNB but it does on the PyMC examples website. You can see it if you visit the documentation preview https://pymcio--763.org.readthedocs.build/projects/examples/en/763/case_studies/ssm_hurricane_tracking.html |
|
View / edit / reply to this conversation on ReviewNB fonnesbeck commented on 2025-01-05T01:33:49Z spelling: "originate"
Also, maybe worth adding a sentence describing what defines an origination point. Dekermanjian commented on 2025-01-05T11:43:24Z Thank you for catching that! Yes, I agree with you. I will add some more context on what constitutes an origination point. |
|
Hi @fonnesneck! Thank you for taking the time to review this notebook. On ReviewNB it seems that the double $ that should generate an equation on its own line doesn't render, but it does on the PyMC examples page. That is why there are stray $ characters. View entire conversation on ReviewNB |
|
Yes, you are correct. I will move the comment on line 6 to the very top of this cell. I have Plotly included in the extra_installs in the myst.md file. Again this does not render in ReviewNB but it does on the PyMC examples website. You can see it if you visit the documentation preview https://pymcio--763.org.readthedocs.build/projects/examples/en/763/case_studies/ssm_hurricane_tracking.html View entire conversation on ReviewNB |
|
Thank you for catching that! Yes, I agree with you. I will add some more context on what constitutes an origination point. View entire conversation on ReviewNB |
2. Improved 24-hour ahead forecasts of simplest model 3. Added a section and model with how to add exogenous variables to the model
2. cleaned up typos and figure sizes/zoom
2. Added a non-linear B-Splines section
2. Added closing remarks section 3. Updated references bib file
Dekermanjian
changed the title
Dekermanjian
changed the title
Dekermanjian
changed the title
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:25Z Some nitpicks on the latex:
Dekermanjian commented on 2025-04-03T23:34:44Z Totally agree with all of the above. I will clean up the equations with latex and make a note on the Joseph form of the filtered covariance equation. I will also look up what the convention is for the t subscript and replace it everywhere in the notebook that occurs. Dekermanjian commented on 2025-04-06T21:46:55Z Hey Jesse, I think maybe I misunderstood one of your comments. The letter "t" itself isn't the issue in your comment but the fact that I have it subscript t within the text under the definition column of the above tables. Is that correct? jessegrabowski commented on 2025-06-14T12:26:01Z Yes that's correct
|
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:26Z Line #3. Generates a 95% CI ellipse via a chi-square multivariate normal approximation Nit: we use numpy docstring formatting in the pymc codebase, might as well use it here too Dekermanjian commented on 2025-04-03T23:35:39Z Thank you for making me aware of the numpy formatting. I will make the change everywhere this applies. |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:27Z Line #20. ): # First handful of ellipses are huge because of little data in the iterative nature of the Kalman filter
I'm reviewing this sequentially so haven't gotten to the model yet, but this might also be because the initial P0 matrix is too diffuse. Dekermanjian commented on 2025-04-03T23:37:51Z I didn't think of that. I will make a note of it in the text and I will play around with different P0 initializations. |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:27Z Line #21. # Hack that should be fixed in StateSpace soon Was this ever fixed? I should do it immediately if not. Dekermanjian commented on 2025-04-03T23:39:29Z I will test it out with the latest versions of pymc + pymc-extras and report back if it isn't. I recall this being fixed, though. Dekermanjian commented on 2025-04-06T14:14:22Z Hey Jesse, the GitHub issue for this issue is still open pymc-devs/pymc-extras#424 and I can confirm that the hack is still required on the latest version of pymc-extras. |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:28Z I guess the points are colored by category? Is it possible to add a legend for that? (This plot is super slick btw) Dekermanjian commented on 2025-04-03T23:41:24Z Yes, that is correct. I agree that a legend should be included. I will add one! |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:29Z In this example we assume constant acceleration (In order to keep our system of equations linear) I don't think constant acceleration is required for linearity in this setup -- you could have a stochastic innovation on the acceleration term that allows it to change over time, and indeed you do! I guess this is a technical point required to discretize the Newtonian ODEs, but it's also not technically true in your model. Make of that what you will.
Our states are the following components derived from the Newtonian equations of motion Not required, but you could show how these matrices arise from integrating the ODEs p_dot(t) = v(t), v_dot(t) = a(t), a_dot(t) = 0 over an interval delta t.
In this example we fix our observation/measurement error to a small value (0.1) reflecting our confidence in the measurements. The latex matrix H has 0.5 on the diagonal, not 0.1. Maybe also make a note about the units of the data, and what the number 0.1 actually represents in terms of measurement error.
variane (diagonals) covariance (off-diagonals) of the Newtonian equations Typo: variance
Dekermanjian commented on 2025-04-04T00:04:13Z Yes, you are correct. I think I need to clarify that with this model we aren't able to model any angular acceleration whenever the hurricane makes maneuvers so we aim to capture turns with the stochastic innovation on the acceleration term.
I will put in the ODE integrations. I think this will be good for completeness. Of course, I will fix the mismatch between the latex and the text and the typo. |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:30Z Line #13. simple_idata = pm.sample(nuts_sampler="numpyro", target_accept=0.95) Try with nutpie, you might be able to get away with not increasing target accept. Dekermanjian commented on 2025-04-04T00:05:03Z Yes, I will give it a go. I believe at the time the Jax backends were not implemented yet. I will give that a try too. |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:31Z It uses data from the entire time series in the sense that your estimated parameters already saw the entire future.
But all it does is re-write your model so that Dekermanjian commented on 2025-04-04T00:06:04Z Thank you so much for clarifying this for me! jessegrabowski commented on 2025-04-04T03:46:16Z It happens here if you're interested:
|
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:32Z The addition to the R matrix imply that the exogenous parameters do not exhibit any process noise. Nit: I wouldn't describe the stochastic innovations in the transition equation as "noise", since they're in some sense "real", as updates to the values of the latent states of the system. I think of "noise" as arising from some flaw in our ability to observe, e.g. from the measurement errors. Dekermanjian commented on 2025-04-04T00:10:53Z I will go back and reword how the stochastic innovations are described in the text. |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:32Z Maybe a forest plot here? It's a bit more compact.
You might also have to play with the prior variances on these. N(0, 10) is pretty aggressive, and you can see that it's not like an OLS regression where the priors quickly wash out. All these posteriors have pretty wide uncertainties. I'm not familiar enough with the science to know if an effect size of -20 * wind pressure is realistic, but your priors but non-zero probability mass on that.
Dekermanjian commented on 2025-04-04T00:13:43Z I agree, that priors are poorly specifies. I will go back and pick more sensible priors and I will condense the output with a forest plot. |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:33Z To keep things simple, we are going to define a constant number of knots that are equal in both variables (same #knots for both longitude and latitude) and we are going to place the knots using a quantile function over the variable space. You can see the knots plotted out below for each variable. Write "same number of knots" -- currently looks like a hashtag
Dekermanjian commented on 2025-04-04T00:14:56Z Ah yes, I should've been more careful. I will make the change. |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-04-03T14:05:34Z Line #201. "exog_dims": [ Could use an iterator + f-string for these names; they're a bit verbose at the moment Dekermanjian commented on 2025-04-04T00:16:14Z Yes, totally agree. I will make the change. |
|
Hey @jessegrabowski, I re-ran the notebook but something is not working correctly with the exogenous forecasts. While the previous time dimension error message is gone, the output of the forecasts don't look correct. I am trying to write a very simple example of a custom state space model to see if I can debug/figure out what is happening. |
|
I'm guessing you're running into this @Dekermanjian, aren't you? |
|
Hey @AlexAndorra, yes I believe that is the same issue. In my case the output is much worse, however, probably because my ‘k_endog=2’. I am going to try to track down the issue. |
|
Yeah that sounds about right. It'd be hugely helpful if you managed to track this down! LMK if there anything I can do to help |
|
Yes that's correct
View entire conversation on ReviewNB |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:49Z Line #4. from typing import Optional This shouldn't be needed (use |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:49Z Line #7. covariance : ndarray This should be the same level of indentation as the |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:50Z Line #12. data : DataFrame Same here, indentation |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:51Z Line #5. exogenous_data_name: Optional[str] = None,
|
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:51Z Line #14. ssm_model : PyMCStateSpace Indentation |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:52Z Line #39. for start in np.arange(0, inference_data.constant_data.time.shape[0], periods):
Extract |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:52Z Line #47. exogenous_data_name: padded_exogenous_data.slice(start, periods).to_numpy() Actually the only thing that changes between the 3 cases seems to be the |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:53Z Line #1. class SimpleSSM(PyMCStateSpace):
Maybe a more descriptive name, like NewtonaianSSM or KinematicSSM? |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:54Z Line #10. mode="JAX",
I want to depreciate |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:54Z Line #11. save_kalman_filter_outputs_in_idata=True,
Instead of setting this flag, you can get the predicted states using Dekermanjian commented on 2025-06-14T16:37:01Z Hey @jessegrabowski, is there a way I can get the predicted covariance from jessegrabowski commented on 2025-06-14T18:01:14Z From the samples -- |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:55Z The one parameter comment is quite important. Maybe a comment about how much work the model design -- e.g. the kinematic equations -- are doing here? People might be used to parametric models where the the structure is quite loose, but the model can flexibly learn stuff. Here we're in the opposite regime -- we impose structure on the data, so we need fewer parameters. But you better be sure the structure is right! |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:56Z Use |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:56Z Maybe also add a comment in the code explaining where and why this is happening |
|
View / edit / reply to this conversation on ReviewNB jessegrabowski commented on 2025-06-14T13:40:57Z The progress bars aren't filled up -- did you cancel sampling on your local machine? Dekermanjian commented on 2025-06-14T16:18:41Z Hey Jesse, no I didn't cancel it. I've had 2 weird things happen using the nutpie progress bar. First, is the above where the progress bar visual freezes but it continues to sample. Second, is specific to when I ssh into a linux machine using vscode on windows the nutpie progress bar (only nutpie) will print out a bunch of empty lines below the progress bar as it samples. jessegrabowski commented on 2025-06-14T18:04:19Z Gross, ok. As long as it's not real it doesn't matter |
|
Posted a final round of suggestions. It's mostly nitpicks, so please feel free to ignore many of them. Let's get this merged, it's really amazing! |
|
Hey Jesse, no I didn't cancel it. I've had 2 weird things happen using the nutpie progress bar. First, is the above where the progress bar visual freezes but it continues to sample. Second, is specific to when I ssh into a linux machine using vscode on windows the nutpie progress bar (only nutpie) will print out a bunch of empty lines below the progress bar as it samples. View entire conversation on ReviewNB |
|
Thank you, Jesse for going through this again. I will make the suggested changes and re-push! |
|
Hey @jessegrabowski, is there a way I can get the predicted covariance from View entire conversation on ReviewNB |
|
From the samples -- View entire conversation on ReviewNB |
|
Gross, ok. As long as it's not real it doesn't matter View entire conversation on ReviewNB |
2. updated generate forecast utility 3. updated how mode=JAX is specified 4. renamed simple model to Newtonian model 5. added explanations to exogenous data duplicating and mirroring 6. added note on how the newtonian model differs from typical parameteric models
|
Hey @OriolAbril, sorry @ you out of the blue but I know you are an expert with how the |
Case Study to showcase the use of the State Space Module in
pymc-extrasThis relates to #715
In this case study I have done the following:
I still wish to add the following:
@jessegrabowski Once I have completed this, would you be willing to be a reviewer for this case study?
EDIT:
I am thinking about skipping the VARMAX model section as this is starting to get a bit long, in my opinion.
I am open to feedback from others on whether to skip or add this section.
EDIT 2:
This is now ready for review.
EDIT 3:
I need some help with referencing a notebook in a repo. Specifically, this
I've read through the Jupyter Style documentation but it is still not apparent to me how to add a reference to a notebook in a repo that doesn't have a published sphinx documentation page.
📚 Documentation preview 📚: https://pymc-examples--763.org.readthedocs.build/en/763/