Create a POMDP with time-dependent parameters

[lumiereljy]

Hello, I found this package in Julia and I am working on solving a POMDP problem with time-dependent parameters. I am wondering what is the best way to implement time-dependent transition probability, observation probability and rewards, as I didn't find an example on the website. For example, if I want to make my transition model, observation model, and reward model depend on time and solve it for T=10 using incremental pruning algorithm, how should I go about the following tiger example?

  using POMDPModels: TabularPOMDP
  using IncrementalPruning
  T = zeros(2,3,2)
  T[:,:,1] = [1. 0.5 0.5; 
              0. 0.5 0.5]
  T[:,:,2] = [0. 0.5 0.5; 
              1. 0.5 0.5]
  
  O = zeros(2,3,2)
  O[:,:,1] = [0.85 0.5 0.5; 
              0.15 0.5 0.5]
  O[:,:,2] = [0.15 0.5 0.5; 
              0.85 0.5 0.5]
  
  R = [-1. -100. 10.; 
       -1. 10. -100.]
  
  pomdp = TabularPOMDP(T, R, O, 0.95)
  solver = PruneSolver() # set the solver
  policy = solve(solver, pomdp) # solve the POMDP  

Also, I am wondering if I can define my problem with a different order of events. Specifically, I take an action and make an observation, and then the state transition. So my observation model is Z(o|s,a), transition model is P(s'|s,a,o), and my reward model is r(s,a,o). I saw it is possible to define our own belief updater, but I am still not sure how this order of event can be handled in the Julia POMDP package. Thanks!

[mykelk]

Perhaps you can have time as a state variable?

[zsunberg]

To expand on @mykelk 's suggestion, if you include time in the state, you can express time dependent transitions, observations, and rewards (essentially the transition matrices will have a sparse block structure). Most current POMDPs.jl solvers will not take advantage of this structure*, however, you can solve surprisingly big problems with SARSOP. That is probably the easiest way forward.

*Note: there has been an attempt to give time information to solvers through https://github.com/JuliaPOMDP/FiniteHorizonPOMDPs.jl, but there are some rough edges there.