Policy formulation for parallel run

[Etwari3]

Hello,

I tried to construct a functionpolicy as follows

struct DesignMaint <: Policy end
POMDPTools.BeliefUpdaters.updater(::DesignMaint) = PreviousObservationUpdater()

function action(::DesignMaint, b::Union{Nothing, Int64})
    if isnothing(b) || b == 1 # not initial or unknown (null) state
        return 1
    elseif b == 2 # second state
        return 2
    elseif b == 3 # third state
        return 2
    elseif b == 4 # fourth state
        return 3
    else # last state
        return 4
    end
end

POMDPs.action(::DesignMaint, b::Int64) = b
POMDPs.action(p::DesignMaint, b::Missing) = 1

xyz = DesignMaint()

q = [] 
push!(q, Sim(pomdp, xyz, max_steps=32, rng=MersenneTwister(1), metadata=Dict(:policy=>3)))
data = run_parallel(q)

I got the following error message:

MethodError: no method matching action(::DesignMaint, ::POMDPModels.DiscreteDistribution{Vector{Float64}})

How do you define the actions for the different states?

Thank you

[zsunberg]

Hi @Etwari3 ,

You are doing it correctly if you want the policy to be a function of only the previous observation. There is just one thing that is likely confusing you: on the first step, since there is no observation available, the PreviousObservationUpdater returns the initial belief.

You can fix this by providing an initial observation as the initial_belief argument to Sim. Since the simulator can't sample the initial state from this "belief", you will also have to provide an initial state. So the full call to create the Sim would look something like

initial_obs = 1
initial_state = 2
Sim(pomdp, xyz, PreviousObservationUpdater(), initial_obs, initial_state, max_steps=32, rng=MersenneTwister(1), metadata=Dict(:policy=>3))

[Etwari3]

Thank you, zsunberg.