TagPOMDP(; kwargs...)

Returns a TagPOMDP <: POMDP{TagState, Int, Int}. Default values are from the original paper: Pineau, Joelle et al. “Point-based value iteration: An anytime algorithm for POMDPs.” IJCAI (2003).

The main difference in this implementation is the use of only 1 terminal state and an opponent transition function that aims to keep the probability of moving away to the specified value if there is a valid action (versus allowing the action and thus increasing the probability of remaining in place). To use the transition function from the original implementation, pass orig_transition_fcn = true.

Keywords

• map_str::String: String representing the map, 'x' for walls, 'o' for open space. Default is the standard map from the original paper.

  Default: """ xxxxxoooxx

  xxxxxoooxx

  xxxxxoooxx

  oooooooooo

  oooooooooo"""

• tag_reward::Float64: Reward for the agent tagging the opponent, default = +10.0

• tag_penalty::Float64: Reward for the agent using the tag action and not being in the same grid cell as the opponent, default = -10.0

• step_penalty::Float64: Reward for each movement action, default = -1.0

• discount_factor::Float64: Discount factor, default = 0.95

• move_away_probability::Float64: Probability associated with the opponent srategy. This probability is the chance it moves away, default = 0.8

• transition_option::Symbol: Option for the transition function. Options are :orig and :modified. Default is :modified.

render(pomdp::TagPOMDP, step::NamedTuple; pre_act_text::String="")

Render a TagPOMDP step as a plot. If the step contains a belief, the belief will be plotted using a color gradient of green for the belief of the target position and belief over the robot position will be plotted as an orange robot with a faded robot representing smaller belief. If the step contains a state, the robot and target will be plotted in their respective positions. If the step contains an action, the action will be plotted in the bottom center of the plot. pre_act_text can be used to add text before the action text.

• pomdp::TagPOMDP: The TagPOMDP to render
• step::NamedTuple: Step step to render with fields b, s, and a
• pre_act_text::String: Text to add before the action text

TagPOMDP <: POMDP{TagState, Int, Int}

POMDP type for the Tag POMDP.

Fields

• mg::MetaDiGraph: metagraph representing the map
• dist_matrix::Matrix{Float64}: distance matrix for the metagraph
• tag_reward::Float64: reward for the agent tagging the opponent
• tag_penalty::Float64: reward for the agent using the tag action and not being in the same grid cell as the opponent
• step_penalty::Float64: reward for each movement action (negative = penalty)
• discount_factor::Float64: discount factor
• move_away_probability::Float64: probability associated with the opponent srategy. This probability is the chance it moves away versus stays in place.
• transition_option::Symbol: option for the transition function

TagState

Represents the state in a TagPOMDP problem.

Fields

• r_pos::Int: vertex of graph representing the position of the robot
• t_pos::Int: vertex of graph representing the position of the target

list_actions(pomdp::TagPOMDP)

Prints a list of actions and their symbol (name).

create_metagraph_from_map(map_str::String)

Returns a MetaDiGraph representing the map. 'x' for walls, 'o' for open space.

Properties of the graph:

• :nrows: number of rows in the map
• :ncols: number of columns in the map
• :num_grid_pos: number of open spaces in the map
• :node_mapping: dictionary mapping (i, j) position in the map to node number
• :node_pos_mapping: dictionary mapping node number to (i, j) position in the map

Properties of the edges:

• :action: action associated with the edge (e.g. :north, :south, :east, :west)

Example mat_str for the original TagPOMDP (the one in the original paper)

xxxxxoooxx

xxxxxoooxx

xxxxxoooxx

oooooooooo

oooooooooo

map_str_from_metagraph(pomdp::TagPOMDP)

Returns a string representing the map. 'x' for walls, 'o' for open space. Uses the node_mapping property of the metagraph to determine which nodes are open spaces.

state_from_index(pomdp::TagPOMDP, si::Int)

Return the state corresponding to the given index.

modified_transition(pomdp::TagPOMDP, s::TagState, a::Int)

Modified transition function for the TagPOMDP. This transition is similar to the original paper but differs with how it redistributes the probabilities of actions where the opponent would hit a wall and stay in place. The original implementation redistributed those probabilities to the stay in place state. This implementation keeps the probability of moving away from the agent at the defined threshold if there is a valid movement option (away and not into a wall). The movement of the agent is deterministic in the direction of the action.

orig_transition(pomdp::TagPOMDP, s::TagState, a::Int)

Alternative transition function for the TagPOMDP. This transition is mimics the original paper. The move away probability is split equally between north/south and east/west pairs. If a move results in moving "away" from the agent while ignoring walls, that move is still considered valid. This results in the the agent hitting a wall and staying in place...thus increasing the probability of staying in place.

move_direction(pomdp::TagPOMDP, v::Int, a::Int)

Move the robot in the direction of the action. Finds the neighbors of the current node and checks if the action is valid (edge with corresponding action exists). If so, returns the node index of the valid action.