Computational Rationalization: The Inverse Equilibrium Problem

Abstract:Modeling the purposeful behavior of imperfect agents from a small number of observations is a challenging task. When restricted to the single-agent decision-theoretic setting, inverse optimal control techniques assume that observed behavior is an approximately optimal solution to an unknown decision problem. These techniques learn a utility function that explains the example behavior and can then be used to accurately predict or imitate future behavior in similar observed or unobserved situations. In this work, we consider similar tasks in competitive and cooperative multi-agent domains. Here, unlike single-agent settings, a player cannot myopically maximize its reward — it must speculate on how the other agents may act to influence the game’s outcome. Employing the game-theoretic notion of regret and the principle of maximum entropy, we introduce a technique for predicting and generalizing behavior, as well as recovering a reward function in these domains.

Moslem Kazemi: working on perception and force guided manipulation. Working with Nancy Pollard and Drew Bagnell.
Kris Katani: working with Martial Hebert and Drew Bagnell on activity prediction.
Paul Vernaza: working with Drew Bagnell on compressed information space reasoning.

Stéphane Ross Geoffrey J. Gordon J. Andrew Bagnell,
Carnegie Mellon University

To Appear in Proceedings of the 14th International Conference on
Artificial Intelligence and Statistics (AISTATS), 2011

Link to Paper

Abstract: Sequential prediction problems such as imitation learning, where
future observations depend on previous predictions (actions), violate the
common i.i.d. assumptions made in statistical learning. This leads to poor
performance in theory and often in practice. Some recent approaches
provide stronger guarantees in this setting, but remain somewhat
unsatisfactory as they train either non-stationary or stochastic policies
and require a large number of iterations. In this paper, we propose a new
iterative algorithm, which trains a stationary deterministic policy, that
can be seen as a no regret algorithm in an online learning setting. We
show that any such no regret algorithm, combined with additional reduction
assumptions, must find a policy with good performance under the
distribution of observations it induces in such sequential settings. We
demonstrate that this new approach outperforms previous approaches on two challenging imitation learning problems and a benchmark sequence labeling
problem.

Brian D. Ziebart, J. Andrew Bagnell, Anind K. Dey
Carnegie Mellon University
To appear at International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2011).

Link to Paper

Motivated by a machine learning perspective|that game theoretic
equilibria constraints should serve as guidelines for
predicting agents’ strategies, we introduce maximum causal
entropy correlated equilibria (MCECE), a novel solution
concept for general-sum Markov games. In line with this
perspective, a MCECE strategy prole is a uniquely-dened
joint probability distribution over actions for each game
state that minimizes the worst-case prediction of agents’ actions
under log-loss. Equivalently, it maximizes the worstcase
growth rate for gambling on the sequences of agents’
joint actions under uniform odds. We present a convex optimization
technique for obtaining MCECE strategy proles
that resembles value iteration in nite-horizon games. We
assess the predictive benets of our approach by predicting
the strategies generated by previously proposed correlated
equilibria solution concepts, and compare against those previous
approaches on that same prediction task.

Dodge has released an amusing new commercial referencing self-driving cars and other robotics advances. Well, those of us on Team Robot take it as a compliment.

3-D Scene Analysis via Sequenced Predictions over Points and Regions
Xuehan Xiong, Daniel Munoz, J. Andrew Bagnell, Martial Hebert
To appear: ICRA 2011.
Preprint (pdf)

We address the problem of understanding scenes from 3-D laser scans via per-point assignment of semantic labels. In order to mitigate the difficulties of using a graphical model for modeling the contextual relationships among the 3-D points, we instead propose a multi-stage inference procedure to capture these relationships. More specifically, we train this procedure to use point cloud statistics and learn relational information (e.g., tree-trunks are below vegetation) over fine (point-wise) and coarse (region-wise) scales. We evaluate our approach on three different datasets, that were obtained from different sensors, and demonstrate improved performance.

Andy, our ARM (Autonomous Robot Manipulation) wishes you a merry christmas! Mihail Pivtoraiko, our motion planning for manipulation expert, demonstrates Andy’s current dexterity.

Stacked Hierarchical Labeling
Daniel Munoz, J. Andrew Bagnell, Martial Hebert
To appear: ECCV 2010.
Preprint (pdf)

In this work we propose a hierarchical approach for labeling semantic objects and regions in scenes. Our approach is reminiscent of early vision literature in that we use a decomposition of the image in order to encode relational and spatial information. In contrast to much existing work on structured prediction for scene understanding, we bypass a global probabilistic model and instead directly train a hierarchical inference procedure inspired by the message passing mechanics of some approximate inference procedures in graphical models. This approach mitigates both the theoretical and empirical difficulties of learning probabilistic models when exact inference is intractable. In particular, we draw from recent work in machine learning and break the complex inference process into a hierarchical series of simple machine learning subproblems. Each subproblem in the hierarchy is designed to capture the image and contextual statistics in the scene. This hierarchy spans coarse-to-fine regions and explicitly models the mixtures of semantic labels that may be present due to imperfect segmentation. To avoid cascading of errors and overfitting, we train the learning problems in sequence to ensure robustness to likely errors earlier in the inference sequence and leverage the stacking approach developed by Cohen et al.

Modeling Interaction via the Principle of Maximum Causal Entropy
ICML runner up for best student paper by Brian Ziebart, J. Andrew Bagnell, and Anind Dey.

@inproceedings{bziebart-maxcausalent,
author = {Brian D. Ziebart and J. Andrew Bagnell
and Anind K. Dey},
title = {Modeling Interaction via the Principle of Maximum Causal Entropy},
year = {2010},
booktitle = {International Conference on Machine Learning}
}

The principle of maximum entropy provides a powerful framework for statistical models of joint, conditional, and marginal distributions. However, there are many important distributions with elements of interaction and feedback where its applicability has not been established. This work presents the principle of maximum causal entropy — an approach based on causally conditioned probabilities that can appropriately model the availability and influence of sequentially revealed side information. Using this principle, we derive Maximum Causal Entropy Influence Diagrams, a new probabilistic graphical framework for modeling decision making in settings with latent information, sequential interaction, and feedback. We describe the theoretical advantages of this model and demonstrate its applicability for statistically framing inverse optimal control and decision prediction tasks.

Reinforcement Planning: RL for Optimal Planners
Matt Zucker and J. Andrew Bagnell

PDF

Search based planners such as A* and Dijkstra’s algorithm are proven methods for guiding today’s robotic systems. Although such planners are typically based upon a coarse approximation of reality, they are nonetheless valuable due to their ability to reason about the future, and to generalize to previously unseen scenarios. However, encoding the desired behavior of a system into the underlying cost function used by the planner can be a tedious and error-prone task. We introduce Reinforcement Planning, which extends gradient based reinforcement learning algorithms to automatically learn useful cost functions for optimal planners. Reinforcement Planning presents several advantages over other learning applications involving planners in that it is not limited by the expertise of a human demonstrator, and that it also recognizes that the domain of the planner is a simplified model of the world. We demonstrate the effectiveness of our method in learning to solve a noisy physical simulation of the well-known “marble maze” toy.