Common Representations for Perception, Prediction, and Planning

ABOUT THE PROJECT

At a glance

In this work, we will combine the classic tiered and end-to-end approach to perception, prediction, and planning.
The key idea of the proposed work is to define a compact, interpretable, differentiable representation of the perceived and state of the world that can be directly estimated from sensor signals using a supervised loss but also can be used as input to prediction and planning.  By ensuring that the representation is compact, we would like to extract only the essential information from perception needed for effective prediction and planning. By adopting an interpretable representation, we can provide direct training signal when estimating the latent state. By ensuring that the representation is differentiable, we would like to be able to compose the perception, prediction, and planning stages into a single differentiable function that can be optimized end-to-end in conjunction with any intermediate training signal.