Wasserstein Auto-encoded MDPs @ ICLR 2023

Mon, 01 May 2023 10:42:42 +0100

WAE-MDPs

Wed, 01 Mar 2023 00:00:00 +0000

Official implementation of the Wasserstein Auto-encoded MDP (WAE-MDP) framework. WAE-MDPs enable the distilation of (any kind of) reinforcement learning policies into simpler controllers, paired with a discrete, tractable model of the environment (a latent space model). The two are provided with bisimulation guarantees, which allow formally verifying the behaviors of the agent operating under the simplified policy. The source code provided allows for replicating the experiments of the paper Wasserstein Auto-encoded MDPs: Formal Verification of Efficiently Distilled RL Policies with Many-sided Guarantees.

The source code is available on GitHub.

Deep RL generally lacks guarantees but are very effective to solve complex tasks in intricate environments. On the other hand, model-checking techniques allow to verify agent’s policies, but need the access to an environment’s model. Since model-checking usually relies on an exhaustive exploration of the input model, the latter must in general be finite (discrete spaces) and tractable.

WAE-MDPs distill RL policies learned via any RL technique (in particular, through deep neural networks) to learn a discrete, latent representation of the intractable input space. The distillation yields both a simpler policy and a world model that are amenable to model-checking.

One of the main features of WAE-MDPs is that the quality of the abstraction can be formally checked through PAC bounds on bisimulation pseudometrics. In short, the world model is guaranteed to be bisimilarly close to the real environment; it replicates closely the agent-environment interaction behaviors through its latent space. In addition, WAE-MDPs are equipped with representation guarantees: the representation learned ensures that input states that are grouped to the same latent state are bisimilarly close (the agent behaves the same way from those states).

Beyond those PAC-verifiable guarantees, the bisimulation guarantees are also maintained during learning. This is in contrast to VAE-MDPs, which learn a surrogate, variational objective to avoid the intractable computation of the Wasserstein metric, in the fixpoint definition of bisimulation. Instead, WAE-MDPs soundly approximate this computation by learning a discriminator (“max”) that distinguishes between states that are likely to be produced in the learned world model, and those of the real environment. This yields a $2$-player game objective (min/max) where the goal of the minimizer is to learn both a representation and a world model that makes the latent states generated by the encoder and the world model indistinguishable.

By doing so, WAE-MDPs avoid posterior collapse issues, which drastically speeds up learning, and yield distilled policies and world models of better quality.

Wasserstein Auto-encoded MDPs: Formal Verification of Efficiently Distilled RL Policies with Many-sided Guarantees

Sun, 01 Jan 2023 00:00:00 +0000

Formal Verification of Efficiently Distilled RL Policies with Many-sided Guarantees @ BNAIC/BeNeLearn 2022

Mon, 07 Nov 2022 10:42:42 +0100

Distillation of RL Policies with Formal Guarantees via Variational Abstraction of Markov Decision Processes

Tue, 28 Jun 2022 00:00:00 +0000

VAE-MDPs

Fri, 17 Dec 2021 00:00:00 +0000

A TensorFlow 2 implementation of Variational Markov Decision Processes, a framework allowing to (i) distill policies learned through (deep) reinforcement learning and (ii) learn discrete abstractions of continuous environments, the two with bisimulation guarantees.

The source code provided allows replicating the experiments presented in the paper Distillation of RL Policies with Formal Guarantees via Variational Abstraction of Markov Decision Processes.

The source code is available on GitHub.

Life is Random, Time is Not: Markov Decision Processes with Window Objectives

Mon, 14 Dec 2020 00:00:00 +0000

Simple Strategies in Multi-Objective MDPs

Thu, 30 Apr 2020 00:00:00 +0000

Life Is Random, Time Is Not: Markov Decision Processes with Window Objectives

Fri, 30 Aug 2019 00:00:00 +0000

Formal Methods | Florent Delgrange