2009 Poster Sessions : Control of Networked Markov Decision Systems with Delays

Student Name : Sachin Adlakha
Advisor : Andrea Goldsmith
Research Areas: Information Systems
We are interested in the control of an interconnected network of subsystems. Each subsystem is modeled as a Markov decision process (MDP) and this network of MDPs is referred to as a networked Markov decision process. Such a network of Markov decision processes is used to model a variety of control problems, such as distributed vehicle coordination, communication systems and distributed scheduling over multiple servers. Each subsystem is coupled to its neighbors via communication links over which the signals are delayed, but are otherwise transmitted noise-free. A centralized controller receives delayed state information from each subsystem. The control action applied to each subsystem takes effect after a certain delay. Such a distributed Markov decision process with inter-subsystem, observation and action delays can be represented as a partially observed Markov decision process (POMDP). We show that this POMDP is equivalent to an MDP with an observable state consisting of f inite number of past states and actions. We provide an explicit bound on the finite history of measurement and control that is required for the optimal control of such networked MDPs. We also show that these bounds depend only on the underlying graph structure and the associated delays. Thus, the POMDP associated with a networked MDP can be converted into an information state MDP, whose state does not grow with time. Because the augmented state used in the information state MDP is finite, we can compute the optimal controller for distributed Markov decision processes using dynamic programming over a finite state space.

Sachin Adlakha received his B.Tech in electrical engineering from Indian Institute of Technology, Delhi in 2000 and his M.S in electrical engineering from University of California, Los Angeles in 2002. From 2002-2005, he was a research engineer at Telogy Networks - A Texas Instruments Company, where he worked on design and implementation of technologies for voice over IP systems. In 2005, he joined Stanford University where he is currently a Ph.D candidate. His interests are in interdisciplinary fields of distributed controls, game theory and wireless networks. Specifically, he is interested in using tools from game theory and stochastic control to understand and design large-scale wireless networks. He is also interested in understanding how to design efficient control systems in presence of networks.