Skip to main content
U.S. flag

An official website of the United States government

MARKOV: A methodology for the solution of infinite time horizon MARKOV decision processes

June 16, 2010

Algorithms are described for determining optimal policies for finite state, finite action, infinite discrete time horizon Markov decision processes. Both value-improvement and policy-improvement techniques are used in the algorithms. Computing procedures are also described. The algorithms are appropriate for processes that are either finite or infinite, deterministic or stochastic, discounted or undiscounted, in any meaningful combination of these features. Computing procedures are described in terms of initial data processing, bound improvements, process reduction, and testing and solution. Application of the methodology is illustrated with an example involving natural resource management. Management implications of certain hypothesized relationships between mallard survival and harvest rates are addressed by applying the optimality procedures to mallard population models.

Citation Information

Publication Year 1988
Title MARKOV: A methodology for the solution of infinite time horizon MARKOV decision processes
DOI 10.1002/asm.3150040405
Authors B.K. Williams
Publication Type Article
Publication Subtype Journal Article
Series Title Applied Stochastic Models and Data Analysis
Series Number
Index ID 5221948
Record Source USGS Publications Warehouse
USGS Organization Patuxent Wildlife Research Center