Rating: 
Summary: An excellent introduction
Review:
As a subfield of artificial intelligence, reinforcement learning has shown great success from both a theoretical and practical viewpoint. Taking the form of numerous applications in finance, network engineering, robot toys, and games, it is clear that his learning paradigm shows even greater promise for future developments. The authors summarize the foundations of reinforcement learning, some of this coming from their own work over the last decade.
The authors define reinforcement learning as learning how to map situations to actions so as to maximize a numerical reward. The machine that is indulging in reinforcement learning discovers on its own which actions will optimize the reward by trying out these actions. It is the ability of such a machine to learn from experience that distinguishes it from one that is indulging in supervised learning, for in the latter examples are needed to guide the machine to the proper concept or knowledge. The authors emphasize the "exploration-exploitation" tradeoffs that reinforcement-learning machines have to deal with as they interact with the environment.
For the authors, a reinforcement learning system consists of a `policy', a `reward function', a `value function', and a `model' of the environment. A policy is a mapping from the states of the environment that are perceived by the machine to the actions that are to be taken by the machine when in those states. The reward function maps each perceived state of the environment to a number (the reward). A value function specifies what is the good for the machine over the long run. A model, as the name implies, is a representation of the behavior of the environment. The authors emphasize that all of the reinforcement learning methods that are discussed in the book are concerned with the estimation of value functions, but they point out that other techniques are available for solving reinforcement learning problems, such as genetic algorithms and simulated annealing.
The authors use dynamic programming, Monte Carlo simulation, and temporal-difference learning to solve the reinforcement learning problem, but they emphasize that each of these methods will not give a free-lunch. An entire chapter is devoted to each of these methods however, giving the reader a good overview of the weaknesses and strengths of each of these approaches. The differences between them usual boil down to issues of performance rather than accuracy in the generated solutions. Temporal difference learning in fact is viewed in the book as a combination of Monte Carlo and dynamic programming techniques, and in the opinion of this reviewer, has resulted in some of the most impressive successes for applications based on reinforcement learning. One of these is TD-Gammon, developed to play backgammon, and which is also discussed in the book.
The authors emphasize that these three main strategies for solving reinforcement learning problems are not mutually exclusive. Instead each of them could be used simultaneously with the others, and they devote a few chapters in the book illustrating how this "unified" approach can be advantageous for reinforcement learning problems. They do this by using explicit algorithms and not just philosophical discussion. These discussions are very interesting and illustrate beautifully the idea that there is no "free lunch" in any of the different algorithms involved in reinforcement learning.
In the last chapter of the book the authors overview some of the more successful applications of reinforcement learning, one of them already mentioned. Another one discussed is the `acrobot', which is a two-link, underactuated robot, which models to some extent the motion of a gymnast on a high bar. The motion of the acrobot is to be controlled by swinging its tip above the first joint, with appropriate rewards given until this goal is reached. The authors use the `Sarsa' learning algorithm, developed earlier in the book, for solving this reinforcement learning problem. The acrobot is an example of the current intense interest in machine learning of physical motion and intelligent control theory.
Another example discussed in this chapter deals with the problem of elevator dispatching, which the authors include as an example of a problem that cannot be dealt with efficiently by dynamic programming. This problem is studied with Q-learning and via the use of a neural network trained by back propagation.
The authors also treat a problem of great importance in the cellular phone industry, namely that of dynamic channel allocation. This problem is formulated as a semi-Markov decision problem, and reinforcement learning techniques were used to minimize the probability of blocking a call. Reinforcement learning has become very important in the communications industry of late, as well as in queuing networks.
Rating: 
Summary: Unified approach to AI, machine learning, control
Review: Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In this book, we provide an explanation of the key ideas and algorithms of reinforcement learning. The discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. This book is meant to be an introductory treatment of reinforcement learning, emphasizing foundations and ideas rather than the latest developments and mathematical proofs. We divide the ideas underlying the field into a half dozen primary dimensions, consider each in detail, and then combine them to form a much larger space of possible methods including all the most popular ones from Q-learning to value iteration and heuristic search. In this way we have tried to make the book interesting to both newcomers and experts alike. We have tried to make the work accessible to the broadest possible audiences in artificial intelligence, control engineering, operations research, psychology, and neuroscience. If you are a teacher, we urge you to consider creating or altering a course to use the book. We have found that the book works very well as the text for a course on reinforcement learning at the graduate or advanced undergraduate level. The eleven chapters can be covered one per week. Exercises are provided in each chapter to help the students think on their own about the material. Answers to the exercises are available to instructors, for now from me, and probably later from MIT Press in an instructor's manual. Programming projects are also suggested throughout the book. Of course, the book can also be used to help teach reinforcement learning as it is most commonly done now, that is, as part of a broader course on machine learning, artificial intelligence, neural networks, or advanced control. I have taught all the material in the book in as little as four weeks, and of course subsets can be covered in less time. Table of contents: Part I: The Problem
1 Introduction
2 Evaluative Feedback
3 The Reinforcement Learning Problem Part II: Elementary Methods
4 Dynamic Programming
5 Monte Carlo Methods
6 Temporal Difference Learning Part III: A Unified View
7 Eligibility Traces
8 Generalization and Function Approximation
9 Planning and Learning
10 Dimensions of Reinforcement Learning
11 Case Studies
Rating:
Summary: Excellent introduction to reinforcement learning
Review: I have this book more than a year now and I am going through it for the second time, so I think I have a pretty good picture about it. The book consists of three parts. In the first part, "The Problem", the authors define the scope of issues reinfocement learning is dealing with and they give some interesting introductory examples. Then, they move on to the concept of evaluative feedback and, eventually, define the reinforcement learning problem formally. The second part, "Elementary Solution Methods" consists of three more-less independent subparts: Dynamic Programming, Monte Carlo Methods and Temporal Difference Learning. All three fundamental reinforcement learning methods are presented in an interesting way and using good examples. Personally, I liked the TD-Learning part best and I agree that this method is indeed the central method and an original contribution of reinforecement learning to the field of machine learning. The third part, "A Unified View" present more advanced techniques. The last chapter gives the most important case studies in reinforcement learning including Samuel's Checkers Player and Thesauro's TD-Gammon. The book is very readable and every chapter ends with illustrative exercises (many of them actually are real programming projects!), always useful summary and very valuable bibliographical and historical remarks. Some subchapters are more advanced and therefore marked with '*'. I really recommend first two parts to any student ofd computer science or anyone interested in machine learning and fuzzy computing. The third part is much more advanced but it would be definitely interesting for advanced computer scientists and graduate students. This is still the first edition of the book which means that the material is almost six years old, but it's the third printing, so there is lot of interest and I would suggest (for second edition) that authors include solutions to (at least selected) exercises, something like Knuth did in "The Art of Computer Programming".
Rating:
Summary: Excellent introduction to reinforcement learning
Review: I have this book more than a year now and I am going through it for the second time, so I think I have a pretty good picture about it. The book consists of three parts. In the first part, "The Problem", the authors define the scope of issues reinfocement learning is dealing with and they give some interesting introductory examples. Then, they move on to the concept of evaluative feedback and, eventually, define the reinforcement learning problem formally. The second part, "Elementary Solution Methods" consists of three more-less independent subparts: Dynamic Programming, Monte Carlo Methods and Temporal Difference Learning. All three fundamental reinforcement learning methods are presented in an interesting way and using good examples. Personally, I liked the TD-Learning part best and I agree that this method is indeed the central method and an original contribution of reinforecement learning to the field of machine learning. The third part, "A Unified View" present more advanced techniques. The last chapter gives the most important case studies in reinforcement learning including Samuel's Checkers Player and Thesauro's TD-Gammon. The book is very readable and every chapter ends with illustrative exercises (many of them actually are real programming projects!), always useful summary and very valuable bibliographical and historical remarks. Some subchapters are more advanced and therefore marked with '*'. I really recommend first two parts to any student ofd computer science or anyone interested in machine learning and fuzzy computing. The third part is much more advanced but it would be definitely interesting for advanced computer scientists and graduate students. This is still the first edition of the book which means that the material is almost six years old, but it's the third printing, so there is lot of interest and I would suggest (for second edition) that authors include solutions to (at least selected) exercises, something like Knuth did in "The Art of Computer Programming".
Rating:
Summary: Great!
Review: If you are interested in AI or more specifically, RL, I recommend buying and reading this book! It is well structured, concise, and complete. I feel like I have a strong background in Reinforcement Learning after reading this book and working some of the example problems.
Rating:
Summary: A thorough introduction to the field.
Review: The book covers all of the basic algorithms in Reinforcement Learning. The exposition mixes theoretical justifications for the algorithms with practical examples. The authors are very expert, and the exposition is very accurate.
Rating:
Summary: A thorough introduction to the field.
Review: The book is easy and interesting to read. The diagrams, especially those on TD, throw a great deal of insight on the basic concept of TD. The intuitive ideas behind RL are developed clearly. At the same time all the fundamental concepts are made mathematically precise using very simple language and notation. Anybody new to RL should find this book extremely useful.
Rating:
Summary: Its a nice introductory text on Reinforcement Leaning!
Review: The book is easy and interesting to read. The diagrams, especially those on TD, throw a great deal of insight on the basic concept of TD. The intuitive ideas behind RL are developed clearly. At the same time all the fundamental concepts are made mathematically precise using very simple language and notation. Anybody new to RL should find this book extremely useful.
Rating: 
Summary: Student
Review: This book is easy to read and understand. But.... For those examples, the authors should provide more details about the solution procedures...How to get the chars. Do not just show the results without any intemediate process. That is the only disappointment in this book. Also, too many exercises, the authors should provided the answers as well
Rating:
Summary: A Standard, Excellent Introductory Book
Review: This book is undoubtedly the standard book on the topic of reinforcement learning by the two leading researchers in this field. Different from many other AI or maching learning books, this book presents not only the technical details of algorithms and methods, but also a uniquely unified view of how intelligent agents can improve by interacting with the environment. Besides, it is very readable, without much math or theory. The exercises are challenging and interesting, and will force you to understand the stuffs in the book!
|
