Four New Extensions to Unsupervised Basis Function Adaptation for Reinforcement Learning
Loading...
Citations
Altmetric:
Altmetric::
Download
Associated Records
Authors
File Licence
Editors
Date
2024-10
Source Title
Journal Title
Journal ISSN
Volume Title
Publisher
Melbourne Authors
Affiliation
School of Mathematics and Statistics
Document Type
PhD thesis
Subject Code
Citation
Access Status
Open Access
DOI
Access URL
Open Access URL
Open Access at PMC
Linked Resource URL
NHMRC Grant Code
ARC Grant Code
Description
© 2025 Edward Watson Barker
Abstract
Reinforcement learning (RL) agents are designed to learn, via a reward signal, effective ways to navigate a new and unfamiliar environment. Recent years have seen rapid progress on many important problems in RL. However many of the most powerful and popular current algorithms still require extremely long and often impractical training times, driving researchers to explore ways in which algorithms can learn more efficiently.
In recent work, researchers have introduced a technique, "unsupervised basis function adaptation," which involves using the frequency with which an agent visits different areas of the state space as it learns in order to dynamically adapt a set of features for use by an underlying RL algorithm. There is evidence that the technique enables an agent to learn an accurate representation of the value function more efficiently than would be possible otherwise. Existing research into the technique, however, is limited by its relatively narrow scope. It is limited to problems with large discrete state spaces.
In this thesis we address the limitations in this earlier work by proposing four extensions to the original idea. The extensions, when combined, greatly expand the scope of the original concept and allow it to be applied to far more interesting real-world problems. We first show that the original idea can be extended to large discrete action spaces, which is a domain known to be highly problematic for stand-alone non-linear approximation architectures such as neural networks. Second, we demonstrate that theoretical results obtained in relation to the original idea can be extended to continuous state and action spaces, an important extension given the wide range of problems with continuous state or action spaces. Third, we examine how the original concept has a natural extension to sequences of states and actions, such that the technique can be used to rapidly learn sequences of actions, further obviating the need for excessive training. And finally, we show how the concept can be applied to environments with a high degree of factorisation, that is, environments with numerous elements which tend to behave independently. This allows for the exploitation of such factorisations in order to make learning more efficient.
Notes
Terms of Use
Keywords
Reinforcement learning, Unsupervised learning, Machine learning, Operations research, Basis function adaptation, State aggregation, SARSA