An MRP formulation for supervised learning: generalized temporal difference learning models

Conference item

In traditional statistical learning, data points are usually assumed to be independently and identically distributed (i.i.d.) following an unknown probability distribution. This paper presents a contrasting viewpoint, perceiving data points as interconnected and employing a Markov reward process (MRP) for data modeling. We reformulate the typical supervised learning as an on-policy policy evaluation problem within reinforcement learning (RL), introducing a generalized temporal difference (TD) learning algorithm as a resolution. Theoretically, our analysis draws connections between the solutions of linear TD learning and ordinary least squares (OLS). We also show that under specific conditions, particularly when noises are correlated, the TD’s solution proves to be a more effective estimator than OLS. Furthermore, we establish the convergence of our generalized TD algorithms under linear function approximation. Empirical studies verify our theoretical results, examine the vital design of our TD algorithm and show practical utility across various datasets, encompassing tasks such as regression and image classification with deep learning.

Authors: Pan, Y; Wen, J; Xiao, C; Torr, PHS

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: OpenReview
• Host title: Proceedings of the ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists (ARLET)
• Article number: 2
• Publication date: 2024-06-19
• Acceptance date: 2024-05-01
• Event title: 38th Workshop on Aligning Reinforcement Learning Experimentalists and Theorists (ARLET 2024) @ ICML 2024
• Event location: Vienna, Austria
• Event website: https://icml.cc/
• Event start date: 2024-07-26
• Event end date: 2024-07-26
• Language: English
• Keywords: Markov reward process; temporal difference learning; reinforcement learning; supervised learning