ELLA Family of Lifelong Learning Algorithms

Lifelong learning

My research group has a 12-year history of pioneering work in lifelong (continual) learning. Our earliest work developed the first general-purpose framework for lifelong learning (ELLA), with subsequent work developing a family of lifelong learning algorithms around this framework. These methods include lifelong RL, cross-domain lifelong learning (best paper nominee), and zero-shot learning using task descriptors (best paper award), among others.

Lifelong learning is essential for an intelligent agent that will persist in the real world with any amount of versatility. Animals learn to solve increasingly complex tasks by continually building upon and refining their knowledge. Virtually every aspect of higher-level learning and memory involves this process of continual learning and transfer. In contrast, most current machine learning methods take a “single-shot” approach in which knowledge is not retained between learning problems.

My research seeks to develop lifelong machine learning for intelligent agents situated for extended periods in an environment and faced with multiple tasks. The agent will continually learn to solve multiple (possibly interleaved) tasks through a combination of knowledge transfer from previously learned models, revision of stored source knowledge from new experience, and optional guidance from external teachers or human experts. The goal of this work is to enable persistent agents to develop increasingly complex abilities over time by continually and synergistically building upon their knowledge. Lifelong learning could substantially improve the versatility of learning systems by enabling them to quickly learn a broad range of complex tasks and adapt to changing circumstances.

ELLA: Lifelong Learning for Classification and Regression

Under this project, we developed the Efficient Lifelong Learning Algorithm (ELLA) (Ruvolo & Eaton, 2013) — the first general-purpose framework for continual or lifelong machine learning. ELLA frames lifelong learning as online multi-task learning of consecutive tasks, and has equivalent performance to batch multi-task learning with over a 1,000x speedup. Our approach learns and maintains a repository of shared knowledge, rapidly learning new task models by building upon previous knowledge. ELLA provides a variety of theoretical guarantees on performance and convergence, along with state-of-the-art performance on supervised multi-task learning problems. It also supports active task selection to intelligently choose the next task to learn in order to maximize performance (Ruvolo & Eaton, 2013).

Multi-Task Policy Gradient Methods for Robotic Control

Lifelong Reinforcement Learning for Robotic Control

We extended the ELLA framework to reinforcement learning settings, focusing on policy gradient methods (Ammar et al., 2014). Policy gradient methods support sequential decision making with continuous state and action spaces, and have been used with great success for robotic control. Our PG-ELLA algorithm incorporates ELLA’s notion of using a shared basis to transfer knowledge between multiple sequential decision making tasks, and provides a computationally efficient method of learning new control policies by building upon previously learned knowledge. Rapid learning of control policies for new systems is essential to minimize both training time as well as wear-and-tear on the robot. We applied PG-ELLA to learn control policies for a variety of dynamical systems with non-linear and highly chaotic behavior, including an application to quadrotor control. We also developed a fully online variant of this approach with sublinear regret that incorporates safety constraints (Bou Ammar et al., 2015), and applied this technique to disturbance compensation in robotics (Isele et al., 2016).

Autonomous Cross-Domain Transfer

Despite their success, these approaches only support transfer between RL problems with the same state-action space. To support lifelong learning over tasks from different domains, we developed an approach for autonomous cross-domain transfer in lifelong learning (Bou Ammar et al., 2015) [IJCAI 2015 Best Paper Nomination]. For the first time, this approach allows transfer between radically different task domains, such as from cart pole balancing to quadrotor control.

Zero-Shot Policy Transfer

Zero-Shot Transfer in Lifelong Learning using High-Level Descriptors

To further accelerate lifelong learning, we showed that providing the agent with a high level description of each task can both improve transfer performance and support zero-shot transfer (Isele et al., 2016) [IJCAI 2016 Best Student Paper]. Given only a high level description of a new task, this approach can predict a high performance controller for the new task immediately through zero-shot transfer, allowing the agent to immediately perform on the new task without expending time to gather data before it can perform.

Lifelong Deep Learning

We developed a deep learning equivalent of ELLA in our DF-CNN architecture (Lee et al., 2019; Lee et al., 2021), demonstrating how the ELLA family can be extended to deep learning settings. We also showed how this approach can be combined with neural architecture search to further improve performance, automatically determining the appropriate levels of knowledge to share between tasks.

References

2021

Sharing less is more: Lifelong learning in deep networks with selective layer transfer

Seungwon Lee, Sima Behpour, and Eric Eaton

In Proceedings of the International Conference on Machine Learning (ICML), Jun 2021

Abstract Bib PDF

Effective lifelong learning across diverse tasks requires the transfer of diverse knowledge, yet transferring irrelevant knowledge may lead to interference and catastrophic forgetting. In deep networks, transferring the appropriate granularity of knowledge is as important as the transfer mechanism, and must be driven by the relationships among tasks. We first show that the lifelong learning performance of several current deep learning architectures can be significantly improved by transfer at the appropriate layers. We then develop an expectation-maximization (EM) method to automatically select the appropriate transfer configuration and optimize the task network weights. This EM-based selective transfer is highly effective, balancing transfer performance on all tasks with avoiding catastrophic forgetting, as demonstrated on three algorithms in several lifelong object classification scenarios.
@inproceedings{Lee2021SharingLessIsMore, author = {Lee, Seungwon and Behpour, Sima and Eaton, Eric}, year = {2021}, title = {Sharing less is more: Lifelong learning in deep networks with selective layer transfer}, booktitle = {Proceedings of the International Conference on Machine Learning (ICML)}, }

2019

Learning shared knowledge for deep lifelong learning using deconvolutional networks

Seungwon Lee, James Stokes, and Eric Eaton

In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19), Jul 2019

Abstract Bib PDF Website

Current mechanisms for knowledge transfer in deep networks tend to either share the lower layers between tasks, or build upon representations trained on other tasks. However, existing work in non-deep multi-task and lifelong learning has shown success with using factorized representations of the model parameter space for transfer, permitting more flexible construction of task models. Inspired by this idea, we introduce a novel architecture for sharing latent factorized representations in convolutional neural networks (CNNs). The proposed approach, called a deconvolutional factorized CNN, uses a combination of deconvolutional factorization and tensor contraction to perform flexible transfer between tasks. Experiments on two computer vision data sets show that the DF-CNN achieves superior performance in challenging lifelong learning settings, resists catastrophic forgetting, and exhibits reverse transfer to improve previously learned tasks from subsequent experience without retraining.
@inproceedings{Lee2019Learning, title = {Learning shared knowledge for deep lifelong learning using deconvolutional networks}, author = {Lee, Seungwon and Stokes, James and Eaton, Eric}, booktitle = {Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)}, pages = {2837--2844}, year = {2019}, month = jul, }

2016

Lifelong Learning for Disturbance Rejection on Mobile Robots

David Isele, Jose Marcio Luna, Eric Eaton, and 4 more authors

In Proceedings of the International Conference on Intelligent Robots and Systems (IROS-16), Oct 2016

Abstract Bib PDF

No two robots are exactly the same—even for a given model of robot, different units will require slightly different controllers. Furthermore, because robots change and degrade over time, a controller will need to change over time to remain optimal. This paper leverages lifelong learning in order to learn controllers for different robots. In particular, we show that by learning a set of control policies over robots with different (unknown) motion models, we can quickly adapt to changes in the robot, or learn a controller for a new robot with a unique set of disturbances. Furthermore, the approach is completely model-free, allowing us to apply this method to robots that have not, or cannot, be fully modeled.
@inproceedings{Isele2016Lifelong, author = {Isele, David and Luna, Jose Marcio and Eaton, Eric and {de la Cruz}, Gabriel V. and Irwin, James and Kallaher, Brandon and Taylor, Matthew E.}, year = {2016}, title = {Lifelong Learning for Disturbance Rejection on Mobile Robots}, booktitle = {Proceedings of the International Conference on Intelligent Robots and Systems (IROS-16)}, month = oct, publisher = {IEEE/RSJ}, bib2html_dl_video = {https://youtu.be/u7pkhLx0FQ0} }
Using task features for zero-shot knowledge transfer in lifelong learning

David Isele, Mohammad Rostami, and Eric Eaton

In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-16), Jul 2016

Award Abstract Bib PDF Poster Slides

IJCAI-16 Distinguished Student Paper (sole Best Student Paper)

Knowledge transfer between tasks can improve the performance of learned models, but requires an accurate estimate of the inter-task relationships to identify the relevant knowledge to transfer. These inter-task relationships are typically estimated based on training data for each task, which is inefficient in lifelong learning settings where the goal is to learn each consecutive task rapidly from as little data as possible. To reduce this burden, we develop a lifelong reinforcement learning method based on coupled dictionary learning that incorporates high-level task descriptors to model the inter-task relationships. We show that using task descriptors improves the performance of the learned task policies, providing both theoretical justification for the benefit and empirical demonstration of the improvement across a variety of dynamical control problems. Given only the descriptor for a new task, the lifelong learner is also able to accurately predict the task policy through zero-shot learning using the coupled dictionary, eliminating the need to pause to gather training data before addressing the task.
@inproceedings{Isele2016Using, author = {Isele, David and Rostami, Mohammad and Eaton, Eric}, year = {2016}, title = {Using task features for zero-shot knowledge transfer in lifelong learning}, booktitle = {Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-16)}, month = jul, bib2html_dl_slides = {papers/Isele2016Using-slides.pdf}, bib2html_dl_poster = {papers/Isele2016Using-poster.pdf} }

2015

Safe policy search for lifelong reinforcement learning with sublinear regret

Haitham Bou Ammar, Rasul Tutunov, and Eric Eaton

In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), Jul 2015

Abstract Bib PDF Poster Slides

Lifelong reinforcement learning provides a promising framework for developing versatile agents that can accumulate knowledge over a lifetime of experience and rapidly learn new tasks by building upon prior knowledge. However, current lifelong learning methods exhibit non-vanishing regret as the amount of experience increases, and include limitations that can lead to suboptimal or unsafe control policies. To address these issues, we develop a lifelong policy gradient learner that operates in an adversarial setting to learn multiple tasks online while enforcing safety constraints on the learned policies. We demonstrate, for the first time, sublinear regret for lifelong policy search, and validate our algorithm on several benchmark dynamical systems and an application to quadrotor control.
@inproceedings{BouAmmar2015Safe, author = {{Bou Ammar}, Haitham and Tutunov, Rasul and Eaton, Eric}, year = {2015}, title = {Safe policy search for lifelong reinforcement learning with sublinear regret}, booktitle = {Proceedings of the 32nd International Conference on Machine Learning (ICML-15)}, month = jul, bib2html_dl_slides = {papers/slides-BouAmmar2015Safe.pdf}, bib2html_dl_poster = {papers/poster-BouAmmar2015Safe.pdf} }
Autonomous cross-domain knowledge transfer in lifelong policy gradient reinforcement learning

Haitham Bou Ammar, Eric Eaton, Jose Marcio Luna, and 1 more author

In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-15), Jul 2015

Award Abstract Bib PDF

Finalist for IJCAI-15 Distinguished Paper

Online multi-task learning is an important capability for lifelong learning agents, enabling them to acquire models for diverse tasks over time and rapidly learn new tasks by building upon prior experience. However, recent progress toward lifelong reinforcement learning (RL) has been limited to learning from within a single task domain. For truly versatile lifelong learning, the agent must be able to autonomously transfer knowledge between different task domains. A few methods for cross-domain transfer have been developed, but these methods are computationally inefficient for scenarios where the agent must learn tasks consecutively. In this paper, we develop the first cross-domain lifelong RL framework. Our approach efficiently optimizes a shared repository of transferable knowledge and learns projection matrices that specialize that knowledge to different task domains. We provide rigorous theoretical guarantees on the stability of this approach, and empirically evaluate its performance on diverse dynamical systems. Our results show that the proposed method can learn effectively from interleaved task domains and rapidly acquire high performance in new domains.
@inproceedings{BouAmmar2015Autonomous, author = {{Bou Ammar}, Haitham and Eaton, Eric and Luna, Jose Marcio and Ruvolo, Paul}, year = {2015}, title = {Autonomous cross-domain knowledge transfer in lifelong policy gradient reinforcement learning}, booktitle = {Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI-15)}, month = jul, }

2014

Online Multi-Task Learning for Policy Gradient Methods

Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and 1 more author

In Proceedings of the 31st International Conference on Machine Learning (ICML-14), Jun 2014

Abstract Bib PDF

Policy gradient algorithms have shown considerable recent success in solving high-dimensional sequential decision-making (SDM) tasks, particularly in robotics. However, these methods often require extensive experience in a domain to achieve high performance. To make agents more sample-efficient, we developed a multi-task policy gradient method to learn SDM tasks consecutively, transferring knowledge between tasks to accelerate learning. Our approach provides robust theoretical guarantees and we show empirically that it dramatically accelerates learning on a variety of dynamical systems, including an application to quadcopter control.
@inproceedings{BouAmmar2014Online, author = {Ammar, Haitham Bou and Eaton, Eric and Ruvolo, Paul and Taylor, Matthew E.}, title = {Online Multi-Task Learning for Policy Gradient Methods}, booktitle = {Proceedings of the 31st International Conference on Machine Learning (ICML-14)}, year = {2014}, month = jun, }

2013

ELLA: An Efficient Lifelong Learning Algorithm

Paul Ruvolo and Eric Eaton

In Proceedings of the 30th International Conference on Machine Learning (ICML-13), Jun 2013

Abstract Bib PDF Poster Slides

The problem of learning multiple consecutive tasks, known as \emphlifelong learning, is of great importance to the creation of intelligent, general-purpose, and flexible machines. In this paper, we develop a method for online multi-task learning in the lifelong learning setting. The proposed Efficient Lifelong Learning Algorithm (ELLA) maintains a sparsely shared basis for all task models, transfers knowledge from the basis to learn each new task, and refines the basis over time to maximize performance across all tasks. We show that ELLA has strong connections to both online dictionary learning for sparse coding and state-of-the-art batch multi-task learning methods, and provide robust theoretical performance guarantees. We show empirically that ELLA yields nearly identical performance to batch multi-task learning while learning tasks sequentially in three orders of magnitude (over 1,000x) less time.
@inproceedings{Ruvolo2013ELLA, author = {Ruvolo, Paul and Eaton, Eric}, title = {ELLA: An Efficient Lifelong Learning Algorithm}, booktitle = {Proceedings of the 30th International Conference on Machine Learning (ICML-13)}, year = {2013}, month = jun, bib2html_dl_slides = {papers/slides-Ruvolo2013ELLA.pdf}, bib2html_dl_poster = {papers/poster-Ruvolo2013ELLA.pdf}, bib2html_dl_code = {software/ELLAv1.0.zip}, }
Active Task Selection for Lifelong Machine Learning

Paul Ruvolo and Eric Eaton

In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI-13), Jul 2013

Abstract Bib PDF Poster Slides

In a lifelong learning framework, an agent acquires knowledge incrementally over consecutive learning tasks, continually building upon its experience. Recent lifelong learning algorithms have achieved nearly identical performance to batch multi-task learning methods while reducing learning time by three orders of magnitude. In this paper, we further improve the scalability of lifelong learning by developing curriculum selection methods that enable an agent to actively select the next task to learn in order to maximize performance on future learning tasks. We demonstrate that active task selection is highly reliable and effective, allowing an agent to learn high performance models using up to 50% fewer tasks than when the agent has no control over the task order. We also explore a variant of transfer learning in the lifelong learning setting in which the agent can focus knowledge acquisition toward a particular target task.
@inproceedings{Ruvolo2013Active, author = {Ruvolo, Paul and Eaton, Eric}, title = {Active Task Selection for Lifelong Machine Learning}, booktitle = {Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI-13)}, year = {2013}, month = jul, bib2html_dl_slides = {papers/slides-Ruvolo2013Active.pdf}, bib2html_dl_poster = {papers/poster-Ruvolo2013Active.pdf}, bib2html_dl_code = {software/ELLAv1.0.zip}, }