[ContinualAI Reading Group]: "Defining Benchmarks for Continual Few-Shot Learning"

[May 8th, 2020] ContinualAI Reading Group: "Defining Benchmarks for Continual Few-Shot Learning"

Abstract: Both few-shot and continual learning have seen substantial progress in the last years due to the introduction of proper benchmarks. That being said, the field has still to frame a suite of benchmarks for the highly desirable setting of continual few-shot learning, where the learner is presented a number of few-shot tasks, one after the other,and then asked to perform well on a validation set stemming from all previously seen tasks. Continual few-shot learning has a small computational footprint and is thus an excellent setting for efficient investigation and experimentation. In this paper we first define a theoretical framework for continual few-shot learning, taking into account recent literature, then we propose a range of flexible benchmarks that unify the evaluation criteria and allows exploring the problem from multiple perspectives. As part of the benchmark, we introduce a compact variant of ImageNet, called SlimageNet64, which retains all original 1000 classes but only contains 200 instances of each one (a total of 200K data-points) downscaled to 64×64 pixels. We provide baselines for the proposed benchmarks using a number of popular few-shot learning algorithms, as a result, exposing previously unknown strengths and weaknesses of those algorithms in continual and data-limited settings.

The speakers for this reading group were:

Antreas Antoniou, Massimiliano Patacchiola, Mateusz Ochal

:round_pushpin: You can find the recording of this meeting on youtube: https://www.youtube.com/watch?v=7G9BlvodXRk
:round_pushpin: Paper pre-print: https://arxiv.org/pdf/2004.11967.pdf

2 Likes

Link to the slides: https://drive.google.com/file/d/1kIGsMfrkw81Wpaaq--31Z2bFP8wNf7Zf/view?usp=sharing

Link to the dataloader: https://github.com/AntreasAntoniou/FewShotContinualLearningDataProvider

Link to the code of the experiments: https://github.com/AntreasAntoniou/FewShotContinualLearning

Link to the dataset: https://zenodo.org/record/3672132

1 Like