Would Task Sequence "A, B, A" be Better Than "A" alone?

1 Like

(sorry this is like the 75th time I post my recent paper here, but super relevant)

So in our new preprint Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning, we specifically propose a new setting where tasks re-occur and where we evaluate models on their online performance. So, for example, whenever you are solving Task A, you better accumulate as much knowledge as possible about the task for the next time the task comes around. The better you model can accumulate and retain knowledge about tasks, the better its cumulative online performance will be!

1 Like

Before I dig into your paper though, would this aid the re-learning of A (A’)? That is, how much of the previous learning from initial learning of the first A task, will remain after the interference from the forgetting from the B task?

As I mentioned, in a biological context, you DO typically see retained information from old tasks, even if the agent/animal/human “forgets” them.

Of course, this will vary by model and parameters, but just generally speaking.

In the paper, we focus on the new proposed settings.

However, we propose an initial baseline that uses meta-learning to accumulate knowledge.
So, when task A comes back around, you can re-learn it much more quickly (so, yes to your question).

As for the “how much” question, i’m not sure how to answer that. What metric do you have in mind?

Catastrophic forgetting may be assessed from two angles. One being how much of task A is forgotten after learning B. Another is to ask how much of A remains.

To me, this is an interesting question as it (the remaining ‘representation’ of A) may be the most heterogeneous from B, or hold some other potentially interesting property.

Maybe. It really will all depend on the task, model, training, etc.

I see.

We do not provide, however, a deep analysis of the forgetting. We only report the cumulative performance (which will reflect how much of the representation of A you are retaining)