Evaluation in compute/accuracy space

IMHO, it doesn’t make sense to study catastrophic forgetting (CF) without looking at computing.

You can solve CF with enough computing, e.g. approximate i.i.d data with replay.
Maybe we should study algorithms in performance/compute space.

I think that we be a more robust evaluation protocol.
I would like to further discuss this.

1 Like

Hi @optimass! I agree that just looking at the accuracy at the end of the training may not be enough! :slight_smile:

In these papers below we listed a number of additional metrics that can be used and are interesting for CL:

In particular in the framework I designed in my dissertation and here, we argue that both computational and memory resources should be bounded, which is another important consideration.

So, yeah, computing resources are extremely important but also memory for many applications.

1 Like

Yes, I’m aware of these papers actually.
My hope is that something similar to the one proposed in your 2nd paper would catch on!
Maybe your team should push it for a conference, to get more reach.

1 Like