On the meaning of Pseudo-rehearsal and generative replay

Hello all,

I would like to hear your opinion on the use of the term “Pseudo-rehearsal”, introduced in [1], in the CL literature.

I often read papers in which pseudo-rehearsal is used as synonym of generative replay, which in turn implies the use of a generative model (e.g. GAN) to create new patterns similar to previously encountered ones.

However, the original paper [1] introduces this technique as a very different one: it creates random binary vectors and then it records the output computed by the network on such input. Hence, this synthetic dataset (random input, recorded output) represents the network behaviour at a particular state.
After the network has been trained on new (real) patterns, the old knowledge can be recalled by interleaving training on real and synthetic datasets.

Do you believe that the CL community should differentiate between the use of pseudo-rehearsal and generative replay? Or, instead, do you think that pseudo-rehearsal has now acquired a different and more suitable meaning and so we should proceed in this way?

Thank you!

[1] A. Robins, “Catastrophic Forgetting; Catastrophic Interference; Stability; Plasticity; Rehearsal.,” Connection Science, vol. 7, no. 2, pp. 123–146, 1995.

2 Likes

IMO we don’t really use the first one anymore, so for me pseudo rehearsal means rehearsal with pseudo labels. So, anytime you train on labels given by a past classifier, for example, I would label this as pseudo rehearsal. Generative replay is a good example of this.

In the end, maybe we don’t need group to these techniques into one category…

2 Likes

Yes, I see your point. I kind of agree, even if I prefer to keep the old meaning for pseudo-rehearsal and use generative replay for GAN-like approaches :slightly_smiling_face:

1 Like

They are based on the same idea of generating synthetic input space data for rehearsal purposes. Even some realtively new generative approaches with GANs or VAEs need and additional forward pass into the network to get the “pseudo” labels if they are not “conditioned”.

The only concrete difference with generative replay is that in this case the generation is more powerful since learned and gives much better results than using random noise!

So in the end I agree it’s ok to have separate names to emphasize this distinction!

1 Like

I agree with @optimass, in principle the term pseudo-replay was used to identify replay with pseudo labels, namely labels calculated by a classifier (either the same classifier that is trained in the CL algorithm or another pre-trained classifier).

In some subsequent work pseudo-replay was used to identify generative replay, in some other work it was used to identify generative replay with labels calculated by a classifier, creating some confusion.

IMO nomenclature is really important. If we all agree on a standard notation it will be easier to catalog different algorithms, and we could save some time when we read a new paper since we’ll immediately know the macro-area of the algorithm without struggling to grasp all the details.

IMO the nomenclature should be:
replay -> if some replay is used.
generative -> if the replay patterns are generated from some generative model, such as GANs or VAEs.
pseudo -> if the labels of the replay patterns are obtained by a past classifier.

2 Likes

but then generative replay will always be pseudo-generative replay :stuck_out_tongue:

1 Like

Not always. You could use a conditional GAN or a conditional variational autoencoder. In these cases you have no need to pass the generated data through a classifier to obtains the labels, since the wanted labels are explicitly provided during the generation.

1 Like