Asking for continual learning with Regression on Single MLP

Hi, everyone. I have a couple of questions regarding continuous learning applied in regression tasks. I have just started my research in the field of CL. So I have to apologize in advance if I say something wrong. :smiling_face_with_three_hearts:

I’d like to train a toy example to know how continual learning works on neural networks and to apply it in my application. Let me talk about the scenario briefly:
There is an MLP composed of 8 fully connected layers. The input of the network is 2D (x,y) value and the output is RGB value. I’d like to assign colors sequentially: first, white at (0,0), second, black at (1,1), thrid, red at (2,2), fourth, blue at (3,3), … and so on. I want to the network do not forget the previous colors while learning new colors.
Also, at the inference time, I will sample all the coordinate including the learning points (e.g, (0.5, 0.5))

I tried this concept without any continual learning strategy and there is the catastrophic forgetting problem. I also tried to apply rehearsal-based and regularization-based methods. Interestingly, the rehearsal-based method shows great effect because it is almost the same with batch-wise learning because the network will “overfit” the data. In contrast, regularization-based method shows worse results than rehearsal. (actually better than no strategy) :sweat_smile:

I think rehearsal for this scenario is cheating, so I’d like to apply any other method.
However, since all the existing CL methods have been studied in classification, there are theoretical difficulties in applying this to my case. :joy: :joy:
So my question is:

  1. Do you know of any papers that are similar to this situation or that might be helpful?
  2. Can I see my scenario as CL with regression? As I know there was hardly any about this topic
  3. Or, how would you approach this situation?

BTW, I came in after seeing a question the author of this paper left on this community. :wave:

Thank you for your time!