View in #questions-forum on Slack
@MUHAMMED_NABEEL_K: I have referred and read many papers over the past few days and I am really confused about what is the major difference between incremental ,continual and online learning. Can someone help me out to clear this concept?
@vlomonaco: many refer to these terms as synonyms, but people have different opinions on this
Personally, on a technical level, I consider continual and incremental as synonyms, while I use online only to consider learning one pattern at a time.
I also had some difficulty with these definitions
I would recommend you to read chapters 1 and 2 of Lifelong Machine Learning, 2nd Edition - Zhiyuan Chen and Bing Liu, Morgan & Claypool 2018.
There you’ll find a nice explanation about related learning paradigms to continual learning, including online learning definition
Lifelong Machine Learning
@Simon_Ouellette: In my opinion, continual learning is a subset of online learning that deals with catastrophic forgetting, whereas standard “online learning” (of the non-continual variety) is more naive and incurs forgetting.
@Renato_Aranha: hello Simon, what do you mean by online learning non-continual variety?
@Simon_Ouellette: When people mention online learning without mentioning continual learning, I think of it as an iterative process of re-training the model as new data comes in.
Which of course leads to forgetting of old concepts, unless “Continual Learning” techniques are used.
@Renato_Aranha: hum right, I got your point. Thanks!
@Weston: Very good question and good answers. I also had same confusions. Still it is there
@vlomonaco: hah I guess we need to create a wiki section about this @andcos https://wiki.continualai.org/
@Weston: @Simon_Ouellette Can you help me to understand those “continual learning” techniques which are exclusively used for avoiding the forgetting in online learning?
@vlomonaco Good idea. Can we explain with some real time examples?
So, we are all in same page like “incremental” and “continual” are same. I have a basic doubt, then which one is the pure scientific term we should use in publications (i mean titles)
Also, I have clarity issues with “Active learning”. Does this fall into any category?
@Simon_Ouellette: @Weston you can look at ContinualAI’s excellent collection of references on Continual Learning, on https://wiki.continualai.org/, to read papers on these techniques (usually grouped under 3 categories: weight consolidation-based, replay-based, and dynamic architectures).
@andcos: Yes, as @Simon_Ouellette already pointed out, our wiki gathers many different CL strategies. The vast majority of them are tailored to mitigate forgetting of previous knowledge. @vlomonaco I think it is a good idea to add a section discussing differences between popular paradigms, like online learning, continual/lifelong, multitask and so on. I will create a new issue
@Weston: @Simon_Ouellette Thank you… I will analyse