View in #questions-forum on Slack
@Michael_Milstead: Beginner question about input sizes:
It seems like being able to handle different input data sizes (and potentially formats) is very important for CL, as we can’t expect to always get new task data from the same source (maybe learning one network with data from two different cameras).
With image data, we can crop/pad input images to make them fit the network’s expected input size, and features/classes can still be found by running filters over the image.
But what about non-image data? For example, imagine I have a network trained on 5 numerical features of a Flower (petal width, petal length, flower height, stem thickness, has thorns), and it has output classes, “Rose” and “Tulip”.
Now I have a few training samples for a new class, “Sunflower”, I’d like to add to the network, but my new data only contains “petal width”, “petal length”, and “flower height” values.
How can I make sure my 3 x values are sent into the correct network inputs when training? I guess I could try every combination of the three values and the five inputs and see which has the best result after training, but is there a better way?
@andcos: This is a good question and I don’t know how much it is treated in the literature (I guess not much). If your new features are simply a subset of the ones you have already seen, you can set to zero (or other values) the missing ones. In case you encounter new input features you could expand the input layer, similar to what happens in the output layer when new classes arrive.
@Michael_Milstead: Thanks! For the idea about setting other values to zero:
What if this new class is being added automatically instead of manually? If it’s manual, I can say, “yeah my 3 values are related to these 3 inputs, so set the other 2 to zero”. But what if it’s being added automatically? How do we decide which 2 input values should be set to zero, and which 3 should be used?
I wonder if this is not even a problem for human learning, since our input dimension is always the same with our physical senses
@Antonio_Carta: what do you mean by “added automatically”? In your example each feature has a name, so you know which columns of the input they belong to. It’s similar to a problem where you have missing input features.
@Michael_Milstead: In that example I just added names to help myself understand how the new task’s values could be a subset of the network’s inputs, but what if they didn’t have names?
@Alex_Torex: you need inductive biases
you know some values are typical for some features
you do some assumptions
you try other combinations
until you get expected predictions
for this you need more advanced nets with meta-learning and augmented memory
and you need to teach them from mini datasets with problems like this which require some reasoning and inductive biases
YouTube Video: Brenden Lake | Compositional generalization in minds and machines
@Michael_Milstead: Thanks Alex, that’s really helpful.
So correct me if this isn’t what you meant, but:
I basically need to train a meta model that can take the raw input and try to predict which inputs each value relates to?
Also, I feel that the video you linked is related, but I’m struggling to understand exactly what you wanted me to get out of it, and how exactly it relates. Can you explain a bit to help me make that step?
I mean, I understand the content of the video on its own, just not the application to my question.
@Alex_Torex: your problem cannot be solved with current tech. the video illustrates just a toy problem. You need to teach the net inductive biases, how the world works. This now requires complex techniques and datasets which are impractical on real problems. The AI needs to live in the real world to acquire necessary inductive biases.