I’ve got an area of model training dynamics I’d like to investigate. It’s about the sub-circuits in a model and the dynamics of their creation, destruction and mutual interference.
To explain exactly what I’m interested in will require sharing my motivation, which comes from a problem in the philosophy of science.
When humans make mental models, they make them modular. For example, we have models for how fluid flows and how agents trade with one another and when we get evidence about the behaviour of fluids that only affects our fluid models, not our trade models. Examining a river, you don’t even consider thinking “what does the river’s motion imply for a nationalistic tariff policy to build up an industrial base?” You focus only on the river’s motion and consider only fluid dynamics.
Curiously, this doesn’t match our best epistemological model of reasoning, Solomonoff Induction. There, every hypothesis is evaluated by how well it does at predicting every bit of data that comes in.
Which irks me. I want my epistemological ideal to be of the same sort of thing as my experience when seeking knowledge. If thinking involves meta-cognition, embededness, and modularity, then so too should idealized thinking, dangnabbit.
Now, you can just narrow the hypothesis space to mimick reasoning about any particular little domain of experience. For instance, when thinking about coin-flips, you can just consider hypothesis about fair, independant coin flips. No need to include Conway’s Game of Life, or Symplective Integrators. No need, indeed, for theories of everything.
Which is good! Otherwise, you’d be constantly wasting time comparing countless theories that obviously has nothing to do with the latest bit of data. But it’s not good enough.
Because I don’t know how you’d go about formalizing this process. To see the bottleneck, imagine you say: OK, I’m going to mimic the limited domains of experience by saying that a domain corresponds to some computable function telling you to ignore bits at certain times. And then I’m going to consider hypothesis seperately for each of these domains. Then what happens?
Well, for domains that only get tiny, infrequent bits of data you’re going to only consider the simplest hypotheses and there aren’t that many of those. Not too interesting. And for the domains that include almost all the data, you’re going to basically wind up with theories of everything again. So for domains somewhere between “ignore all data” and “attend to all data” is where you have the thing that corresponds to things like “model trade” or “model fluid dynamics”. And probably, most of the domains don’t correspond to anything interesting, really.
So in order to model what we find interesting, we need to only certain domains, to only evaluate theories on those domains. How do we do that? Well, I suspect interesting domains will be those where the same models crop up. In other words, when we can compress a lot of domains into one effectively, that’s when we’re golden. Perhaps coupled by there being one simple domain that covers them all.
This process rhymes with Solomonoff Induction itself, Natural Abstractions and probably more stuff I’m forgetting right now. But then you have to do infinite comparisons of all domains with each other at each time step, in order to get away with infinite comparisons of theories with data each time step! That’s worse, if anything.
So I don’t think this approach is quite the right way. But clearly, humans know how to attend to things without doing infinite amounts of comparisons. Unfortuntely, ethics committes will be on your arse if you expose toddlers to carefully considered curriculums of data and stick probes in their heads to see how their neural networks change. So to get some data on this, we’ll need to look at artificial baby neural networks instead.
So, what I propose doing is to take some training task where there are clearly multiple distinct sub-tasks which are mostly independant, training a NN on said data, and seeing how circuits evolve in the network over time. Then we vary the structure of the data to see how that affects circuit formation in the network.
To be concrete, consider modular addition of two pairs of numbers. We have (a,d) + (b,e) = (a+b % c, d+e% c). At first we just show the network addition of numbers like (a,0) and (b,0) xor (0,a) and (0,b). Does the network grok two copies of the same circuits? If you then start showing it (a,b) + (c,d), do those networks merge? What if you do modular addition and then modular multiplication seperately focusing on different subets of one finite field? And then you move to data covering all of the field? How do the circuits intefere with each other? Does one subsume the other? Does a new, more general circuit emerge apart from the initial ones? How does the network form and maintain different sets of “hypothesis” for different domains and when does it consider domains to be different?
I’d like to see someone study that.