It turned out the neural networks are really good at classification. The largest neural networks today are 100k-250k classifiers, meaning they produce outputs that are interpreted as distributions over centi-thousands of classes. For example the OG gpt-2 used a vocabulary size of 50257 (later made to contain more factors of 2 for efficiency).
We then know what the loss value should start on because a neural network with initializers like xavier etc, will produce predictions $\hat{y}\approx \frac{1}{2}$ hence the cross entropy is $\log C$ where C are the number of classes. So according to this the gemini loss curve would start with $C=|V|$, $\log (256128) \approx 12.4$ while GPT4 would start at 11.5
After training for a while and reaching let's say 2.7 in nats, then we can say it's like being random classifier in $e^2.7$ classes, which would be 15 in this case. So out of 50k tokens, we are predicting amongst 15 tokens of which one is correct. That is pretty good.
This is a good way to sanity-check your classifiers.
TODO:
- [ x ] Add plot