During training a data mixture config will say something like code (data domain) is 10 percent of the run.
That number is a plan and it is not what the model actually experiences during training. There are four places where the plan and the real exposure come apart and most pipelines do not measure any of them.
a) Documents are not the same as tokens. For example if a corpus allocates 10 percent of documents to a domain and if those documents average 1000 tokens and all documents in other domains contain on an average 100 tokens. Then in this example one domain is contributing 53 percent of the tokens even if the mixture config says 10 percent.
b) Tokens loaded are not the same as tokens presented. Truncation will drop tokens that your inventory already counted. Padding does the opposite and it adds tokens that carry no information. For example if your average training example is 400 tokens and your context window is 8192. Then most of what you present on every step is padding. Packing removes almost all of this waste and I will cover it in my next post.
c) Tokens presented are not the same as tokens that contribute to the loss. Prompt tokens in instruction data are masked out. System prompts are masked out and they repeat on every single example. For example if a training example has a 400 token system prompt and 2000 tokens of user content and a 300 token assistant response. Then this example presents 2700 tokens to the model but only 300 tokens contribute to the loss. Most people will report the 2700 number for this dataset when the model is only learning from the 300.
d) Repetition goes up as the training budget goes up. For example if the budget is one trillion tokens and you allocate 10 percent of it to a domain that holds 20 billion unique tokens. Then you are drawing 100 billion tokens from a pool of 20 billion and the model makes five passes over that domain. Now if you raise the budget to five trillion tokens and keep the same mixture then that becomes 25 passes.
Lets look at that calculation in reverse as it can be a more useful metric. Repetition past about four passes returns very little. So a domain with 20 billion unique tokens supports around 80 billion tokens of useful allocation and at a one trillion budget that works out to 8 percent. A 10 percent allocation is already asking the data for more than it holds. The unique data you have puts a ceiling on the weight you can assign to it.
Monitoring 4 things will make these decisions a lot easier.
a) Tokens loaded for every source at every processing stage.
b) Tokens presented after truncation and padding and packing.
c) Tokens that contribute to the loss after all masking is applied.
d) The full distribution of passes across documents and not only the average. Memorization tracks the tail and an average of five passes will not show you a document seen 200 times.