During training pipeline issues can manifest themselves as data deficiency from the outside. Here are few things you can do
a) Normally teams will audit the data mixture config. It is actually more important to monitor data the loader actually sampled. For example if the config asks for 15 percent code and the realized sample is 4 percent code then in this example the model is not weak at code because the data is less but it is weak at code because of pipeline issues or config.
b) Quality filters can drop data indiscriminately. Some domains may not have enough data so if quality filters drop that data we create an artificial deficiency. For example a filter tuned on web text can remove most of a smaller domain e.g math if it does not like formatting without anyone noticing. So its useful to track the drop rate of the quality filters per domain and not globally.
c) First let me talk about packing and then what to be cautious about.
The model trains on sequences of a fixed size such as 8192 tokens and most documents are much shorter than that and so if every document is given its own sequence then the remainder of that sequence is filled with padding tokens. Those padding tokens are masked out of the loss and they contribute nothing to learning but they still occupy the sequence.
What packing does is bin packs many documents into each sequence until it is full. However there is a cost. First you need a block diagonal attention mask so that each document only attends to itself otherwise a document learns from whatever unrelated document happens to sit before it in the sequence. The second is that you have to decide what happens to documents longer than the sequence length and if you truncate them you lose the tail of every long document and if you split them across sequences you break the context that made them valuable in the first place.
Its important to measure packing efficiency per domain because you could lose a lot of domain specific data in truncation.
d) One last simple thing - keep the mapping from shard to step so that a loss spike can be traced back to the exact batch that produced it.