Parsewave and the Problem of “More Data” in Post-Training

[u/trashnash007]

I've been wondering whether more post-training data become less useful with increasing model power.

With synthetic data generation, it's easy to produce a huge number of examples. However, the problem is that not all of them could be truly novel – most of them will be only shallow variations of the knowledge that the model already knows. At some point, more examples just won't give much more learning signal.

What could be more valuable, it seems, is having small amounts of challenging real-world tasks with ground-truth answers and proper review. What's tricky is to find the right balance: too easy tasks are useless, too difficult tasks could give a noisily learned signal.

One of the things which got me interested in Parsewave is that they work on the post-training data generation for engineering-type tasks, along with evaluation and traces. The company per se is not my focus, what interests me more is whether it's a new trend in data collection.

For people doing SFT, RL and post-training: how do you select examples for your datasets?

Is it easier to scale the dataset size or collect challenging examples for particular failure modes?

[u/racc15]

Is this an ad for your company?