Today, if you want to tweak the configuration of a model (which you could do for a benchmarking or goal alignment use case), you have to re-run datasets one-by-one.

Ideally, it should be possible to select a certain segment of your datasets, and then re-run with a different configuration profile on representative datasets.