Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons
A case study shows AutoML engine Orcetra appeared to beat FLAML and AutoGluon on 513 OpenML datasets, winning 57.1% at a 60-second budget and 78.4% against FLAML at 30 seconds. Protocol defects: test-set selection and unenforced budgets. Re-running with fixes, win rate fell from 59.4% to 34.3%.
发展脉络
- 首次出现Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML ComparisonsarXiv cs.AI
- 当前判断Short-budget AutoML comparisons are common in tool READMEs and workshop papers, and this case study shows they can be easily wrong. The findings suggest that the AutoML community needs to adopt more rigorous evaluation standards to avoid misleading claims. This could impact how tools are marketed and compared, potentially leading to more trustworthy benchmarks.Agent Pulse · 分析
A case study reveals that short-budget AutoML comparisons can be misleading due to protocol defects. The Orcetra engine appeared to beat FLAML and AutoGluon on 513 OpenML datasets, winning 57.1% at a nominal 60-second budget and 78.4% against FLAML at 30 seconds. However, these results stemmed from two defects: the search loop scored candidates on the test split and reported the best, while baselines selected on training data; and the budget was checked before launching a candidate but not enforced during execution, leading to a median 120 seconds against a 60-second budget. After re-running with validation split selection, external deadline enforcement, and equal resource sharing, Orcetra's win rate fell from 59.4% to 34.3%, and no significant pairwise differences remained.
The case study highlights the importance of protocol rigor in AutoML evaluation. The observed performance inflation came from test-set peeking and unenforced budgets. For practitioners, this implies that reported short-budget AutoML results may be unreliable unless protocols are explicitly stated. The next signal to watch is whether benchmark suites adopt standardized evaluation protocols with enforced budgets and proper validation splits.
Short-budget AutoML comparisons are common in tool READMEs and workshop papers, and this case study shows they can be easily wrong. The findings suggest that the AutoML community needs to adopt more rigorous evaluation standards to avoid misleading claims. This could impact how tools are marketed and compared, potentially leading to more trustworthy benchmarks.
For AutoML tool vendors, this study underscores the risk of overclaiming performance based on flawed evaluations. Adopting rigorous protocols can build trust with users and differentiate products. For users, it highlights the need to scrutinize benchmark claims and prefer tools with transparent evaluation methodologies.
Expect a push towards standardized evaluation protocols for short-budget AutoML, possibly with external budget enforcement and validation-based model selection. This may lead to more reliable comparisons and better-informed tool selection by practitioners.