{ "evaluation_id": "EVAL-20260402-204743", "question_id": "ANALYSIS-024", "question_text": "A quantitative trading firm backtests a strategy: 15% annual return, Sharpe ratio 2.1, max drawdown 8%. They want to go live. (1) What could go wrong between backtest and live trading? List at least 5 risks. (2) The backtest used 5 years of data and tested 200 parameter combinations. Calculate the probability this outperformance is due to overfitting. (3) Design a live testing protocol that minimizes capital at risk while validating the strategy.", "category": "analysis", "timestamp": "2026-04-02T20:47:43.323Z", "display_date": "Apr 02, 2026", "winner": { "name": "GPT-5.4", "provider": "openrouter", "score": 9.05 }, "avg_score": 6.935, "matrix_size": 78, "models_used": [ { "id": "gpt_5_4", "name": "GPT-5.4", "provider": "openrouter" }, { "id": "deepseek_v4", "name": "DeepSeek V4", "provider": "openrouter" }, { "id": "mimo_v2_flash", "name": "MiMo-V2-Flash", "provider": "Xiaomi" }, { "id": "claude_sonnet_46", "name": "Claude Sonnet 4.6", "provider": "openrouter" }, { "id": "grok_420", "name": "Grok 4.20", "provider": "openrouter" }, { "id": "gpt_oss_120b", "name": "GPT-OSS-120B", "provider": "OpenAI" }, { "id": "gemini_3_flash", "name": "Gemini 3 Flash Preview", "provider": "Google" }, { "id": "gemini_31_pro", "name": "Gemini 3.1 Pro", "provider": "openrouter" }, { "id": "minimax_m25", "name": "MiniMax M2.5", "provider": "openrouter" }, { "id": "claude_opus_46", "name": "Claude Opus 4.6", "provider": "openrouter" } ], "rankings": { "gpt_5_4": { "display_name": "GPT-5.4", "provider": "openrouter", "average_score": 9.05, "score_count": 9, "min_score": 8.4, "max_score": 9.8, "rank": 1 }, "grok_420": { "display_name": "Grok 4.20", "provider": "openrouter", "average_score": 8.34, "score_count": 9, "min_score": 6.8, "max_score": 9.2, "rank": 2 }, "gpt_oss_120b": { "display_name": "GPT-OSS-120B", "provider": "OpenAI", "average_score": 8.32, "score_count": 9, "min_score": 6.1, "max_score": 9.2, "rank": 3 }, "gemini_3_flash": { "display_name": "Gemini 3 Flash Preview", "provider": "Google", "average_score": 7.86, "score_count": 9, "min_score": 6.25, "max_score": 8.8, "rank": 4 }, "mimo_v2_flash": { "display_name": "MiMo-V2-Flash", "provider": "Xiaomi", "average_score": 7.69, "score_count": 9, "min_score": 6.45, "max_score": 9, "rank": 5 }, "claude_sonnet_46": { "display_name": "Claude Sonnet 4.6", "provider": "openrouter", "average_score": 7.01, "score_count": 9, "min_score": 3.2, "max_score": 9.4, "rank": 6 }, "deepseek_v4": { "display_name": "DeepSeek V4", "provider": "openrouter", "average_score": 6.99, "score_count": 8, "min_score": 5.25, "max_score": 9, "rank": 7 }, "minimax_m25": { "display_name": "MiniMax M2.5", "provider": "openrouter", "average_score": 5.73, "score_count": 4, "min_score": 0.5, "max_score": 8.35, "rank": 8 }, "claude_opus_46": { "display_name": "Claude Opus 4.6", "provider": "openrouter", "average_score": 5.56, "score_count": 4, "min_score": 0.45, "max_score": 8.05, "rank": 9 }, "gemini_31_pro": { "display_name": "Gemini 3.1 Pro", "provider": "openrouter", "average_score": 2.81, "score_count": 8, "min_score": 1.35, "max_score": 5, "rank": 10 } }, "judgments": [ { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 9.8, "brief_justification": "The response is exceptionally accurate, well-structured, and practical. It correctly identifies the statistical limitations in calculating overfitting probability while providing a solid baseline calculation and referencing advanced quant metrics like PBO. The testing protocol is highly actionable and reflects industry best practices." }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 5.25, "brief_justification": "" }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 7.5, "brief_justification": "" }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 6.75, "brief_justification": "The response provides excellent, deep, and accurate information for Part 1 and the beginning of Part 2, but it cuts off mid-sentence and completely fails to address Part 3, severely impacting completeness and usefulness." }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 8.3, "brief_justification": "" }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 7.55, "brief_justification": "The response provides excellent, accurate answers for the first two parts, but cuts off abruptly in the middle of the third part, failing to deliver the requested live testing protocol." }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.15, "brief_justification": "" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 2.25, "brief_justification": "The response is extremely incomplete - it only addresses a tiny fragment of part (3) of the question, completely ignoring parts (1) and (2). The question specifically asks for at least 5 risks for backtest-to-live issues, a probability calculation for overfitting given 200 parameter combinations over 5 years, and a full live testing protocol. The response appears to be cut off mid-sentence and only covers capital scaling and begins mentioning monitoring metrics. What little is provided is direct" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 9.55, "brief_justification": "Exceptionally thorough and well-structured response across all three parts. Part 1 goes well beyond the requested 5 risks with 12 well-explained items. Part 2 correctly notes the calculation cannot be done precisely, provides the standard multiple-testing estimate (1 - 0.95^200), and importantly caveats the interpretation—acknowledging it's not a true posterior probability and mentioning more sophisticated approaches (PBO, Deflated Sharpe, White's Reality Check). Part 3 is outstanding: a detaile" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 6.2, "brief_justification": "The response covers the three parts but has notable issues. For part (1), the five risks are valid and well-articulated but somewhat standard. For part (2), the quantitative analysis is problematic: the formula for expected max Sharpe ratio under the null is applied incorrectly (the result of ~0.10 is actually an annualized Sharpe, and the comparison logic is muddled), and the 'Deflated Sharpe Ratio' formula presented is not the actual DSR formula from Bailey & López de Prado. The response never" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 6.5, "brief_justification": "The response covers all three parts with reasonable structure and clarity. Part (1) is solid with 5 well-explained risks. However, Part (2) is problematic: the PBO calculation is largely hand-waved with a formula that appears fabricated rather than being the actual Lopez de Prado CSCV methodology. The 35-45% estimate is presented as if computed but is essentially a guess. A more honest approach would use Bonferroni correction, deflated Sharpe ratio, or the actual CSCV combinatorial method with p" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 6.85, "brief_justification": "The response provides a strong and well-structured Part 1 with more than 5 risks organized into clear categories with practical examples. However, Part 2 is incomplete - the calculation cuts off mid-formula and never delivers a final probability of overfitting, which was explicitly requested. Part 3 (live testing protocol) appears to be entirely missing. The Bailey-López de Prado framework is correctly invoked and the initial steps are reasonable, but the expected maximum Sharpe calculation cont" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 8.55, "brief_justification": "The response is comprehensive, well-structured, and practically useful. The risks enumerated in part 1 go beyond the minimum 5 and are all legitimate. Part 2 correctly identifies the multiple-testing problem and references appropriate literature (Bailey et al., López de Prado), though the actual calculation is more of a heuristic estimate than a rigorous derivation—the 72% figure is plausible but not precisely derived from a transparent formula, which slightly reduces the correctness score. Part" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8.2, "brief_justification": "The response is well-structured, thorough, and demonstrates strong quantitative finance knowledge. Part 1 exceeds the requirement with 10 well-explained risks. Part 2 correctly applies the multiple-testing framework with proper calculations (Bonferroni approach yielding ~99.997%), though it could be critiqued for treating this as a definitive 'probability of overfitting' rather than a conditional probability under the null hypothesis - the framing slightly conflates P(at least one false positive" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 7.55, "brief_justification": "The response is well-structured and clearly written. The five risks are accurate and well-explained. However, the overfitting calculation in Part 2 is weak: it references the Haircut Sharpe Ratio framework and Bonferroni correction but doesn't actually perform a rigorous calculation. The numbers given (20-30% probability, 40-60% haircut) are plausible ballpark estimates but are presented without showing the actual math (e.g., computing the expected maximum Sharpe under the null, or applying the " }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 0.5, "brief_justification": "The response is completely empty - no content was provided to evaluate. The model failed to address any of the three parts of the question." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 1.35, "brief_justification": "The response is severely incomplete and truncated. It only mentions capital scaling and partial monitoring, fails to list at least five backtest-to-live risks, does not calculate overfitting probability, and does not fully design a live testing protocol." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 6.45, "brief_justification": "Clear structure and practical live-testing advice, with more than five relevant risks. However, the key weakness is part (2): it claims to calculate overfitting probability but gives an unsupported 35–45% estimate and cites/misstates PBO methodology without enough information to compute a valid probability from only 5 years and 200 parameter combinations." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 3.2, "brief_justification": "Lists many relevant live-trading risks, but fails to answer parts (2) and (3) properly. The overfitting 'calculation' is incomplete, gives no actual probability, and the response is truncated mid-formula. Live testing protocol is missing entirely." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 6.8, "brief_justification": "Well-structured and practical, with strong risk discussion and a sensible live-testing protocol. However, the key flaw is part (2): it presents a probability of overfitting as if it can be calculated from the given information alone, then gives a broad range and arbitrary point estimate without a valid derivation. That materially hurts correctness." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 6.1, "brief_justification": "Clear and reasonably thorough on qualitative risks, but the key probability calculation is not actually derivable from the given information and is overstated by assuming alpha=5% and independence. The live-testing protocol is partially developed but truncated/incomplete." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 6.25, "brief_justification": "Clear and reasonably thorough on risks and live-testing protocol, but the key calculation is not actually computed and presents unsupported probability ranges. It discusses overfitting well conceptually, yet fails to justify a specific probability from the given information." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 5, "brief_justification": "Incomplete response missing key details and calculations." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 8.05, "brief_justification": "Accurate and comprehensive with practical insights." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 9.2, "brief_justification": "Comprehensive and detailed response covering all aspects thoroughly with clear explanations and practical recommendations." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 9, "brief_justification": "Comprehensive, detailed, and practical with strong analytical depth." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 9.4, "brief_justification": "Extremely thorough and detailed response covering all aspects of the question with high accuracy and depth." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 9.2, "brief_justification": "Comprehensive and detailed response covering all aspects of the question with high accuracy and practical insights." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 9.2, "brief_justification": "Comprehensive and detailed response addressing all aspects of the question with clear structure and actionable insights." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.8, "brief_justification": "Comprehensive and insightful analysis with practical recommendations." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 8.05, "brief_justification": "Accurate and comprehensive with practical insights." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 4.15, "brief_justification": "Response is incomplete, cuts off mid-sentence, and fails to address all parts of the question. Only partially addresses part (3) and ignores parts (1) and (2)." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 7.75, "brief_justification": "Response is well-structured and clear, covers most risks and protocol design, but lacks precise calculation for overfitting probability and some depth in risk analysis." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 9.35, "brief_justification": "Comprehensive, accurate, and highly actionable response covering all parts with strong statistical reasoning and practical protocols." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 8.35, "brief_justification": "Accurate and well-structured response with practical insights, though some calculations could be more precise." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 8.75, "brief_justification": "Comprehensive risk list, solid overfitting math, and actionable protocol; minor incompleteness in protocol details." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 9, "brief_justification": "Excellent response covering all three parts thoroughly. Risks are comprehensive and well-explained. Overfitting probability calculation correctly identifies the multiple testing issue and provides a reasonable estimate range. Live testing protocol is detailed, practical, and follows industry best practices (phased deployment, strict risk limits)." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8.6, "brief_justification": "Excellent coverage of risks and testing protocols. The overfitting probability calculation is technically correct under the stated assumptions, though the assumptions themselves are somewhat simplified." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.55, "brief_justification": "Response is accurate, thorough, and well-structured with practical protocols; minor imprecision in probability calculation." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 8.35, "brief_justification": "Accurate and thorough response with clear structure, insightful analysis on overfitting, and practical live testing protocol." }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 2.05, "brief_justification": "The response is severely incomplete - it appears to be a fragment of a larger answer, covering only a small portion of one sub-question (part 3). It doesn't address the 5 risks (part 1), the overfitting probability calculation (part 2), or provide a complete live testing protocol (part 3). What little content exists is reasonable but the response is essentially truncated and unusable as a complete answer." }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8.55, "brief_justification": "Comprehensive and well-structured response covering all three parts thoroughly. The overfitting probability calculation is mathematically correct and appropriately caveated—the model correctly notes this is a multiple-testing false positive estimate rather than a true posterior probability, and mentions more rigorous alternatives (Deflated Sharpe Ratio, PBO). The live testing protocol is detailed and practical with clear phases, metrics, and pass/fail criteria. Minor gaps: could have elaborated " }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 6.55, "brief_justification": "The response covers the key areas but has notable issues. The overfitting calculation in section 2 is mathematically inconsistent - the formula for E[max SR] uses T=1260 days but the DSR formula mixes days and years incorrectly, and the DSR formula itself is non-standard. The PBO framework is mentioned but not properly applied. Section 1 lists valid risks but misses important ones like look-ahead bias, liquidity assumptions, and transaction cost modeling errors. Section 3 provides reasonable pra" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 7.75, "brief_justification": "The response covers all three parts well with clear structure and practical recommendations. The risks section is solid and well-explained. The live testing protocol is actionable and thoughtfully designed. However, the overfitting probability calculation is the weakest section—the PBO formula presented is garbled and not the actual Lopez de Prado methodology, and the '35-45%' estimate lacks rigorous derivation. A more honest approach would have been to use the Bonferroni correction or the defla" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 8.55, "brief_justification": "The response is comprehensive and well-structured, covering all three parts thoroughly. The risk enumeration is excellent with 8 well-explained risks. The overfitting probability calculation is directionally correct and references appropriate literature (Bailey et al., López de Prado), though the specific 72% estimate lacks a rigorous derivation and the methodology could be more precisely explained. The live testing protocol is practical and actionable with clear phases. Minor weakness: the math" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8.8, "brief_justification": "Excellent response covering all three parts thoroughly. The risk enumeration is comprehensive (10 risks vs. minimum 5 required). The overfitting probability calculation is mathematically correct using Bonferroni correction with proper derivation. The live testing protocol is well-structured and practical. Minor deductions: the response appears truncated at section 3.1D, missing the complete live testing protocol. The overfitting probability framing is slightly misleading - the Bonferroni calcula" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8, "brief_justification": "The response is well-structured and covers all three parts effectively. The risks section is solid with 5 relevant risks clearly explained. The overfitting probability section references the Harvey-Liu framework appropriately but the quantitative calculation is vague and hand-wavy—it gives ranges without showing actual math (e.g., no explicit Bonferroni correction calculation, no Bailey-Lopez de Prado formula application). The live testing protocol is practical and actionable with clear phases a" }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 3, "brief_justification": "The response is incomplete and cuts off mid-sentence, addressing only part of question (3) while ignoring (1) and (2) entirely. It provides some relevant but superficial ideas on scaling and monitoring without depth or full coverage." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 6, "brief_justification": "The response covers the main points but lacks specific details on overfitting probability calculation and a structured live testing protocol. Some risks are generic." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8.55, "brief_justification": "The response is factually accurate on trading risks and protocols, with a logically sound (if simplified) multiple-testing calculation. It thoroughly covers all three questions with clear structure, insightful practical details, and actionable advice, though part 2 notes limitations of the exact probability." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 6.75, "brief_justification": "Good structure and covers the questions with relevant concepts, but contains mathematical errors in part (2) that reduce factual accuracy; depth is solid but not exceptional." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 7.35, "brief_justification": "The response is well-structured and clear. It lists relevant risks, provides a reasonable live testing protocol, and attempts a PBO calculation. However, the PBO formula and probability estimate (~35-45%) are incorrect or misapplied; with m=200 and n=1260 the actual overfitting risk is much lower than claimed according to standard methods like Lopez de Prado's PBO." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 8.15, "brief_justification": "Response provides comprehensive, well-structured risks with examples and formulas. Overfitting calculation uses relevant Bailey-Lopez de Prado framework but is incomplete (formula cuts off). Live testing protocol is entirely missing, reducing completeness. High clarity and depth in covered sections." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8.8, "brief_justification": "The response is factually accurate with strong quantitative analysis on overfitting, provides a comprehensive list of risks exceeding the minimum, and offers a clear, actionable live testing protocol. Minor incompleteness in the final section where the table is cut off." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 7.75, "brief_justification": "The response accurately identifies key risks and provides a solid live testing protocol. The overfitting probability calculation is approximate and somewhat vague on the exact method (mixing concepts without precise formula), lowering correctness slightly, but overall insightful and well-structured." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 6, "brief_justification": "The response addresses all three parts but contains inaccuracies in the overfitting probability calculation (misapplies multiple testing formulas and arrives at an implausible 99.9999% figure without proper justification). Risks and protocol are reasonably listed but lack depth and specificity for a quant context." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8.4, "brief_justification": "Accurately lists risks and offers a detailed live testing protocol; probability estimate is simplistic but acknowledged, overall a solid, actionable answer." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 6.05, "brief_justification": "Risks listed well and protocol clear, but overfitting probability not actually computed and formulas are oversimplified, reducing factual accuracy and completeness." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 7.5, "brief_justification": "Risks and protocol are well explained and actionable, but the overfitting probability calculation is vague and not rigorously derived, affecting factual accuracy." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 3.4, "brief_justification": "The answer lists many risks but cuts off before completing the overfitting probability calculation and does not provide a live testing protocol, making it factually incomplete and only partially clear." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 6.95, "brief_justification": "Lists many relevant risks and gives a solid live testing protocol, but the overfitting probability estimate is vague and likely inaccurate, reducing overall factual correctness and completeness." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 7.3, "brief_justification": "Lists key risks and a solid live testing protocol, but the overfitting probability is vague and not rigorously calculated, limiting factual accuracy." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 2.2, "brief_justification": "The response is severely truncated and incomplete. It fails to answer the first two parts of the prompt entirely and cuts off mid-sentence in the third part." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 9.55, "brief_justification": "The response is exceptionally thorough and practical. It correctly identifies the statistical challenges of multiple testing (Part 2) while providing a comprehensive list of risks and a professional-grade staging protocol for live trading." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 9, "brief_justification": "The response is highly accurate and professional. It correctly identifies key risks and provides a solid live testing protocol. For the overfitting calculation, it correctly references the Deflated Sharpe Ratio and PBO concepts, though the specific DSR formula provided is a simplified approximation rather than the full Bailey/Lopez de Prado version. The logic remains sound." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 9, "brief_justification": "The response is highly accurate and professional. It correctly identifies key risks like alpha decay and slippage. The PBO section correctly references Lopez de Prado and provides a realistic heuristic estimate, though the manual formula provided is more of a conceptual representation. The testing protocol is practical and follows industry best practices." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 8.65, "brief_justification": "The response provides an excellent technical breakdown of backtesting risks and uses the sophisticated Bailey-López de Prado framework for calculating overfitting probability. However, the response is truncated at the end, failing to provide the final DSR calculation and the requested live testing protocol (Part 3)." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 9.15, "brief_justification": "The response is excellent. It correctly identifies the core risks, provides a realistic (though necessarily estimated) probability of overfitting based on established quantitative finance literature (López de Prado), and offers a highly practical, industry-standard protocol for live deployment." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8.8, "brief_justification": "The response is technically sound and well-structured. It identifies key risks accurately and provides a mathematically rigorous explanation of multiple testing bias (Bonferroni/FDR). The live testing protocol is practical, though the response ends slightly abruptly in the final section." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 2.45, "brief_justification": "Response is severely incomplete - it only provides a fragment of part 3 (capital scaling) and is cut off mid-sentence. It fails to address parts 1 (listing 5+ risks) and 2 (calculating overfitting probability) entirely. The partial content on capital scaling shows some basic correctness but lacks the depth and completeness required." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 0.45, "brief_justification": "The response fails to answer any part of the question. It incorrectly claims the message is cut off when the question is complete with three clear parts: (1) listing at least 5 risks, (2) calculating overfitting probability, and (3) designing a live testing protocol." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8.5, "brief_justification": "Comprehensive coverage of backtest-to-live risks and well-designed phased testing protocol. Part 2 calculation is misleading (conflates false positive rate with overfitting probability), but response correctly acknowledges limitations." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 7.8, "brief_justification": "Clear and accurate with good depth." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 8.2, "brief_justification": "Well-structured response covering all three parts with accurate risks, proper PBO methodology using Lopez de Prado's framework, and a practical multi-phase live testing protocol with specific thresholds and kill-switch criteria." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 7.9, "brief_justification": "Well-structured response covering 12 distinct risks across 3 categories with good depth and examples. Mathematical framework for overfitting is sound (Bailey-López de Prado). However, response is incomplete - cuts off mid-formula in Part 2 and Part 3 (live testing protocol) appears to be missing entirely. This significantly impacts completeness and usefulness." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 8.6, "brief_justification": "Comprehensive coverage of all three parts with accurate, well-documented risks. The overfitting probability uses proper academic frameworks (Bailey et al., López de Prado) though the explanation could be clearer. The phased live testing protocol is highly actionable with specific parameters. Minor扣分 for some ambiguity in the probability calculation and slightly dense explanation in part 2." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8.8, "brief_justification": "Comprehensive response covering all three parts with accurate mathematical calculations (Bonferroni overfitting probability ~99.997%), detailed risk analysis (10 risks beyond required 5), and a structured live testing protocol. Slight deduction for incomplete ending in section 3.1D. Well-structured with tables and equations." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.4, "brief_justification": "Well-structured response covering all three parts with accurate financial concepts. Identifies 5 key risks with explanations. Uses appropriate statistical frameworks (Harvey-Liu haircut Sharpe, Bonferroni) for overfitting probability. Provides actionable multi-phase testing protocol with specific timelines and thresholds. Slight扣分 for not showing exact mathematical calculation for overfitting probability." } ], "meta": { "source": "The Multivac (app.themultivac.com)", "methodology": "10x10 blind peer matrix evaluation", "criteria": "correctness, completeness, clarity, depth, usefulness", "self_judgments": "excluded from rankings", "license": "Open data — cite as: The Multivac (2026)" } }