{ "evaluation_id": "EVAL-20260402-192743", "question_id": "ANALYSIS-011", "question_text": "A SaaS startup shares these metrics: MRR $50K, growth 15% month-over-month, CAC $500, LTV $2,400, churn 5% monthly, burn rate $200K/month, runway 8 months. (1) Is this business healthy? (2) Calculate the LTV:CAC ratio and explain why it's misleading at 5% monthly churn. (3) At what churn rate does this business become viable? (4) They're raising at $20M valuation. Would you invest? Why or why not?", "category": "analysis", "timestamp": "2026-04-02T19:27:43.803Z", "display_date": "Apr 02, 2026", "winner": { "name": "GPT-5.4", "provider": "openrouter", "score": 9.12 }, "avg_score": 7.6833334, "matrix_size": 80, "models_used": [ { "id": "mimo_v2_flash", "name": "MiMo-V2-Flash", "provider": "Xiaomi" }, { "id": "claude_opus_46", "name": "Claude Opus 4.6", "provider": "openrouter" }, { "id": "gpt_5_4", "name": "GPT-5.4", "provider": "openrouter" }, { "id": "deepseek_v4", "name": "DeepSeek V4", "provider": "openrouter" }, { "id": "claude_sonnet_46", "name": "Claude Sonnet 4.6", "provider": "openrouter" }, { "id": "grok_420", "name": "Grok 4.20", "provider": "openrouter" }, { "id": "gpt_oss_120b", "name": "GPT-OSS-120B", "provider": "OpenAI" }, { "id": "gemini_3_flash", "name": "Gemini 3 Flash Preview", "provider": "Google" }, { "id": "gemini_31_pro", "name": "Gemini 3.1 Pro", "provider": "openrouter" } ], "rankings": { "gpt_5_4": { "display_name": "GPT-5.4", "provider": "openrouter", "average_score": 9.12, "score_count": 9, "min_score": 8.45, "max_score": 9.8, "rank": 1 }, "gemini_3_flash": { "display_name": "Gemini 3 Flash Preview", "provider": "Google", "average_score": 8.43, "score_count": 9, "min_score": 7.15, "max_score": 9.45, "rank": 2 }, "grok_420": { "display_name": "Grok 4.20", "provider": "openrouter", "average_score": 8.28, "score_count": 9, "min_score": 6.35, "max_score": 9.8, "rank": 3 }, "claude_opus_46": { "display_name": "Claude Opus 4.6", "provider": "openrouter", "average_score": 8.02, "score_count": 9, "min_score": 6.25, "max_score": 9.8, "rank": 4 }, "mimo_v2_flash": { "display_name": "MiMo-V2-Flash", "provider": "Xiaomi", "average_score": 7.89, "score_count": 9, "min_score": 6.3, "max_score": 9.8, "rank": 5 }, "deepseek_v4": { "display_name": "DeepSeek V4", "provider": "openrouter", "average_score": 7.87, "score_count": 9, "min_score": 6.1, "max_score": 9.8, "rank": 6 }, "claude_sonnet_46": { "display_name": "Claude Sonnet 4.6", "provider": "openrouter", "average_score": 7.42, "score_count": 9, "min_score": 5.35, "max_score": 9, "rank": 7 }, "gpt_oss_120b": { "display_name": "GPT-OSS-120B", "provider": "OpenAI", "average_score": 6.65, "score_count": 9, "min_score": 4.7, "max_score": 8.8, "rank": 8 }, "gemini_31_pro": { "display_name": "Gemini 3.1 Pro", "provider": "openrouter", "average_score": 5.47, "score_count": 8, "min_score": 2.7, "max_score": 7.6, "rank": 9 } }, "judgments": [ { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 6.75, "brief_justification": "" }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 6.65, "brief_justification": "" }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 9.8, "brief_justification": "" }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 6.1, "brief_justification": "" }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 7.55, "brief_justification": "The response provides excellent, deep analysis with clear formatting, but it cuts off mid-sentence and completely fails to answer the final question about valuation and investment." }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 6.35, "brief_justification": "" }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 4.7, "brief_justification": "" }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 7.15, "brief_justification": "" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 4.55, "brief_justification": "The response correctly identifies the business as unhealthy and accurately notes that 5% monthly churn compounds to ~46% annual churn. The 15% MoM growth translating to ~400%+ annualized is roughly correct. However, the response is severely incomplete - it only addresses part of question 1 and cuts off abruptly. It doesn't calculate the LTV:CAC ratio (4.8x), doesn't explain why it's misleading at 5% churn, doesn't determine the viable churn rate, and doesn't provide investment analysis on the $2" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 9.2, "brief_justification": "Excellent, well-structured response that correctly calculates LTV:CAC at 4.8x, accurately explains why it's misleading at 5% monthly churn (annual retention ~54%, payback timing vs burn, reacquisition costs), provides reasonable churn viability benchmarks (2-3%), and gives a well-reasoned investment decision. The analysis of the 33x ARR multiple, burn/runway concerns, and growth quality is thorough. Minor nitpick: could have reverse-engineered ARPU from the given LTV and churn to verify consiste" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 7.75, "brief_justification": "The analysis is well-structured and covers all four parts thoroughly. However, there are some correctness issues: (1) The LTV:CAC misleading explanation could be stronger—the key insight is that the stated LTV of $2,400 may not be internally consistent with 5% churn (if ARPU is ~$50/mo, then LTV = $50/0.05 = $1,000, not $2,400, suggesting the given LTV is already overstated or assumes different ARPU). The response notices the 20-month average lifetime but doesn't fully reconcile the inconsistenc" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 7.7, "brief_justification": "The response is well-structured, thorough, and provides excellent practical analysis. However, there are notable correctness issues: (1) The churn analysis confuses simple vs compound annual churn (5% monthly ≈ 46% annual, not 60-80% as stated in part 4, though 60% is mentioned elsewhere as a simple approximation). (2) The LTV:CAC discussion in part 2 is muddled - the response correctly calculates 4.8:1 from given numbers but then creates a confusing hypothetical about ARPU being $50 that underm" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 7.4, "brief_justification": "The response demonstrates strong analytical depth and creative presentation, but contains several errors and inconsistencies. The burn multiple calculation is wrong: net new MRR of $7,500/month means $90K annualized net new ARR, and net burn is $150K/month (not $200K, since $50K is revenue), giving a burn multiple of ~20x, not 26.7x. The LTV:CAC threshold analysis confuses itself - first showing 8% monthly satisfies >3x, then pivoting to say the benchmark assumes annual churn without clear justi" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 8.95, "brief_justification": "Excellent, well-structured analysis that correctly identifies the core problems. The LTV:CAC calculation of 4.8x is correct, and the explanation of why it's misleading at high churn is insightful. The churn viability analysis is reasonable, though the back-calculation of ARPU could be more explicit about gross margin assumptions. Minor issue: stating unit economics are 'broken' when LTV:CAC is 4.8x is slightly contradictory—the point is that the LTV itself is unreliable, which they do explain we" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 5.65, "brief_justification": "The response starts well with a structured TL;DR and reasonable investment analysis, but contains significant analytical errors. The claim that LTV:CAC of 4.8 'over-states true lifetime value' is confused - the LTV already incorporates the 5% churn, so the ratio doesn't overstate anything; rather, the issue is that 5% monthly churn makes the LTV itself quite low and the business unsustainable. The churn threshold calculation is backwards in logic - stating the company is 'under the 8% mark' as i" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.95, "brief_justification": "Excellent, well-structured analysis. The LTV:CAC calculation is correct at 4.8x, and the explanation of why it's misleading is insightful. The burn multiple calculation has a minor issue (burn multiple should be net burn / net new ARR, which would be different), and the churn viability threshold discussion could have been more rigorous with explicit math showing the growth ceiling formula. The investment analysis is strong and well-reasoned. The annualized churn approximation of 60% is slightly " }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 2.7, "brief_justification": "The response is only a partial answer. It correctly flags high churn as a serious issue and reasonably converts 5% monthly churn to about 46% annual churn, but it fails to answer most of the prompt: no LTV:CAC calculation, no discussion of why LTV is misleading, no viability churn threshold, and no investment view." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 6.25, "brief_justification": "Clear and analytically strong in parts, especially on churn and LTV:CAC caveats, but incomplete: it cuts off before answering parts (3) and (4) fully. Some claims are overstated or based on unsupported assumptions like cost growth and NRR framing." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 7.95, "brief_justification": "Clear and mostly sound, with good investor-oriented reasoning. Main issues: annual churn conversion is understated, the viability threshold is framed somewhat arbitrarily, and LTV treatment mixes given LTV with assumed gross margin and ARPU assumptions." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 6.3, "brief_justification": "Well-structured and fairly thorough, but several core claims are flawed or overstated: 15% MoM growth can coexist with 5% churn if acquisition is strong, the LTV discussion is confused, and the 'viable churn rate' answer is not rigorously derived from the given data. Investment skepticism is directionally reasonable, especially on burn/runway and valuation, but some conclusions rely on unsupported assumptions." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 5.35, "brief_justification": "Strong framing and some valid concerns about high churn/runway, but key calculations are flawed or inconsistent. Burn multiple is miscomputed by using total burn instead of net burn and annualization is wrong, LTV discussion mixes monthly/annual concepts inaccurately, and the answer is incomplete since section (3) is cut off and no clear investment conclusion is provided." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 7.7, "brief_justification": "Well-structured and insightful, with strong discussion of churn, runway, and valuation. However, it overstates some conclusions: annual churn conversion is wrong (5% monthly is ~54% annual logo churn), claims of 'negative unit economics' are not supported by the stated LTV:CAC, and the churn threshold for viability is partly based on inferred ARPU/gross margin assumptions not clearly justified. Still, it is practical and mostly directionally sound." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 5.2, "brief_justification": "Clear structure and some correct calculations, but the response is truncated and incomplete. It correctly gets LTV:CAC ≈ 4.8 and notes 5% monthly churn is high, but its claim that LTV is overstated because of high churn is conceptually weak since higher churn usually lowers formulaic LTV. The 'viable churn rate' analysis is also under-justified because ARPU/customer count is not provided, so the churn threshold cannot be reliably derived from the given data alone." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 7.55, "brief_justification": "Well-structured and thoughtful, with strong discussion of churn, valuation, and fundraising risk. However, it contains notable inaccuracies: the burn multiple is miscalculated from the given data, the annual churn implication is overstated, and the answer to the specific churn threshold for viability is more heuristic than derived from the provided metrics." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 7.05, "brief_justification": "Accurate analysis but incomplete, lacks calculations and investment rationale." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 9, "brief_justification": "Comprehensive and accurate analysis with clear explanations and deep insights into the startup's viability." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8.8, "brief_justification": "Detailed, accurate analysis with clear structure and actionable insights." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 9.8, "brief_justification": "Highly accurate, thorough, and insightful analysis with clear actionable insights." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 8.8, "brief_justification": "Highly accurate and thorough analysis with deep insights, though slightly dense presentation." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 9, "brief_justification": "Comprehensive, accurate, and insightful analysis with clear actionable insights." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8.8, "brief_justification": "Accurate and thorough analysis with clear insights, though slightly verbose." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 9.45, "brief_justification": "Highly accurate and thorough analysis with clear, actionable insights." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 7.6, "brief_justification": "Strong start with correct identification of the leaky bucket problem, but the response is incomplete and cuts off abruptly." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 9, "brief_justification": "The response is highly accurate, comprehensive, and insightful. It correctly identifies the business as unhealthy due to the burn rate and high churn, provides a nuanced critique of the LTV:CAC ratio (explaining how it's misleading due to payback period, margin, and growth costs), and calculates a viable churn rate based on multiple dimensions (unit economics, NRR, and runway). The analysis is structured and actionable." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 9, "brief_justification": "Comprehensive and accurate analysis covering all sub-questions with strong benchmarks, correct math, and clear investment rationale." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 8.8, "brief_justification": "Accurate calculations and strong analysis, minor omission in burn rate context." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 8.15, "brief_justification": "Strong analysis with clear structure and depth, but some minor factual inaccuracies in calculations and incomplete answers to parts (3) and (4)." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 9, "brief_justification": "Excellent, comprehensive analysis. Correctly identifies the core issues (churn, burn, misleading LTV), performs relevant calculations, and provides actionable investment advice with clear thresholds. Minor deduction in correctness for a simplified LTV calculation assumption, but overall highly accurate." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 7.15, "brief_justification": "Accurate on key points but incomplete response; cutoff mid-sentence and missing final calculation for viability threshold." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 9.2, "brief_justification": "Accurate calculations, thorough analysis of health and viability, clear structure, insightful on churn's impact, and actionable investment advice." }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 5.1, "brief_justification": "The response is cut off and incomplete - it only addresses part of question 1 and fails to answer questions 2, 3, and 4 entirely. The partial content is accurate (correctly identifies the churn problem and calculates ~46% annual churn), but the response is severely incomplete, missing LTV:CAC calculation, viability analysis, and investment recommendation." }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 9.2, "brief_justification": "Exceptionally thorough analysis with strong quantitative reasoning. The response correctly identifies the existential churn problem, properly deconstructs why the LTV:CAC ratio is misleading through multiple lenses (time-value, gross margin adjustment, effective CAC calculation), and provides rigorous multi-dimensional viability analysis. The leaky bucket illustration, payback period calculation, and NRR framework demonstrate genuine depth. Minor note: the response appears truncated before compl" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 9.2, "brief_justification": "Exceptionally thorough analysis covering all four questions with accurate calculations (LTV:CAC = 4.8x, annual retention ~54%, average lifetime = 20 months). The response correctly identifies the core tension between strong growth metrics and destructive churn/burn dynamics. Depth is excellent—goes beyond surface ratios to explain timing of cash recovery, replacement treadmill effect, and cohort analysis needs. The viability churn threshold analysis is well-reasoned with appropriate benchmarks. " }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 7.75, "brief_justification": "Well-structured and clear analysis with good practical insights. The LTV:CAC calculation correctly identifies the misleading nature at 5% churn. However, the churn viability calculation in part 3 has an error - it uses $50 as MRR per customer without establishing this figure clearly, and the LTV formula should use average revenue per account divided by churn rate. The valuation analysis is solid but could go deeper on growth-adjusted multiples. The burn rate concern is well-articulated. Minor in" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 8.55, "brief_justification": "Strong, well-structured analysis covering all four questions with solid financial reasoning. The LTV:CAC misleading explanation is thoughtful, though the ARPU derivation introduces assumptions not in the original data. The churn viability section is somewhat circular but arrives at reasonable conclusions. The valuation analysis with ARR multiples is practical and actionable. Minor issue: the response initially confuses itself on the LTV calculation before self-correcting, which slightly undermin" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 8.8, "brief_justification": "Excellent analysis covering all four questions with accurate calculations and sound reasoning. The LTV:CAC explanation correctly identifies why 4.8x is misleading at high churn. Churn viability thresholds are well-reasoned. The investment thesis is clear and well-justified with specific counter-offer terms. Minor gap: doesn't explicitly show the LTV formula derivation or note that the stated $2,400 LTV may already incorporate the 5% churn assumption, and the 'back-loaded value' claim slightly mi" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 7, "brief_justification": "The response is well-structured and covers the key questions with good clarity. The LTV:CAC calculation (4.8) is correct. However, the churn viability analysis contains errors - the math for 'churn < ARPU/CAC ≈ 24%/mo' is incorrect and the logic is confused. The response also appears truncated, cutting off mid-table. The valuation analysis at 33x ARR is sound and the investment recommendation is well-reasoned. The response would benefit from clearer churn threshold calculations (ideally showing " }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.55, "brief_justification": "Strong analysis with accurate calculations (LTV:CAC 4.8x correct, burn multiple correct). The explanation of why LTV:CAC is misleading at high churn is insightful. Minor issue: the payback period calculation (4.2 months) is correct but the reasoning around 'probability of reaching average lifetime' could be more rigorous. The churn viability threshold analysis is practical. The valuation analysis correctly identifies the 33x ARR multiple concern. Overall a well-structured, actionable response th" }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 5.25, "brief_justification": "The partial response accurately identifies the high churn issue and correctly notes it's unhealthy, but it is incomplete as it only addresses part of question (1) and ignores (2), (3), and (4) entirely. Lacks calculations for LTV:CAC, viability analysis, or investment reasoning." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 7.65, "brief_justification": "Strong analysis with good depth and clear structure, but incomplete as it cuts off before answering questions 3 and 4, and some calculations (e.g. NRR math) contain minor errors." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8.65, "brief_justification": "Response is factually accurate on SaaS metrics, provides thorough analysis across all questions with clear structure, good benchmarks, and insightful caveats on why LTV:CAC is misleading and the risks involved. Minor deduction for depth as it could explore exact viability math or cohort implications slightly more." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 6.75, "brief_justification": "Response provides structured analysis but contains several calculation errors (e.g. LTV formula misuse, incorrect gross margin application to LTV, wrong churn solving) and inconsistent assumptions about LTV value. Valuation multiple is misstated (should be on ARR). Still covers all questions with decent structure." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 7.8, "brief_justification": "The response is mostly accurate with good structure and insight, but contains some factual errors around LTV formulas, ARPU assumptions, and churn calculations that reduce correctness." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 7.4, "brief_justification": "Response provides detailed, structured analysis with good use of tables and calculations, but contains several mathematical errors (e.g. incorrect burn multiple formula and churn threshold math) and fails to fully address question part (4) about investing at $20M valuation." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 7, "brief_justification": "Response is mostly accurate but contains calculation errors on churn thresholds and LTV formula; provides structured analysis with good tables but is incomplete as it cuts off mid-sentence. Valuation multiple and investment reasoning are insightful yet partially flawed on benchmarks." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 7.75, "brief_justification": "Response is mostly accurate with good structure and insight on SaaS metrics, but contains some factual issues like incorrect ARPU calculation, imprecise LTV formula application, and overly simplistic churn viability thresholds." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 7.1, "brief_justification": "Accurate calculations and solid analysis of LTV:CAC and churn impact, but fails to give a clear churn threshold answer and omits a direct investment recommendation, reducing completeness and overall usefulness." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8.45, "brief_justification": "Accurate calculations and solid coverage of all four questions, clear structure, good insight into churn impact and valuation risk, though could include more quantitative thresholds for viability." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 7.35, "brief_justification": "The answer covers all parts clearly and offers useful insights, but contains a flawed churn viability calculation and makes unsupported margin assumptions, reducing factual accuracy." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 6.4, "brief_justification": "The answer covers all questions and is well-structured, but contains some inaccurate or confusing calculations (e.g., churn impact, LTV assumptions) and vague churn viability thresholds, limiting its overall accuracy and practical value." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 6.3, "brief_justification": "The response offers detailed analysis and clear explanations for health, LTV:CAC, and churn impact, but it fails to finish the churn viability calculation and does not address the investment decision, leaving key questions unanswered." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 6.35, "brief_justification": "The answer covers all questions and is clearly organized, but contains several factual errors (e.g., ARPU calculation, churn thresholds) that undermine accuracy and practical value." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.45, "brief_justification": "Accurate calculations and solid analysis covering all parts, clear presentation, good depth and actionable advice." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 6.65, "brief_justification": "The response is incomplete as it cuts off after the first section, failing to address the LTV:CAC calculation, the viability threshold, or the investment decision." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 9.8, "brief_justification": "The response is exceptionally high quality. It correctly identifies the existential threat of the burn rate and churn, provides a sophisticated breakdown of why LTV:CAC is misleading (including gross margin and time-value adjustments), and uses logical modeling to determine viability. It was cut off at the very end due to length, but the core content is complete." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 9.8, "brief_justification": "The response is exceptionally clear, accurate, and provides a sophisticated analysis of the SaaS metrics. It correctly identifies the 'leaky bucket' problem and provides the specific LTV:CAC calculation and churn benchmarks requested." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 9.8, "brief_justification": "The response is exceptionally accurate and provides a professional-grade financial analysis. It correctly identifies the 400x MRR valuation as a major red flag, accurately calculates the churn impact on LTV, and provides a realistic target for viability. The distinction between gross and net LTV adds great depth." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 9.55, "brief_justification": "The response provides a high-quality, professional analysis of the SaaS metrics. It correctly identifies the 'leaky bucket' problem, accurately calculates the ARR multiple (33x), and provides a sophisticated explanation of why the LTV:CAC ratio is misleading due to the relationship between churn and customer lifetime. The investment decision is well-reasoned based on the provided data." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 9, "brief_justification": "The evaluation of the business health is excellent, particularly the calculation of the Burn Multiple and the breakdown of why 5% monthly churn is catastrophic. The response correctly identifies that the LTV:CAC ratio is misleading because it ignores the time value of money and the high probability of early churn. It loses a point for completeness as it cuts off mid-sentence at the end." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 9.8, "brief_justification": "The response is exceptionally accurate and provides a professional-grade analysis of the SaaS metrics. It correctly identifies the 'leaky bucket' problem, accurately calculates the ARR multiple (33x), and provides a sophisticated explanation of why LTV is misleading when churn is high. The advice regarding the valuation and investment risk is sound and reflects real-world venture capital logic." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 6.8, "brief_justification": "The response is cut off at the end, failing to complete the detailed walk-through. However, the TL;DR section correctly identifies the LTV:CAC ratio and the valuation concerns. The analysis of why the LTV:CAC is misleading is slightly weak (it claims it 'overstates' LTV, but LTV is calculated using churn, so the ratio already accounts for it; the real issue is the payback period and cash flow). The churn viability calculation is also mathematically questionable in its presentation." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 4.85, "brief_justification": "Response correctly identifies the churn problem but is severely incomplete - it only addresses part of question 1 and doesn't calculate LTV:CAC ratio, explain why it's misleading, determine viable churn rate, or provide an investment recommendation. The response appears to be cut off." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 7.5, "brief_justification": "Accurate calculations and strong analysis of LTV:CAC misleading factors. However, the response is incomplete - it cuts off mid-calculation and never answers question (4) about whether to invest at $20M valuation. The analysis up to that point is excellent but leaves the evaluation unfinished." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 9.2, "brief_justification": "Accurate calculations and analysis covering all four questions. Well-structured with clear explanations of why LTV:CAC is misleading at 5% churn. Provides useful benchmarks and clear investment recommendation. Slight扣分 for not calculating payback period explicitly." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 8.55, "brief_justification": "Accurate calculations and analysis covering all four questions. Well-structured with clear headings. Explains why LTV:CAC is misleading at 5% churn, provides specific viable churn rate target, and gives clear investment recommendation with reasoning. Actionable insights and practical considerations." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 8.15, "brief_justification": "Well-structured response covering all four questions with clear calculations and reasoning. The analysis of the growth-churn conflict and 'leaky bucket' dynamic is insightful. Some minor issues: the LTV explanation is somewhat confusing (the $2,400 LTV should already incorporate the 20-month lifetime), and the 2-4x ARR valuation suggestion seems too low for context. Overall accurate, actionable, and demonstrates good SaaS financial understanding." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 6.8, "brief_justification": "Well-structured analysis with good depth, but contains a significant error in burn multiple calculation (incorrectly calculates net new MRR without accounting for churned revenue). Also incomplete - cuts off mid-sentence and fails to answer question 4 about investing at $20M valuation." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 8.6, "brief_justification": "Accurate calculations and analysis covering all four questions. Well-structured with clear explanations of why high churn makes LTV misleading. Provides specific, actionable thresholds for viability and clear investment recommendation with conditions." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 7.55, "brief_justification": "Well-structured response with correct LTV:CAC calculation (4.8) and reasonable investment conclusion. However, the churn threshold calculation in point 3 contains mathematical errors (the 24% threshold doesn't align with the LTV formula), and the response could better explain why 5% monthly churn specifically undermines the LTV:CAC ratio's reliability." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.8, "brief_justification": "Accurate calculations and thorough analysis covering all four questions. Well-structured with clear explanations of why LTV:CAC is misleading at high churn. Provides specific thresholds for viability and a reasoned investment recommendation with market context." } ], "meta": { "source": "The Multivac (app.themultivac.com)", "methodology": "10x10 blind peer matrix evaluation", "criteria": "correctness, completeness, clarity, depth, usefulness", "self_judgments": "excluded from rankings", "license": "Open data — cite as: The Multivac (2026)" } }