{ "evaluation_id": "EVAL-20260402-141544", "question_id": "CODE-022", "question_text": "Implement the OAuth 2.0 Authorization Code flow with PKCE (Proof Key for Code Exchange) from scratch in Python. Include: code verifier/challenge generation, authorization URL construction, token exchange, token refresh, and secure token storage. Explain why PKCE prevents authorization code interception attacks.", "category": "code", "timestamp": "2026-04-02T14:15:44.483Z", "display_date": "Apr 02, 2026", "winner": { "name": "Grok 4.20", "provider": "openrouter", "score": 8.9 }, "avg_score": 7.419, "matrix_size": 89, "models_used": [ { "id": "minimax_m25", "name": "MiniMax M2.5", "provider": "openrouter" }, { "id": "mimo_v2_flash", "name": "MiMo-V2-Flash", "provider": "Xiaomi" }, { "id": "gpt_oss_120b", "name": "GPT-OSS-120B", "provider": "OpenAI" }, { "id": "claude_opus_46", "name": "Claude Opus 4.6", "provider": "openrouter" }, { "id": "gemini_31_pro", "name": "Gemini 3.1 Pro", "provider": "openrouter" }, { "id": "claude_sonnet_46", "name": "Claude Sonnet 4.6", "provider": "openrouter" }, { "id": "grok_420", "name": "Grok 4.20", "provider": "openrouter" }, { "id": "deepseek_v4", "name": "DeepSeek V4", "provider": "openrouter" }, { "id": "gemini_3_flash", "name": "Gemini 3 Flash Preview", "provider": "Google" }, { "id": "gpt_5_4", "name": "GPT-5.4", "provider": "openrouter" } ], "rankings": { "grok_420": { "display_name": "Grok 4.20", "provider": "openrouter", "average_score": 8.9, "score_count": 8, "min_score": 8, "max_score": 9.8, "rank": 1 }, "gemini_3_flash": { "display_name": "Gemini 3 Flash Preview", "provider": "Google", "average_score": 8.24, "score_count": 9, "min_score": 7.45, "max_score": 8.8, "rank": 2 }, "gpt_5_4": { "display_name": "GPT-5.4", "provider": "openrouter", "average_score": 7.75, "score_count": 9, "min_score": 5.25, "max_score": 8.8, "rank": 3 }, "deepseek_v4": { "display_name": "DeepSeek V4", "provider": "openrouter", "average_score": 7.58, "score_count": 9, "min_score": 6.5, "max_score": 9.4, "rank": 4 }, "gpt_oss_120b": { "display_name": "GPT-OSS-120B", "provider": "OpenAI", "average_score": 7.57, "score_count": 9, "min_score": 5.45, "max_score": 8.6, "rank": 5 }, "minimax_m25": { "display_name": "MiniMax M2.5", "provider": "openrouter", "average_score": 7.53, "score_count": 9, "min_score": 3.85, "max_score": 9.4, "rank": 6 }, "mimo_v2_flash": { "display_name": "MiMo-V2-Flash", "provider": "Xiaomi", "average_score": 7.39, "score_count": 9, "min_score": 4.8, "max_score": 9.4, "rank": 7 }, "claude_sonnet_46": { "display_name": "Claude Sonnet 4.6", "provider": "openrouter", "average_score": 7.14, "score_count": 9, "min_score": 4.1, "max_score": 9, "rank": 8 }, "claude_opus_46": { "display_name": "Claude Opus 4.6", "provider": "openrouter", "average_score": 6.62, "score_count": 9, "min_score": 2.85, "max_score": 9.25, "rank": 9 }, "gemini_31_pro": { "display_name": "Gemini 3.1 Pro", "provider": "openrouter", "average_score": 5.47, "score_count": 9, "min_score": 2.3, "max_score": 8.8, "rank": 10 } }, "judgments": [ { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 6.25, "brief_justification": "The response provides accurate code and a good explanation of PKCE, but it cuts off abruptly before covering token refresh and secure token storage." }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 6.3, "brief_justification": "The code correctly implements the PKCE flow using standard libraries. However, the response is truncated at the end and completely fails to explain why PKCE prevents authorization code interception attacks, as requested." }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 6.4, "brief_justification": "The provided code is highly accurate, secure, and well-explained, but the response is truncated mid-code, missing the token exchange completion and token refresh implementations." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 2.85, "brief_justification": "The PKCE explanation is mostly accurate and the writing is organized, but the implementation is severely incomplete and cut off mid-property. It does not deliver authorization URL construction, token exchange, refresh, or secure token storage as requested, so it has very low practical usefulness despite decent conceptual framing." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 2.3, "brief_justification": "The response is truncated and fails to provide the requested implementation details. What is present about PKCE is broadly accurate, but it is incomplete and not actionable." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 4.1, "brief_justification": "The PKCE explanation and initial code are mostly accurate, but the response is truncated mid-implementation and does not actually include authorization URL construction, token exchange, refresh flow, or secure token storage, making it only partially correct and of limited practical use." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 8, "brief_justification": "Clear and mostly accurate implementation covering PKCE generation, auth URL, token exchange, refresh, storage, and PKCE rationale. Minor issues: state is generated but not validated in the callback flow, secure storage is only basic file permissions rather than robust encrypted storage, and the example asks for the raw code rather than parsing the redirect URL despite the comment. Overall strong and practical." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 6.8, "brief_justification": "Generally accurate and clearly structured, with working PKCE mechanics and token exchange/refresh. However, the explanation of why PKCE prevents interception is incomplete/truncated, state is generated but not properly validated, and the claimed secure token storage is not actually secure beyond a note. Useful as a starting point, but not fully production-safe." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 5.45, "brief_justification": "Well-structured and mostly accurate on PKCE basics and secure storage ideas, but the response is incomplete: the OAuth2Client implementation is truncated before token exchange, refresh, and full end-to-end flow, so it does not fully satisfy the request." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 7.8, "brief_justification": "Clear and mostly accurate PKCE walkthrough with working core pieces and a good explanation of interception protection. It covers verifier/challenge generation, auth URL, code exchange, refresh, and mentions secure storage, but the storage implementation is not actually secure, state handling is only mentioned rather than implemented, and some production details like permission setting, redirect handling, token validation, and robust error handling are missing." }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 7.45, "brief_justification": "" }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 3.85, "brief_justification": "The response is partially correct and clearly structured, but it is incomplete: it cuts off before finishing token refresh and secure storage, so key requested parts are missing. It also contains some technical issues, such as incorrect claims about base64url output characters in verifier generation and questionable guidance around using Basic auth/client secrets in a PKCE context." }, { "judge_key": "gpt_5_4", "judge_name": "GPT-5.4", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 4.8, "brief_justification": "Covers core PKCE pieces and token exchange/refresh, but the response is truncated, missing the PKCE security explanation and full token summary logic. It also labels plain JSON file storage as 'secure-ish' without actual secure storage measures, and omits robust state validation and browser launch steps." }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 7.85, "brief_justification": "The response provides a well-structured, mostly correct implementation of OAuth 2.0 Authorization Code flow with PKCE. The PKCE helpers (verifier/challenge generation) are correctly implemented per RFC 7636. The authorization URL construction, token exchange, refresh flow, state parameter validation, and token storage are all present and well-explained. The plain-English explanation of why PKCE prevents interception attacks is clear and accurate. However, the response is truncated - the refresh_" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 4.3, "brief_justification": "The response starts well with a clear explanation of why PKCE prevents authorization code interception attacks, but it is severely incomplete. It cuts off mid-sentence during the PKCE explanation and never delivers the promised implementation code for code verifier/challenge generation, authorization URL construction, token exchange, token refresh, or secure token storage. The portion that exists is accurate and clearly written, but the response fails to address the majority of what was asked." }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 6.5, "brief_justification": "The response provides a well-structured and clearly explained implementation of PKCE core cryptography, OAuth configuration, and token management with good security practices (timing-safe comparison, proper RFC 7636 compliance, CSRF state parameter). The PKCE explanation is accurate and intuitive. However, the code is clearly truncated mid-implementation - the TokenSet class is cut off at `@class`, and critical sections like authorization URL construction, token exchange (HTTP request to token e" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 8.2, "brief_justification": "The implementation is well-structured and covers all requested components: PKCE generation, authorization URL construction, token exchange, token refresh, and token storage. The explanation of why PKCE prevents interception attacks is clear and accurate. Minor issues: token storage to a JSON file with chmod 600 is not truly 'secure' (no encryption, not cross-platform for Windows), the code verifier generation could use secrets.token_urlsafe directly for simplicity, and the implementation stores " }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 6.6, "brief_justification": "The implementation covers the core PKCE flow components (code verifier/challenge generation, authorization URL, token exchange, refresh, and basic token storage) and is mostly correct. However, the explanation of why PKCE prevents interception attacks is truncated/incomplete - it cuts off mid-sentence. The token storage is plaintext JSON files with only comments about encryption, not actual secure storage. The state parameter is generated but never validated on callback, which is a security over" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 7.05, "brief_justification": "The response is well-structured and clearly written, with correct PKCE generation, secure token storage using AES-GCM, and a good explanation of why PKCE prevents interception attacks. However, the code is critically incomplete - the `exchange_code_for_token` method is cut off mid-line, and the `refresh_token` method is entirely missing. This means two of the five explicitly requested features (token exchange and token refresh) are not fully implemented. The PKCE helper and token storage are cor" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.45, "brief_justification": "The implementation is correct and well-structured, covering all requested components: PKCE pair generation, authorization URL construction, token exchange, token refresh, and token storage. The explanation of why PKCE prevents interception attacks is clear and accurate. Minor gaps: the state parameter is mentioned only in security considerations rather than being implemented, the secure token storage is essentially just writing to a plain JSON file (though this is acknowledged), and there's no a" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 7.65, "brief_justification": "The response provides a well-structured, educational implementation covering code verifier/challenge generation, authorization URL construction, token exchange, and begins explaining token refresh and secure storage. The code is correct and follows the PKCE spec (S256 method, proper base64url encoding). The explanation of why PKCE prevents interception attacks is present but brief. However, the response appears truncated - the token exchange section cuts off mid-JSON, and the token refresh and s" }, { "judge_key": "claude_opus_46", "judge_name": "Claude Opus 4.6", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 6.45, "brief_justification": "The implementation covers the core PKCE flow components (verifier/challenge generation, authorization URL, token exchange, refresh, basic storage) and is mostly correct. However, the code is truncated at the end (print_token_summary is incomplete), the critical explanation of WHY PKCE prevents authorization code interception attacks is completely missing (a key part of the question), state validation is not performed, token storage is just plain JSON with no encryption (acknowledged but not addr" }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 5.25, "brief_justification": "The response provides a solid start with correct PKCE implementation and token storage, but it cuts off abruptly, leaving the code incomplete and unusable. It also lacks a detailed explanation of how PKCE prevents interception attacks." }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 6.3, "brief_justification": "The response starts exceptionally well with a clear explanation of PKCE and accurate code for verifier/challenge generation, but it cuts off abruptly, missing the majority of the requested implementation (URL construction, token exchange, refresh, and storage)." }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 6.45, "brief_justification": "The response provides an excellent explanation of PKCE and correct cryptographic implementations, but it cuts off abruptly, missing the requested authorization URL construction, token exchange, refresh, and storage." }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 9.8, "brief_justification": "The response provides a complete, accurate, and highly practical Python implementation of the OAuth 2.0 PKCE flow. The explanation of PKCE is clear and insightful, though state validation in the callback could have been briefly mentioned for maximum depth." }, { "judge_key": "gemini_31_pro", "judge_name": "Gemini 3.1 Pro", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 6.5, "brief_justification": "The code correctly implements the PKCE flow, but the response cuts off before finishing the explanation. Additionally, the 'secure token storage' requirement is only addressed with comments rather than an actual secure implementation." }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8, "brief_justification": "The implementation is technically correct and well-structured, covering PKCE generation, authorization URL construction, token exchange, refresh flow, and storage. The code is clean and readable with good comments. However, the response appears truncated (cuts off mid-sentence during refresh token handling), missing the local callback server implementation, the PKCE security explanation, and the main flow orchestration. The code_verifier generation has a minor issue (token_urlsafe with length=64" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 7.3, "brief_justification": "The response demonstrates strong correctness in its PKCE explanation, threat model diagrams, and the code it does provide. The ASCII diagrams clearly illustrate why PKCE prevents interception attacks, and the PKCEParameters class is well-implemented with proper RFC references. However, the response is clearly incomplete - it cuts off mid-sentence in the code_challenge_method property, and the promised sections (authorization URL construction, token exchange, token refresh, secure token storage) " }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 3.95, "brief_justification": "The response is severely incomplete - it cuts off mid-sentence during the PKCE explanation and provides no actual Python implementation code despite the question explicitly requesting code for code verifier/challenge generation, authorization URL construction, token exchange, token refresh, and secure token storage. The partial explanation of PKCE is accurate but the response fails to deliver on the core requirements." }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 7.6, "brief_justification": "The implementation is technically sound with correct PKCE mechanics (S256 challenge, proper base64url encoding, secrets module usage). Code verifier generation is slightly off (token_urlsafe(96)[:128] doesn't guarantee RFC 7636 compliance but works). Includes callback server, token refresh, and storage. The PKCE explanation appears to be cut off mid-sentence, losing completeness and depth. Token storage uses plaintext JSON file with only a comment about encryption rather than demonstrating secur" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8, "brief_justification": "The implementation is well-structured, technically accurate, and covers most required components with good explanations. The PKCE security rationale is clear and correct. However, the response is cut off mid-sentence at the token exchange method, leaving that section and the token refresh incomplete. The secure token storage using AES-GCM is a strong addition. Deductions for the truncated code that leaves key functionality unimplemented." }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.45, "brief_justification": "The implementation is technically correct with proper PKCE generation (SHA256, base64url encoding, padding removal), correct token exchange and refresh flows, and a clear explanation of why PKCE prevents interception attacks. The code is well-structured and readable. Minor gaps: no state parameter in the implementation (only mentioned in security notes), file permissions aren't actually set (chmod), and token storage is acknowledged as insecure without a real alternative shown. The explanation o" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 7.45, "brief_justification": "The response is well-structured and technically accurate for the parts it covers. Code verifier/challenge generation, authorization URL construction, and token exchange are implemented correctly. However, the response appears truncated - token refresh, secure token storage, and the PKCE security explanation are either missing or cut off. The warning about production use is appropriate. The code quality is good with proper use of secrets module and SHA-256 for PKCE. Deducted points for incomplete" }, { "judge_key": "claude_sonnet_46", "judge_name": "Claude Sonnet 4.6", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 7.6, "brief_justification": "The implementation is technically correct and covers most required components: PKCE generation, authorization URL construction, token exchange, refresh, and basic storage. The code uses only standard library which is educational. However, the response appears truncated (print_token_summary function is cut off), the explanation of why PKCE prevents interception attacks is missing entirely (a key requirement), and token storage is acknowledged as insecure without providing a more secure alternativ" }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8.45, "brief_justification": "The implementation is factually correct, logically sound, and covers all requested components with good explanations. Minor issues include incomplete refresh token response handling and placeholder endpoints, but overall high quality." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 6.05, "brief_justification": "The response provides a strong explanation of PKCE and a solid start to the PKCE class, but the code is incomplete (cuts off mid-property), lacks full flow implementation (auth URL, token exchange, refresh, storage), and has minor inaccuracies like the token_urlsafe length handling." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 5.85, "brief_justification": "The response starts correctly by explaining the PKCE vulnerability and solution but is incomplete: it cuts off mid-sentence after listing only the first PKCE step and provides no actual code implementation for any of the requested components (verifier generation, auth URL, token exchange, refresh, or storage). The explanation is accurate but truncated." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 7.65, "brief_justification": "The response provides a solid, well-structured implementation of PKCE with clear explanations and secure practices. However, the code is incomplete (cuts off mid-class), lacks the full token exchange, refresh, and storage functions mentioned in the query, and the server-side verification method is not used in a client flow." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 7.45, "brief_justification": "The implementation covers all requested components with mostly correct code and good explanations. PKCE explanation is accurate but incomplete (cuts off). Token storage is noted as insecure in comments. Callback handling has some issues with shared state in the handler class. Overall solid educational example." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8.45, "brief_justification": "The implementation is factually accurate, logically sound, and covers all requested components with clear structure and good explanations. Minor issues include an incomplete code snippet (RuntimeError cut off) and reliance on an external 'cryptography' library despite claiming 'plain Python/no external OAuth libraries', but overall high quality." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.45, "brief_justification": "The response accurately implements the PKCE flow with all requested components, provides a clear explanation of the security mechanism, and includes practical production advice. Minor deduction for using a plain JSON file in the example despite noting better alternatives, and lacking explicit state parameter in the provided code." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 8.45, "brief_justification": "The response provides a mostly accurate, clear, and well-structured implementation covering all requested components with good explanations. Minor issues include incomplete token refresh and storage code, but overall educational value is high." }, { "judge_key": "grok_420", "judge_name": "Grok 4.20", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 8.6, "brief_justification": "The implementation is factually correct, uses only stdlib as claimed, covers all requested components including PKCE, auth URL, token exchange/refresh and basic storage. The explanation of PKCE is present though brief. Code has minor issues (incomplete print_token_summary, global server state mutation, missing import for urllib.error) but overall high quality and educational." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8.8, "brief_justification": "Comprehensive and accurate implementation with clear explanation of PKCE security benefits." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 8.6, "brief_justification": "Detailed and accurate explanation of PKCE with a comprehensive Python implementation. Missing minor details in the code snippet." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 7.05, "brief_justification": "Accurate explanation of PKCE but lacks full implementation details." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 9, "brief_justification": "Comprehensive implementation with clear explanation of PKCE security benefits." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 9.6, "brief_justification": "Comprehensive implementation with clear explanation of PKCE security benefits." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8.6, "brief_justification": "Highly accurate and complete with clear explanations and practical implementation." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.8, "brief_justification": "Accurate, thorough, and clear with practical implementation and security considerations." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 8.65, "brief_justification": "Detailed and accurate implementation with clear explanations." }, { "judge_key": "deepseek_v4", "judge_name": "DeepSeek V4", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 9.35, "brief_justification": "Comprehensive and accurate implementation with clear explanations and practical value." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 7, "brief_justification": "The response provides a solid, well-structured implementation and clear explanation, but the code is cut off before completing the refresh flow and lacks a full usage example, reducing completeness and overall usefulness." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 4.5, "brief_justification": "The answer explains PKCE well and provides clear code for verifier/challenge generation, but it is incomplete—missing URL construction, token exchange, refresh logic, and secure storage, limiting correctness and completeness." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 5.9, "brief_justification": "Explanation of PKCE is accurate and clear, but the response lacks the required code components (verifier/challenge generation, URL building, token exchange, refresh, secure storage), making it incomplete and less useful." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 5.85, "brief_justification": "The answer correctly explains PKCE and provides solid code for verifier/challenge generation, but it stops short of implementing token exchange, refresh, and secure storage, making it incomplete for a full end-to-end solution." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 8.8, "brief_justification": "Accurate implementation of PKCE flow with all required components, clear structure and explanation, and practical utility; minor depth improvements possible." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 7.8, "brief_justification": "" }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.6, "brief_justification": "Accurate PKCE implementation and clear explanation; covers all required components with practical code, though token storage is simplistic and state handling is only mentioned." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 7.45, "brief_justification": "The response gives accurate code for verifier/challenge, URL building, and token exchange with a solid PKCE explanation, but omits full token refresh and secure storage implementations, limiting completeness while remaining clear and fairly useful." }, { "judge_key": "gpt_oss_120b", "judge_name": "GPT-OSS-120B", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 7.4, "brief_justification": "Implementation is correct and clear, but missing explicit explanation of PKCE security and full state verification, reducing completeness and depth." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8.8, "brief_justification": "The implementation is factually accurate and follows RFC 7636 standards for PKCE. It includes all requested components: verifier/challenge generation, URL construction, and token exchange. The code is well-structured and the explanation of the security mechanism is clear. The response is slightly incomplete as the code block cuts off at the very end, but the core logic is fully present." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 9.25, "brief_justification": "The explanation of PKCE's security model is excellent and the code implementation for verifier/challenge generation is highly accurate and follows RFC 7636 strictly. However, the response was cut off before completing the full implementation (token exchange, refresh, and storage), likely due to output length limits." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 7.7, "brief_justification": "The explanation of PKCE is accurate and clear. However, the response is incomplete as it cuts off before providing the actual Python implementation, code verifier generation, and token storage requested in the prompt." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 9, "brief_justification": "The implementation of PKCE (RFC 7636) is cryptographically sound and follows best practices, including timing-safe comparisons and proper base64url encoding. The explanation of the security mechanism is concise and accurate. The response was cut off at the very end (TokenSet methods), but the core requirements were met." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 9.8, "brief_justification": "The implementation is factually correct, follows RFC 7636 standards, and provides a clear, actionable Python class. The explanation of the security mechanism is precise and easy to understand." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 9.4, "brief_justification": "The implementation is technically sound, follows RFC 7636 correctly, and includes all requested components including a local callback server. The explanation of PKCE is clear, though the very last sentence of the response appears to be cut off." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8.6, "brief_justification": "The response provides a high-quality, secure implementation of OAuth 2.0 with PKCE. It correctly uses SHA-256 for the challenge and implements a robust AES-GCM encrypted storage solution. The code is well-structured and the explanation of PKCE security is accurate. The response is slightly incomplete as the final code block is cut off, but the core logic for PKCE and storage is fully present." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 9.4, "brief_justification": "The implementation is technically accurate, following RFC 7636 (PKCE) and RFC 6749 (OAuth 2.0) specifications. It correctly handles the cryptographic requirements for the code verifier and challenge. The explanation of why PKCE prevents interception attacks is concise and accurate. While the response was cut off at the very end of a JSON example, the core requirements (verifier generation, URL construction, exchange, and storage advice) were fully addressed." }, { "judge_key": "gemini_3_flash", "judge_name": "Gemini 3 Flash Preview", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 9.4, "brief_justification": "The implementation is excellent, using only standard libraries as requested. It correctly implements the PKCE SHA256 hashing and Base64URL encoding. It covers all requested features including a local callback server, token exchange, and refresh. The explanation of PKCE's security benefits is missing from the text but implied in the code structure; however, the prompt specifically asked for an explanation of why it prevents interception attacks, which was the only minor omission." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 7, "brief_justification": "Well-structured implementation with correct PKCE details and security explanation, but the response is abruptly truncated mid-code - the refresh_access_token method is incomplete and the code cuts off with a dangling partial line." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 5.95, "brief_justification": "Excellent explanation of PKCE with clear diagrams and correct code verifier/challenge implementation. However, the response is severely incomplete - it stops mid-implementation and does not cover authorization URL construction, token exchange, token refresh, or secure token storage as requested. The code snippet ends abruptly with a broken property definition." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 3.4, "brief_justification": "Response is severely incomplete - it was cut off mid-explanation and contains no actual Python code implementation. Only began explaining the PKCE concept before terminating abruptly." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 7.1, "brief_justification": "Technically accurate with good PKCE explanation and cryptography. However, response is truncated (ends mid-code at @class), missing token refresh implementation, secure token storage, and complete token exchange. The partial implementation covers core PKCE but is not fully functional." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 8, "brief_justification": "Technically correct implementation covering all required components. Well-structured code with clear explanations. Good depth on PKCE mechanism but could elaborate more on attack scenarios. Practical and usable code with proper security practices like S256 method and file permissions." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 7.45, "brief_justification": "Correct implementation of PKCE flow with code verifier/challenge generation, token exchange, and refresh. Covers most required components but has gaps: incomplete PKCE explanation at the end, plaintext token storage (not actually secure), no state validation. Clear code structure but could use more security best practices for production use." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 7, "brief_justification": "Well-structured implementation with correct PKCE generation and secure AES-GCM token storage. However, token exchange is incomplete (method cut off with 'raise Runtime') and token refresh functionality is entirely missing. The PKCE security explanation is clear and accurate." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 7.6, "brief_justification": "Clear and accurate implementation with good PKCE explanation. Correctly covers all required components. However, the 'secure storage' is misleading as it just writes plain JSON to a file without actual encryption, and the state parameter is mentioned but not implemented. The explanation of why PKCE prevents attacks is solid." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "mimo_v2_flash", "respondent_name": "MiMo-V2-Flash", "weighted_score": 6.6, "brief_justification": "Good implementation of code verifier/challenge generation, authorization URL, token exchange and refresh. However, missing the critical explanation of why PKCE prevents authorization code interception attacks (explicitly requested). Also, the code is truncated - print_token_summary function is incomplete." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gpt_5_4", "respondent_name": "GPT-5.4", "weighted_score": 8.6, "brief_justification": "Excellent implementation covering all requested aspects. Correct PKCE logic, secure storage considerations, and clear explanation. Minor deduction for completeness as the code snippet was cut off mid-response." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "claude_opus_46", "respondent_name": "Claude Opus 4.6", "weighted_score": 8.8, "brief_justification": "Excellent PKCE explanation and implementation. Missing token refresh and secure storage completion." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gemini_31_pro", "respondent_name": "Gemini 3.1 Pro", "weighted_score": 8.8, "brief_justification": "The response provides a correct and secure implementation using standard libraries, covering all requested steps. The explanation of PKCE is technically accurate and insightful." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "claude_sonnet_46", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 8.6, "brief_justification": "Excellent implementation with strong security practices, comprehensive PKCE explanation, and modular design. Missing token refresh implementation and secure storage details, but overall highly accurate and useful." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "grok_420", "respondent_name": "Grok 4.20", "weighted_score": 9, "brief_justification": "Implementation is accurate, comprehensive, and well-structured. PKCE explanation is clear and insightful. Minor deduction for depth as some advanced security considerations could be expanded." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "deepseek_v4", "respondent_name": "DeepSeek V4", "weighted_score": 8.6, "brief_justification": "Implementation is accurate and covers all requested components; explanation of PKCE is clear but could be more detailed on the cryptographic binding." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gpt_oss_120b", "respondent_name": "GPT-OSS-120B", "weighted_score": 8.6, "brief_justification": "Excellent implementation covering all requested steps with secure practices. Missing the token refresh method in the provided code snippet, but the explanation and structure are clear and actionable." }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "gemini_3_flash", "respondent_name": "Gemini 3 Flash Preview", "weighted_score": 8.6, "brief_justification": "The response provides a correct and functional implementation of the OAuth 2.0 Authorization Code flow with PKCE. It covers all requested components (verifier/challenge generation, auth URL, token exchange, refresh, and storage) and explains the security rationale clearly. The code is well-structured and uses appropriate standard libraries. Minor deduction in completeness for not explicitly mentioning the 'state' parameter in the implementation code (though it's mentioned in security considerati" }, { "judge_key": "mimo_v2_flash", "judge_name": "MiMo-V2-Flash", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 8.6, "brief_justification": "Accurate implementation of PKCE and token flows, includes secure storage via keyring, and explains the security rationale clearly. Missing token refresh implementation in the provided snippet, though mentioned in text." } ], "meta": { "source": "The Multivac (app.themultivac.com)", "methodology": "10x10 blind peer matrix evaluation", "criteria": "correctness, completeness, clarity, depth, usefulness", "self_judgments": "excluded from rankings", "license": "Open data — cite as: The Multivac (2026)" } }