{ "evaluation_id": "EVAL-20260318-161503", "question_id": "EVAL-20260318-161503", "question_text": "This distributed lock implementation has a subtle race condition that can cause two processes to hold the lock simultaneously. Find the bug and fix it.\n\n```python\nimport redis\nimport time\nimport uuid\n\nclass DistributedLock:\n def __init__(self, redis_client, lock_name, timeout=10):\n self.redis = redis_client\n self.lock_name = f\"lock:{lock_name}\"\n self.timeout = timeout\n self.token = str(uuid.uuid4())\n\n def acquire(self):\n while True:\n if self.redis.setnx(self.lock_name, self.token):\n self.redis.expire(self.lock_name, self.timeout)\n return True\n time.sleep(0.1)\n\n def release(self):\n if self.redis.get(self.lock_name) == self.token:\n self.redis.delete(self.lock_name)\n```\n\nExplain why this is dangerous in production and provide a correct implementation.", "category": "code", "timestamp": "2026-03-18T16:15:03.622Z", "display_date": "Mar 18, 2026", "winner": { "name": "GPT-5.4", "provider": "openrouter", "score": 9.97 }, "avg_score": 8.217143, "matrix_size": 42, "models_used": [ { "id": "minimax_m25", "name": "MiniMax M2.5", "provider": "openrouter" }, { "id": "minimax_m1", "name": "MiniMax M1", "provider": "openrouter" }, { "id": "minimax_01", "name": "MiniMax-01", "provider": "openrouter" }, { "id": "judge_claude_sonnet", "name": "Claude Sonnet 4.6", "provider": "openrouter" }, { "id": "judge_gpt54", "name": "GPT-5.4", "provider": "openrouter" }, { "id": "minimax_m27", "name": "MiniMax M2.7", "provider": "openrouter" }, { "id": "minimax_m21", "name": "MiniMax M2.1", "provider": "openrouter" } ], "rankings": { "judge_gpt54": { "display_name": "GPT-5.4", "provider": "openrouter", "average_score": 9.97, "score_count": 7, "min_score": 9.8, "max_score": 10, "rank": 1 }, "judge_claude_sonnet": { "display_name": "Claude Sonnet 4.6", "provider": "openrouter", "average_score": 9.78, "score_count": 7, "min_score": 8.65, "max_score": 10, "rank": 2 }, "minimax_m1": { "display_name": "MiniMax M1", "provider": "openrouter", "average_score": 9.65, "score_count": 7, "min_score": 8.8, "max_score": 10, "rank": 3 }, "minimax_m27": { "display_name": "MiniMax M2.7", "provider": "openrouter", "average_score": 9.18, "score_count": 7, "min_score": 7.4, "max_score": 10, "rank": 4 }, "minimax_m25": { "display_name": "MiniMax M2.5", "provider": "openrouter", "average_score": 8.78, "score_count": 6, "min_score": 6.65, "max_score": 9.6, "rank": 5 }, "minimax_01": { "display_name": "MiniMax-01", "provider": "openrouter", "average_score": 8.31, "score_count": 7, "min_score": 7.75, "max_score": 8.8, "rank": 6 }, "minimax_m21": { "display_name": "MiniMax M2.1", "provider": "openrouter", "average_score": 1.85, "score_count": 1, "min_score": 1.85, "max_score": 1.85, "rank": 7 } }, "judgments": [ { "judge_key": "minimax_m27", "judge_name": "MiniMax M2.7", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 9.4, "brief_justification": "Correctly identifies both bugs (non-atomic acquire and release race condition) with accurate Lua script fix and comprehensive production impact analysis; minor deduction for incomplete code ending." }, { "judge_key": "minimax_m27", "judge_name": "MiniMax M2.7", "respondent_key": "minimax_m1", "respondent_name": "MiniMax M1", "weighted_score": 10, "brief_justification": "Excellent response that correctly identifies both bugs (non-atomic acquire and non-atomic release), explains production dangers with concrete scenarios, and provides a complete fix using SET NX EX for atomic acquisition and Lua scripts for atomic release. Includes practical context manager pattern and thorough examples." }, { "judge_key": "minimax_m27", "judge_name": "MiniMax M2.7", "respondent_key": "minimax_01", "respondent_name": "MiniMax-01", "weighted_score": 8.8, "brief_justification": "The response correctly identifies the non-atomic check-and-delete race condition and provides a solid fix using Redis Lua scripts. It thoroughly explains why the bug is dangerous and offers practical, production-ready code. Minor deduction for not addressing edge cases like clock skew or the subtle setnx+expire non-atomicity issue in detail." }, { "judge_key": "minimax_m27", "judge_name": "MiniMax M2.7", "respondent_key": "judge_claude_sonnet", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 10, "brief_justification": "The response comprehensively identifies both race conditions (non-atomic SETNX+EXPIRE and GET+DELETE), explains their production dangers with a clear timeline, and provides a production-ready solution using atomic Redis operations (SET with NX/PX and Lua scripts), including additional features like lock extension, context managers, and proper error handling." }, { "judge_key": "minimax_m27", "judge_name": "MiniMax M2.7", "respondent_key": "judge_gpt54", "respondent_name": "GPT-5.4", "weighted_score": 10, "brief_justification": "Comprehensive identification of both bugs with correct fixes, detailed explanation of race conditions, working code with context manager support, and important caveats about lease-based lock limitations in distributed systems." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "minimax_m27", "respondent_name": "MiniMax M2.7", "weighted_score": 9.35, "brief_justification": "Response correctly identifies both race conditions (TOCTOU in release and non-atomic setnx+expire), explains production dangers thoroughly, and provides a complete fix using Lua scripts. Minor issue: the acquire script could use Redis SET with NX option for better atomicity, but the Lua approach is still correct." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "minimax_m1", "respondent_name": "MiniMax M1", "weighted_score": 9.4, "brief_justification": "Response correctly identifies both bugs (non-atomic acquire causing lock leakage, and non-atomic release causing the subtle race condition where one process can delete another's lock). Provides technically correct fixes using Redis SET with NX/EX and Lua scripts, with clear explanations, production impact analysis, and practical code examples including a context manager." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "minimax_01", "respondent_name": "MiniMax-01", "weighted_score": 8.6, "brief_justification": "The response correctly identifies and fixes the release method race condition using Lua scripts, and provides a correct acquire implementation using atomic set with nx/ex. However, it doesn't explicitly explain the separate race condition in the original acquire method (setnx + expire not being atomic)." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "judge_claude_sonnet", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 9.8, "brief_justification": "Excellent response that correctly identifies both race conditions (non-atomic SETNX+EXPIRE and non-atomic GET+DELETE), explains the dangerous production scenarios with a clear timeline, and provides a complete, well-implemented fix using Redis SET with NX/PX flags and Lua scripts for atomic operations. Minor deduction for a small typo at the end ('lock.' instead of 'lock.release()')." }, { "judge_key": "minimax_m25", "judge_name": "MiniMax M2.5", "respondent_key": "judge_gpt54", "respondent_name": "GPT-5.4", "weighted_score": 10, "brief_justification": "Response correctly identifies both bugs (non-atomic SETNX+EXPIRE and non-atomic GET+DELETE), explains the race conditions in detail, provides correct implementations using atomic Redis operations and Lua scripts, and goes beyond to discuss broader distributed systems considerations like lease expiration and fencing." }, { "judge_key": "minimax_m21", "judge_name": "MiniMax M2.1", "respondent_key": "minimax_m27", "respondent_name": "MiniMax M2.7", "weighted_score": 9.8, "brief_justification": "Excellent technical analysis identifying both race conditions with correct Lua script fixes. Minor completeness deduction for truncated usage example at the end." }, { "judge_key": "minimax_m21", "judge_name": "MiniMax M2.1", "respondent_key": "minimax_m1", "respondent_name": "MiniMax M1", "weighted_score": 10, "brief_justification": "Excellent response that correctly identifies both race conditions (non-atomic acquire and non-atomic release), explains production dangers comprehensively, and provides a production-ready implementation using SET NX EX and Lua scripts for atomic operations. Well-structured with clear explanations and practical code." }, { "judge_key": "minimax_m21", "judge_name": "MiniMax M2.1", "respondent_key": "minimax_01", "respondent_name": "MiniMax-01", "weighted_score": 7.8, "brief_justification": "Correctly identifies the race condition in the release method and provides a working Lua script fix. Could have been more thorough by also addressing the non-atomic acquire issue and simplifying the explanation." }, { "judge_key": "minimax_m21", "judge_name": "MiniMax M2.1", "respondent_key": "judge_claude_sonnet", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 10, "brief_justification": "Excellent response that correctly identifies both race conditions with clear timeline diagrams, provides a production-ready implementation using atomic Redis SET with NX/PX and Lua scripts, and includes comprehensive usage examples with context managers and lock extension." }, { "judge_key": "minimax_m21", "judge_name": "MiniMax M2.1", "respondent_key": "judge_gpt54", "respondent_name": "GPT-5.4", "weighted_score": 10, "brief_justification": "Response correctly identifies both bugs (non-atomic SETNX+EXPIRE and GET+DELETE), provides proper fixes using Redis SET with NX/EX and Lua scripts, explains production dangers thoroughly, and includes useful enhancements like context manager support." }, { "judge_key": "minimax_m2", "judge_name": "MiniMax M2", "respondent_key": "minimax_m27", "respondent_name": "MiniMax M2.7", "weighted_score": 9.8, "brief_justification": "Excellent response identifying both race conditions (TOCTOU and non-atomic operations), explaining production dangers with concrete scenarios, and providing a comprehensive corrected implementation using atomic Lua scripts with additional useful features." }, { "judge_key": "minimax_m2", "judge_name": "MiniMax M2", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 9, "brief_justification": "Excellent response that correctly identifies both race conditions, explains production dangers clearly, and provides a complete atomic implementation. Minor deduction for incomplete code example at the end." }, { "judge_key": "minimax_m2", "judge_name": "MiniMax M2", "respondent_key": "minimax_m1", "respondent_name": "MiniMax M1", "weighted_score": 10, "brief_justification": "Excellent response that correctly identifies both race conditions (non-atomic acquire and release), explains the production dangers with clear scenarios, and provides a complete, production-ready implementation using SET NX EX for atomic acquire and Lua scripts for atomic release. The context manager and additional features add practical value." }, { "judge_key": "minimax_m2", "judge_name": "MiniMax M2", "respondent_key": "minimax_01", "respondent_name": "MiniMax-01", "weighted_score": 8.35, "brief_justification": "The response correctly identifies the non-atomic release bug and provides a valid fix using Lua scripts, along with an improved acquire method using SET with NX/EX options. The explanation is clear and practical, though it could have more thoroughly covered edge cases and the specific acquire() race condition." }, { "judge_key": "minimax_m2", "judge_name": "MiniMax M2", "respondent_key": "judge_claude_sonnet", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 10, "brief_justification": "The response excellently identifies both race conditions (non-atomic SETNX+EXPIRE and non-atomic GET+DELETE), provides a detailed timeline showing how two processes can hold the lock simultaneously, and delivers a production-ready implementation using Redis SET with NX/PX flags and Lua scripts for atomic operations." }, { "judge_key": "minimax_m2", "judge_name": "MiniMax M2", "respondent_key": "judge_gpt54", "respondent_name": "GPT-5.4", "weighted_score": 10, "brief_justification": "Response correctly identifies both bugs (non-atomic SETNX+EXPIRE and non-atomic GET+DELETE), explains the production dangers with concrete examples, and provides a complete fix using atomic Redis operations and Lua scripts. Also addresses broader distributed systems limitations and provides additional practical features like context manager support." }, { "judge_key": "minimax_m1", "judge_name": "MiniMax M1", "respondent_key": "minimax_m27", "respondent_name": "MiniMax M2.7", "weighted_score": 8.35, "brief_justification": "Correctly identifies both race conditions (TOCTOU in release and non-atomic setnx+expire) with excellent depth, but the provided code has bugs - 'import time' appears at the bottom of the file instead of the top, causing NameError, and there's a syntax error in the usage example with a missing closing quote." }, { "judge_key": "minimax_m1", "judge_name": "MiniMax M1", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 9, "brief_justification": "Correctly identifies both race conditions (non-atomic setnx+expire and non-atomic check-then-delete), explains production dangers with a clear table, and provides a proper fix using SET NX EX and Lua script for atomic operations. The code implementation is mostly complete (though truncated)." }, { "judge_key": "minimax_m1", "judge_name": "MiniMax M1", "respondent_key": "minimax_01", "respondent_name": "MiniMax-01", "weighted_score": 8.35, "brief_justification": "The response correctly identifies the race condition and provides a valid fix using Lua scripts for atomic operations. However, the explanation of the race condition is slightly convoluted, and it could have more clearly addressed the acquire-side race condition (setnx + expire not being atomic)." }, { "judge_key": "minimax_m1", "judge_name": "MiniMax M1", "respondent_key": "judge_claude_sonnet", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 10, "brief_justification": "Excellent response that correctly identifies both race conditions (non-atomic SETNX+EXPIRE and non-atomic GET+DELETE), provides a clear timeline showing how two processes can hold the lock simultaneously, explains production dangers (data corruption, deadlocks, security issues), and provides a complete, production-ready implementation using Redis SET with NX/PX flags and Lua scripts for atomic operations. The code includes context managers, proper error handling, logging, and extension methods." }, { "judge_key": "minimax_m1", "judge_name": "MiniMax M1", "respondent_key": "judge_gpt54", "respondent_name": "GPT-5.4", "weighted_score": 10, "brief_justification": "Excellent response that correctly identifies both bugs (non-atomic SETNX+EXPIRE and non-atomic GET+DELETE), provides correct fixes using Redis SET with NX/EX and Lua scripts, explains production dangers thoroughly, and includes working code with valuable caveats about lease-based lock limitations." }, { "judge_key": "minimax_01", "judge_name": "MiniMax-01", "respondent_key": "minimax_m27", "respondent_name": "MiniMax M2.7", "weighted_score": 10, "brief_justification": "The response thoroughly explains the race conditions, provides a detailed and accurate fix using atomic Lua scripts, and includes practical considerations for production environments." }, { "judge_key": "minimax_01", "judge_name": "MiniMax-01", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 9.6, "brief_justification": "The response provides a comprehensive and accurate explanation of the race condition and deadlock issues, along with a correct implementation using atomic Redis commands and Lua scripts. The explanation is clear and insightful, though the code for the acquire method is incomplete, slightly detracting from clarity and depth." }, { "judge_key": "minimax_01", "judge_name": "MiniMax-01", "respondent_key": "minimax_m21", "respondent_name": "MiniMax M2.1", "weighted_score": 1.85, "brief_justification": "The response is entirely missing, making it impossible to evaluate correctness, completeness, or any other criteria. No meaningful analysis or solution was provided." }, { "judge_key": "minimax_01", "judge_name": "MiniMax-01", "respondent_key": "minimax_m1", "respondent_name": "MiniMax M1", "weighted_score": 9.8, "brief_justification": "The response thoroughly identifies and explains the race condition bugs, provides a correct implementation with atomic operations, and includes practical usage examples, making it highly valuable for production environments." }, { "judge_key": "minimax_01", "judge_name": "MiniMax-01", "respondent_key": "judge_claude_sonnet", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 10, "brief_justification": "The response provides a detailed explanation of the race conditions, a corrected implementation with atomic operations, and practical usage examples, demonstrating a comprehensive and insightful solution." }, { "judge_key": "minimax_01", "judge_name": "MiniMax-01", "respondent_key": "judge_gpt54", "respondent_name": "GPT-5.4", "weighted_score": 9.8, "brief_justification": "The response correctly identifies and explains the race conditions, provides a robust fix with atomic operations, and includes practical implementation details. It also discusses broader distributed systems considerations, though it could elaborate slightly more on the limitations of lease-based locks." }, { "judge_key": "judge_claude_sonnet", "judge_name": "Claude Sonnet 4.6", "respondent_key": "minimax_m27", "respondent_name": "MiniMax M2.7", "weighted_score": 9.55, "brief_justification": "The response correctly identifies both race conditions (TOCTOU in release and non-atomic setnx+expire), provides accurate Lua-based atomic fixes, and includes production-relevant context. Minor note: the ACQUIRE_SCRIPT Lua approach works but SET with NX and EX flags in one command would be slightly more idiomatic; however, the solution is functionally correct and the response is exceptionally well-structured with practical examples, a context manager, and lock extension support." }, { "judge_key": "judge_claude_sonnet", "judge_name": "Claude Sonnet 4.6", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 9, "brief_justification": "The response correctly identifies both race conditions (non-atomic setnx+expire and non-atomic get+delete), explains the production dangers clearly with a helpful table, and provides a correct implementation using atomic SET NX EX and a Lua script for release. The response appears to be cut off at the end but still covers the essential fixes comprehensively." }, { "judge_key": "judge_claude_sonnet", "judge_name": "Claude Sonnet 4.6", "respondent_key": "minimax_m1", "respondent_name": "MiniMax M1", "weighted_score": 9.55, "brief_justification": "The response correctly identifies both race conditions (non-atomic setnx+expire and non-atomic get+delete), provides accurate fixes using SET NX EX and Lua scripting, and includes excellent production-ready code with context manager, retry logic, and lock extension. The brief self-correction mid-explanation is slightly awkward but demonstrates honest reasoning." }, { "judge_key": "judge_claude_sonnet", "judge_name": "Claude Sonnet 4.6", "respondent_key": "minimax_01", "respondent_name": "MiniMax-01", "weighted_score": 8.55, "brief_justification": "The response correctly identifies both bugs (non-atomic setnx+expire in acquire, and non-atomic get+delete in release) and provides proper fixes using SET with NX/EX flags and a Lua script for atomic release. The explanation of the race condition scenario is clear and accurate, though the description of the acquire race condition could be more explicitly stated as the primary bug alongside the release bug." }, { "judge_key": "judge_claude_sonnet", "judge_name": "Claude Sonnet 4.6", "respondent_key": "judge_gpt54", "respondent_name": "GPT-5.4", "weighted_score": 10, "brief_justification": "The response correctly identifies both race conditions (non-atomic SETNX+EXPIRE and non-atomic GET+DELETE), provides accurate fixes using SET NX EX and Lua scripting, explains production dangers clearly, and goes beyond the question to address lock expiry edge cases, fencing tokens, and context manager patterns—all technically sound and immediately actionable." }, { "judge_key": "judge_gpt54", "judge_name": "GPT-5.4", "respondent_key": "minimax_m27", "respondent_name": "MiniMax M2.7", "weighted_score": 7.4, "brief_justification": "The response correctly identifies the unsafe non-atomic release and the separate SETNX/EXPIRE issue, and proposes atomic release via Lua plus fresh tokens per acquire. However, its acquire 'fix' still uses a Lua-wrapped SETNX+EXPIRE instead of Redis's native atomic SET key value NX EX, and the timeline illustrating the primary bug is muddled/inaccurate, with a truncated example at the end." }, { "judge_key": "judge_gpt54", "judge_name": "GPT-5.4", "respondent_key": "minimax_m25", "respondent_name": "MiniMax M2.5", "weighted_score": 6.65, "brief_justification": "The response correctly identifies the two key races and the proper fixes (atomic SET NX EX and atomic token-check-delete via Lua), but it is incomplete because the provided implementation is cut off before the acquire/release code finishes. It explains the production danger well, yet the truncated code limits practical usefulness." }, { "judge_key": "judge_gpt54", "judge_name": "GPT-5.4", "respondent_key": "minimax_m1", "respondent_name": "MiniMax M1", "weighted_score": 8.8, "brief_justification": "The response correctly identifies both the acquire and release atomicity bugs and provides the standard Redis fixes using SET NX EX and a Lua compare-and-delete. It is clear and practical, though the race explanation briefly stumbles before correcting itself and it omits some advanced production caveats such as lock renewal semantics and Redlock/distributed failure model nuances." }, { "judge_key": "judge_gpt54", "judge_name": "GPT-5.4", "respondent_key": "minimax_01", "respondent_name": "MiniMax-01", "weighted_score": 7.75, "brief_justification": "The response correctly identifies the non-atomic release bug and also fixes the separate acquire bug by using SET NX EX atomically. However, it overstates some production risks (e.g. deadlocks here) and its renew method is unsafe because it extends TTL without verifying ownership, missing another important race in a production-ready lock." }, { "judge_key": "judge_gpt54", "judge_name": "GPT-5.4", "respondent_key": "judge_claude_sonnet", "respondent_name": "Claude Sonnet 4.6", "weighted_score": 8.65, "brief_justification": "The response correctly identifies both major races and fixes them with atomic SET NX PX and atomic release via Lua, plus helpful production context. It loses some points because the sample is truncated at the end, and the extend() method uses PEXPIRE with only the additional time rather than clearly resetting/renewing to a full lease duration, which could be misleading." } ], "meta": { "source": "The Multivac (app.themultivac.com)", "methodology": "10x10 blind peer matrix evaluation", "criteria": "correctness, completeness, clarity, depth, usefulness", "self_judgments": "excluded from rankings", "license": "Open data — cite as: The Multivac (2026)" } }