{"case_id": "challenge-coding-001", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "parser", "state"], "scorer": "code_python_tests", "expected": {"tests": ["lines = ['# comment', '[db]', ' host = localhost ', 'port=5432', '', '[feature]', 'enabled=true', 'enabled = false']\nassert solution.solve(lines) == {'db': {'host': 'localhost', 'port': '5432'}, 'feature': {'enabled': 'false'}}", "assert solution.solve(['[x]', 'a=1', 'a=2', '[y]', 'b = hi']) == {'x': {'a': '2'}, 'y': {'b': 'hi'}}"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(lines: list[str]) -> dict。lines 是简化 INI：空行和以 # 开头的行忽略；[section] 开始新 section；key=value 写入当前 section；key/value 两侧空白要去掉；同一 section 内重复 key 后者覆盖前者。返回嵌套 dict。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "2ca132befd6923601a17f24e5efb427ea787e3ae63ae265dd45f0c700cd895d6", "started_at": "2026-06-23T12:07:01.572323+00:00", "latency_ms": 70271, "output": "```python\ndef solve(lines: list[str]) -> dict:\n config = {}\n current_section = None\n for line in lines:\n stripped = line.strip()\n if not stripped or stripped.startswith('#'):\n continue\n if stripped.startswith('[') and stripped.endswith(']'):\n section_name = stripped[1:-1].strip()\n current_section = section_name\n if section_name not in config:\n config[section_name] = {}\n continue\n key, value = stripped.split('=', 1)\n config[current_section][key.strip()] = value.strip()\n return config\n```", "usage": {"prompt_tokens": 138, "completion_tokens": 3551, "total_tokens": 3689, "completion_tokens_details": {"reasoning_tokens": 3408}}, "response_id": "resp_021782216411855cd2cfaf2275915feb63370f28ffb18cbd1b9da", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-002", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "ledger", "state"], "scorer": "code_python_tests", "expected": {"tests": ["events = ['a credit 100', 'b debit 40', 'a credit 999', 'rev b', 'c debit 10']\nassert solution.solve(events) == 90", "events = ['x debit 5', 'rev y', 'rev x', 'rev x', 'z credit 7']\nassert solution.solve(events) == 7"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(events: list[str]) -> int。事件格式：\" credit \" 增加余额，\" debit \" 减少余额，\"rev \" 撤销此前尚未撤销过的同 id 事件。重复交易 id 只保留第一次，未知撤销忽略。返回最终余额。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "3d2d5a46a600444476b5ddfd04634035ff394bcb76fcd230d785597c9cf295ae", "started_at": "2026-06-23T12:08:11.887700+00:00", "latency_ms": 82808, "output": "```python\ndef solve(events: list[str]) -> int:\n transactions = {}\n cancelled = set()\n for event in events:\n parts = event.split()\n if parts[0] == 'rev':\n target_id = parts[1]\n if target_id in transactions and target_id not in cancelled:\n cancelled.add(target_id)\n else:\n tx_id = parts[0]\n if tx_id in transactions:\n continue\n tx_type = parts[1]\n amount = int(parts[2])\n transactions[tx_id] = amount if tx_type == 'credit' else -amount\n total = 0\n for tx_id, delta in transactions.items():\n if tx_id not in cancelled:\n total += delta\n return total\n```", "usage": {"prompt_tokens": 138, "completion_tokens": 4695, "total_tokens": 4833, "completion_tokens_details": {"reasoning_tokens": 4516}}, "response_id": "resp_0217822164815447097180e458ae3c6e16036676cea376f27f6c7", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-003", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "topological-sort"], "scorer": "code_python_tests", "expected": {"tests": ["deps = {'build': ['lint', 'test'], 'test': ['install'], 'lint': ['install']}\nassert solution.solve(deps) == [['install'], ['lint', 'test'], ['build']]", "assert solution.solve({'a': ['b'], 'b': ['a']}) == []", "assert solution.solve({'deploy': ['build'], 'build': []}) == [['build'], ['deploy']]"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(deps: dict[str, list[str]]) -> list[list[str]]。deps 表示 task -> prerequisites。返回可并行执行的拓扑层，每层任务按字典序排序；如果存在环，返回空列表。所有只出现在 prerequisites 中的任务也要包含。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "413c5b602b0fd1ae1e9c3a49b09ecf03c61769fce4b34dbd96b5d8c6a2c68f90", "started_at": "2026-06-23T12:09:34.743422+00:00", "latency_ms": 177881, "output": "```python\ndef solve(deps: dict[str, list[str]]) -> list[list[str]]:\n nodes = set()\n for task, prereqs in deps.items():\n nodes.add(task)\n for p in prereqs:\n nodes.add(p)\n \n in_degree = {n: 0 for n in nodes}\n adj = {n: [] for n in nodes}\n \n for task, prereqs in deps.items():\n for p in set(prereqs):\n adj[p].append(task)\n in_degree[task] += 1\n \n res = []\n while True:\n current_layer = [n for n in nodes if in_degree[n] == 0]\n if not current_layer:\n break\n current_layer.sort()\n res.append(current_layer)\n for u in current_layer:\n in_degree[u] = -1\n for v in adj[u]:\n in_degree[v] -= 1\n \n if sum(len(layer) for layer in res) != len(nodes):\n return []\n return res\n```", "usage": {"prompt_tokens": 131, "completion_tokens": 9686, "total_tokens": 9817, "completion_tokens_details": {"reasoning_tokens": 9438}}, "response_id": "resp_0217822165643660249e54cf23e108bd1702536cbe0373f9bda77", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-004", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "lru"], "scorer": "code_python_tests", "expected": {"tests": ["ops = ['put a 1', 'put b 2', 'get a', 'put c 3', 'get b', 'get a', 'get c']\nassert solution.solve(2, ops) == [1, -1, 1, 3]", "assert solution.solve(0, ['put a 1', 'get a']) == [-1]", "assert solution.solve(1, ['put a 1', 'put a 2', 'get a']) == [2]"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(capacity: int, ops: list[str]) -> list[int]，模拟 LRU cache。ops 包含 \"put key value\" 和 \"get key\"。get 返回 value，不存在返回 -1。put 已存在 key 要更新并变成最近使用。容量为 0 时所有 get 都返回 -1。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "840e7b4d0c404f3d7b34ebc9931e87612608d6339755d5f1066fda55bd957375", "started_at": "2026-06-23T12:12:32.665451+00:00", "latency_ms": 70476, "output": "```python\nfrom collections import OrderedDict\n\ndef solve(capacity: int, ops: list[str]) -> list[int]:\n res = []\n if capacity <= 0:\n for op in ops:\n parts = op.split()\n if parts[0] == 'get':\n res.append(-1)\n return res\n cache = OrderedDict()\n for op in ops:\n parts = op.split()\n if parts[0] == 'get':\n key = int(parts[1])\n if key not in cache:\n res.append(-1)\n else:\n cache.move_to_end(key)\n res.append(cache[key])\n elif parts[0] == 'put':\n key = int(parts[1])\n value = int(parts[2])\n if key in cache:\n cache[key] = value\n cache.move_to_end(key)\n else:\n if len(cache) == capacity:\n cache.popitem(last=False)\n cache[key] = value\n return res\n```", "usage": {"prompt_tokens": 142, "completion_tokens": 4115, "total_tokens": 4257, "completion_tokens_details": {"reasoning_tokens": 3875}}, "response_id": "resp_0217822167423711c4c28e26c266d30c807e3f14d371f957306c1", "finish_reason": "stop", "score": {"passed": false, "score": 0.0, "details": {"returncode": 1, "stdout": "", "stderr": "Traceback (most recent call last):\n File \"/var/folders/x_/4cht1kfj1_j31gdv5w01sgmw0000gn/T/dracobench-code-4wmtsl7u/test_solution.py\", line 4, in \n assert solution.solve(2, ops) == [1, -1, 1, 3]\n ^^^^^^^^^^^^^^^^^^^^^^\n File \"/private/var/folders/x_/4cht1kfj1_j31gdv5w01sgmw0000gn/T/dracobench-code-4wmtsl7u/solution.py\", line 22, in solve\n key = int(parts[1])\n ^^^^^^^^^^^^^\nValueError: invalid literal for int() with base 10: 'a'\n"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-005", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "json", "copy"], "scorer": "code_python_tests", "expected": {"tests": ["base = {'a': 1, 'b': {'x': 1, 'y': 2}, 'c': 3}\npatch = {'b': {'y': None, 'z': 4}, 'c': None}\nassert solution.solve(base, patch) == {'a': 1, 'b': {'x': 1, 'z': 4}}\nassert base == {'a': 1, 'b': {'x': 1, 'y': 2}, 'c': 3}", "assert solution.solve({'a': {'b': 1}}, {'a': 5}) == {'a': 5}"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(base: dict, patch: dict) -> dict，执行 JSON Merge Patch：patch 中值为 None 表示删除该 key；两个值都是 dict 时递归合并；其他值覆盖。不能修改输入对象。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "8dc22406b418d74ea74b50692fe086d4f80748258269dc93f82290e6c344b5a3", "started_at": "2026-06-23T12:13:43.193515+00:00", "latency_ms": 323602, "output": "", "usage": {"prompt_tokens": 117, "completion_tokens": 16384, "total_tokens": 16501, "completion_tokens_details": {"reasoning_tokens": 16384}}, "response_id": "resp_021782216812833ec2a974ffc08eb8a410ac7a7438c00493b9c16", "finish_reason": "incomplete", "score": {"passed": false, "score": 0.0, "details": {"returncode": 1, "stdout": "", "stderr": "Traceback (most recent call last):\n File \"/var/folders/x_/4cht1kfj1_j31gdv5w01sgmw0000gn/T/dracobench-code-wjtmhlr7/test_solution.py\", line 5, in \n assert solution.solve(base, patch) == {'a': 1, 'b': {'x': 1, 'z': 4}}\n ^^^^^^^^^^^^^^\nAttributeError: module 'solution' has no attribute 'solve'\n"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-006", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "regex", "duration"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve('1h 30m 500ms') == 5400500", "assert solution.solve('2d 1h 2s') == 176402000", "assert solution.solve('100ms 2s 3ms') == 2103"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(text: str) -> int，把时长字符串转换成毫秒。支持整数单位 d、h、m、s、ms，可有空格，单位可重复，顺序任意。例如 \"1h 30m 500ms\"。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "ce87366a2f6f080a35c0fdd0872b6a8304a8233e0c5c9361c19fec614c0d1fcb", "started_at": "2026-06-23T12:19:06.861396+00:00", "latency_ms": 66652, "output": "```python\nimport re\n\ndef solve(text: str) -> int:\n unit_map = {\n 'd': 86400000,\n 'h': 3600000,\n 'm': 60000,\n 's': 1000,\n 'ms': 1\n }\n total = 0\n for num_str, unit in re.findall(r'(\\d+)\\s*(ms|[dhms])', text):\n total += int(num_str) * unit_map[unit]\n return total\n```", "usage": {"prompt_tokens": 122, "completion_tokens": 3580, "total_tokens": 3702, "completion_tokens_details": {"reasoning_tokens": 3444}}, "response_id": "resp_021782217136538d7de4542f2cb4665397a3eedddf0de611c6308", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-007", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "parser", "quotes"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve('{a:[(1)]}') is True", "assert solution.solve(\"{'x': '(]'}\") is True", "assert solution.solve(\"['unterminated]\") is False", "assert solution.solve('([)]') is False", "assert solution.solve(\"\\\"paren \\\\\\\" ) still quoted\\\"\") is True"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(s: str) -> bool，检查括号 ()[]{} 是否配对。单引号或双引号内部的括号忽略；反斜杠会转义引号。输入字符串可以包含普通字符。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "2f2e8f01533244c8104f74b3f9c9bc261154f2ff4c46c03a031258f6a5706f9f", "started_at": "2026-06-23T12:20:13.560755+00:00", "latency_ms": 324003, "output": "", "usage": {"prompt_tokens": 111, "completion_tokens": 16384, "total_tokens": 16495, "completion_tokens_details": {"reasoning_tokens": 16384}}, "response_id": "resp_02178221720314651482108684de07484f06ebb12145571a0901d", "finish_reason": "incomplete", "score": {"passed": false, "score": 0.0, "details": {"returncode": 1, "stdout": "", "stderr": "Traceback (most recent call last):\n File \"/var/folders/x_/4cht1kfj1_j31gdv5w01sgmw0000gn/T/dracobench-code-lu3xh3fg/test_solution.py\", line 3, in \n assert solution.solve('{a:[(1)]}') is True\n ^^^^^^^^^^^^^^\nAttributeError: module 'solution' has no attribute 'solve'\n"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-008", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "dedupe", "sorting"], "scorer": "code_python_tests", "expected": {"tests": ["rows = [{'user':'b','ts':1,'value':'old'}, {'user':'a','ts':2,'value':'x'}, {'user':'b','ts':3,'value':'new'}]\nassert solution.solve(rows) == [{'user':'a','ts':2,'value':'x'}, {'user':'b','ts':3,'value':'new'}]", "rows = [{'user':'a','ts':1,'value':'first'}, {'user':'a','ts':1,'value':'second'}]\nassert solution.solve(rows) == [{'user':'a','ts':1,'value':'second'}]"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(rows: list[dict]) -> list[dict]。每行有 user、ts、value。对每个 user 保留 ts 最大的一行；若 ts 相同，保留在输入中更靠后的行。输出按 user 字典序排序。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "2a9a2d40a756627e38d28a8ac6afc98311acbafa718d7b3aaaecbedb9f020179", "started_at": "2026-06-23T12:25:37.615907+00:00", "latency_ms": 116691, "output": "```python\ndef solve(rows: list[dict]) -> list[dict]:\n best = {}\n for row in rows:\n user = row['user']\n if user not in best or row['ts'] >= best[user]['ts']:\n best[user] = row\n return sorted(best.values(), key=lambda x: x['user'])\n```", "usage": {"prompt_tokens": 125, "completion_tokens": 5886, "total_tokens": 6011, "completion_tokens_details": {"reasoning_tokens": 5802}}, "response_id": "resp_021782217527373ebdbbc37c8ece91f45ac418a6cca0631352956", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-001", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "iterator-default", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*def\\s+summarize\\(items=None\\)\\s*:", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码在 total 之后再次读取数据会变成空列表。请指出根因，并给出最小修复。最后单独一行写 FIX: <修复后的第一行函数定义>。\n\n```python\ndef summarize(items=iter([1, 2, 3])):\n return sum(items)\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "7b02a617eebe49b179cee84c9b8347c6bebe382f42a2b199d92b3aa9ed844102", "started_at": "2026-06-23T12:27:34.347852+00:00", "latency_ms": 325703, "output": "### 根因\n1. 核心问题是多余的`iter()`包装将列表转为了**单遍迭代器**：迭代器是有内部状态的一次性可迭代对象，`sum()`遍历会持续推进迭代器指针直到耗尽所有元素，计算完total后迭代器已无剩余元素，再次读取自然得到空结果。\n2. 叠加Python默认参数的特性：默认参数在函数定义阶段仅求值一次，所有无参调用共享同一个默认对象，首次调用耗尽迭代器后，后续调用会持续复用这个已耗尽的迭代器，进一步导致跨调用返回空结果。\n### 最小修复\n移除默认值外层多余的`iter()`包装，直接使用列表`[1, 2, 3]`作为默认参数。列表是可重复迭代的序列容器，遍历时会生成临时迭代器访问元素，不会修改原列表本身的状态，既解决了同一次调用中sum后读取为空的问题，也避免了迭代器复用导致的跨调用异常。\nFIX: def summarize(items=[1, 2, 3]):", "usage": {"prompt_tokens": 125, "completion_tokens": 15839, "total_tokens": 15964, "completion_tokens_details": {"reasoning_tokens": 15603}}, "response_id": "resp_0217822176438568b056a623bf426f62c797dc732369cf94656b8", "finish_reason": "stop", "score": {"passed": false, "score": 0.0, "details": {"pattern": "FIX:\\s*`?\\s*def\\s+summarize\\(items=None\\)\\s*:"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-002", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "defaultdict", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*counts\\s*=\\s*defaultdict\\(lambda\\s*:\\s*\\{'seen'\\s*:\\s*\\[\\]\\}\\)\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望统计每个用户的事件数，但所有用户计数会互相影响。请指出根因，并给出最小修复。最后单独一行写 FIX: <修复后的 defaultdict 初始化行>。\n\n```python\nfrom collections import defaultdict\nshared = {'seen': []}\ncounts = defaultdict(lambda: shared)\n\ndef add(user, event):\n counts[user]['seen'].append(event)\n return counts[user]\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "fc1eb426cf811456432a61a306862f8bf804417edb8c8fca6dd3b86c1f9a9940", "started_at": "2026-06-23T12:33:00.066764+00:00", "latency_ms": 37069, "output": "### 根因分析\ndefaultdict 的默认值工厂函数 `lambda: shared` 在为不存在的用户键生成默认值时，始终返回**全局唯一的 `shared` 字典对象的引用**，导致所有用户在 `counts` 中映射到的是同一个字典实例，共享内部的 `seen` 列表。任意用户追加事件时，实际修改的是所有用户共用的全局 `seen` 列表，因此不同用户的计数会互相干扰。\n### 最小修复\n让默认工厂在每次遇到新用户时，创建独立的全新字典实例（包含专属的空 `seen` 列表），而非返回全局共享的 `shared` 对象。\n\nFIX: counts = defaultdict(lambda: {'seen': []})", "usage": {"prompt_tokens": 156, "completion_tokens": 1567, "total_tokens": 1723, "completion_tokens_details": {"reasoning_tokens": 1407}}, "response_id": "resp_0217822179695917097180e458ae3c6e16036676cea376f6ebe0e", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*counts\\s*=\\s*defaultdict\\(lambda\\s*:\\s*\\{'seen'\\s*:\\s*\\[\\]\\}\\)\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-003", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "off-by-one", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*for\\s+start\\s+in\\s+range\\(0,\\s*len\\(items\\),\\s*page_size\\)\\s*:", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面分页代码会漏掉最后不足 page_size 的一页。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 range 行>。\n\n```python\ndef pages(items, page_size):\n out = []\n for start in range(0, len(items) - page_size, page_size):\n out.append(items[start:start + page_size])\n return out\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "c6fe209d60968a3b2d253a065c8ce8255865f73575665a685d957cb52ce82011", "started_at": "2026-06-23T12:33:37.139802+00:00", "latency_ms": 35614, "output": "### Bug 分析\nPython 中 `range` 的结束边界是**开区间**（迭代值不会包含边界本身），原代码错误将 range 的上界设为 `len(items) - page_size`，导致最后一个合法的分页起始索引被排除在迭代范围外：不仅会漏掉末尾不足 `page_size` 的尾页，当 items 总长度恰好是 `page_size` 的整数倍时，甚至会漏掉最后一个满额页。\nPython 切片天然支持结束索引超出序列长度时自动截断到序列末尾，无需手动调整上界做尾页适配。\n\nFIX: for start in range(0, len(items), page_size):", "usage": {"prompt_tokens": 148, "completion_tokens": 1829, "total_tokens": 1977, "completion_tokens_details": {"reasoning_tokens": 1685}}, "response_id": "resp_021782218006622bba9dcb446c9769753d8e06563bdd07f6b8c79", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*for\\s+start\\s+in\\s+range\\(0,\\s*len\\(items\\),\\s*page_size\\)\\s*:"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-004", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "sorting", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*return\\s+sorted\\(ids,\\s*key\\s*=\\s*int\\)\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望把数字字符串按数值排序，但结果是字典序。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 return 行>。\n\n```python\ndef order(ids):\n return sorted(ids)\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "b3f10943f43eaa04524058b0dd7fb30cbd21740003559e79de619ae210a697e4", "started_at": "2026-06-23T12:34:12.757987+00:00", "latency_ms": 12726, "output": "### Bug 说明\nPython 内置的 `sorted()` 对字符串元素默认按**字典序**（逐字符比较 Unicode 码点）排序，原代码没有指定数值维度的排序规则，因此数字字符串不会按实际数值大小排序（例如`\"10\"`会因首字符`'1' < '2'`排在`\"2\"`前面）。\n### 最小修复\n仅需为`sorted()`传入`key=int`参数，指定排序时将每个元素转换为整数作为比较依据，无需改动其他逻辑。\n\nFIX: return sorted(ids, key=int)", "usage": {"prompt_tokens": 115, "completion_tokens": 470, "total_tokens": 585, "completion_tokens_details": {"reasoning_tokens": 341}}, "response_id": "resp_021782218042903b6ab61b028b46eec1be4b86a7f38a82b6cc8f2", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*return\\s+sorted\\(ids,\\s*key\\s*=\\s*int\\)\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-005", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "memoization", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*key\\s*=\\s*\\(fn\\.__name__,\\s*args,\\s*(?:tuple\\(sorted\\(kwargs\\.items\\(\\)\\)\\)|frozenset\\(kwargs\\.items\\(\\)\\))\\)\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面缓存装饰器会把不同参数的调用混在一起。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 key 赋值行>。\n\n```python\ncache = {}\ndef memo(fn):\n def wrapper(*args, **kwargs):\n key = fn.__name__\n if key not in cache:\n cache[key] = fn(*args, **kwargs)\n return cache[key]\n return wrapper\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "fdaf2df9bffeacf442086f01d24eb1fce336e874cebec386f24acd9175c42fc6", "started_at": "2026-06-23T12:34:25.486826+00:00", "latency_ms": 258477, "output": "### Bug 分析\n缓存键`key`仅被赋值为被装饰函数的名称`fn.__name__`，完全没有将调用时传入的位置参数`args`、关键字参数`kwargs`纳入键的计算逻辑，导致同一个函数无论传入什么参数，都映射到同一个缓存条目：第一次调用的结果会被后续所有不同参数的调用直接返回，造成不同参数调用的结果混淆。\n（注：未对关键字参数排序会导致相同键值对但传入顺序不同的kwargs生成不同键，仅影响缓存命中率不影响正确性，属于键生成的细节优化；同名函数缓存冲突是原有全局cache设计的次要问题，不属于题目描述的「不同参数调用混在一起」的核心bug。）\n\n### 最小修复\n保留原有函数名作为函数标识，将位置参数、按key排序后转为可哈希元组的关键字参数共同组成元组作为缓存键，即可区分同函数不同参数的调用，同时保证关键字参数传入顺序不影响键的一致性。\n\nFIX: key = (fn.__name__, args, tuple(sorted(kwargs.items())))", "usage": {"prompt_tokens": 159, "completion_tokens": 12818, "total_tokens": 12977, "completion_tokens_details": {"reasoning_tokens": 12582}}, "response_id": "resp_0217822180549301dd291674643982da436c46312b5f1c9788d4a", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*key\\s*=\\s*\\(fn\\.__name__,\\s*args,\\s*(?:tuple\\(sorted\\(kwargs\\.items\\(\\)\\)\\)|frozenset\\(kwargs\\.items\\(\\)\\))\\)\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-006", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "aliasing", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*copied\\s*=\\s*(?:items\\.copy\\(\\)|items\\[:\\]|list\\(items\\))\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望浅拷贝列表后独立追加，但原列表也被修改。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 copy 行>。\n\n```python\ndef append_copy(items, value):\n copied = items\n copied.append(value)\n return copied\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "d79d9daa2c034ff1badc1d0951de62e17f2421bac20433e520a48c2992cae000", "started_at": "2026-06-23T12:38:43.974579+00:00", "latency_ms": 101230, "output": "### Bug 分析\n`copied = items` 是引用别名赋值，并没有创建新的列表对象：`copied` 和入参 `items` 指向内存中同一个列表实例。此时对 `copied` 调用 `append()` 修改列表结构时，本质是操作二者共享的原列表对象，导致原列表被同步修改，完全没有实现“浅拷贝后独立追加”的预期。\n浅拷贝仅要求外层容器是独立新对象、内部元素复用原引用即可，这种场景下对新列表的追加操作只会修改新列表自身结构，不会影响原列表，符合需求。\n### 最小修复\n无需额外导入模块，使用列表切片语法即可创建原列表的浅拷贝，改动最小、兼容性强，创建的新列表和原列表结构独立，追加操作不会影响原列表。\n\nFIX: copied = items[:]", "usage": {"prompt_tokens": 126, "completion_tokens": 4737, "total_tokens": 4863, "completion_tokens_details": {"reasoning_tokens": 4552}}, "response_id": "resp_021782218313900081222809c6b24dcd35df0bef1acd71dc9bada", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*copied\\s*=\\s*(?:items\\.copy\\(\\)|items\\[:\\]|list\\(items\\))\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-007", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "json", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*data\\s*=\\s*(payload\\s+if\\s+isinstance\\(payload,\\s*dict\\)\\s+else\\s+json\\.loads\\(payload\\)|json\\.loads\\(payload\\)\\s+if\\s+isinstance\\(payload,\\s*\\(str,\\s*bytes,\\s*bytearray\\)\\)\\s+else\\s+payload|json\\.loads\\(payload\\)\\s+if\\s+isinstance\\(payload,\\s*str\\)\\s+else\\s+payload)\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望解析 JSON 字符串，但传入已经是 dict 时会报错。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 data 赋值行>。\n\n```python\nimport json\n\ndef load(payload):\n data = json.loads(payload)\n return data['id']\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "dae0481dbfca35623cf6eaf5f44910cc59204edc238411f6ec1221586215b6a3", "started_at": "2026-06-23T12:40:25.210418+00:00", "latency_ms": 24118, "output": "### Bug 说明\n`json.loads()` 的作用是把**JSON格式的字符串/字节/字节数组**反序列化为Python对象（如dict），它不接受dict类型作为入参。当前代码没有对入参类型做判断，当传入的`payload`已经是dict时，调用`json.loads(payload)`会抛出`TypeError: the JSON object must be str, bytes or bytearray, not dict`，导致报错。\n\n### 最小修复思路\n仅修改`data`的赋值逻辑：如果入参已经是dict就直接使用，否则再走`json.loads`反序列化，不改动其余代码逻辑。\n\nFIX: data = payload if isinstance(payload, dict) else json.loads(payload)", "usage": {"prompt_tokens": 129, "completion_tokens": 1083, "total_tokens": 1212, "completion_tokens_details": {"reasoning_tokens": 925}}, "response_id": "resp_0217822184147227567f706121169fe4c693c57ce45241c9deb06", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*data\\s*=\\s*(payload\\s+if\\s+isinstance\\(payload,\\s*dict\\)\\s+else\\s+json\\.loads\\(payload\\)|json\\.loads\\(payload\\)\\s+if\\s+isinstance\\(payload,\\s*\\(str,\\s*bytes,\\s*bytearray\\)\\)\\s+else\\s+payload|json\\.loads\\(payload\\)\\s+if\\s+isinstance\\(payload,\\s*str\\)\\s+else\\s+payload)\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-008", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "finally", "debugging"], "scorer": "contains_any", "expected": {"answers": ["删除 finally 中的 return None", "去掉 finally 的 return None", "不要在 finally 里 return None", "remove return None", "去掉 finally 中的 return None", "删除或替换 finally 中的 return None", "去掉 finally 中的 `return None`", "删除或替换 finally 中的 `return None`", "删除 finally 块中的 `return None`", "删除 finally 块中的 return None", "删除 finally 中的 `return None`", "删除 `finally` 块中的 `return None`", "删除 `finally` 中的 `return None`", "删除 `finally: return None`", "将 `finally:` 中的 `return None` 删除", "finally 中的 return None 删除", "finally: 中的 return None 删除", "删除 finally 块中的 'return None'", "删除 finally 子句中的 `return None`", "删除 finally 子句中的 return None", "delete the line `return None` inside the finally block", "delete the line return None inside the finally block", "finally 块不应包含 return 语句", "finally 块不应包含 return", "finally 不应包含 return"]}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望 finally 后仍保留 try 中返回值，但实际总是返回 None。请指出 bug，并给出最小修复。最后单独一行写 FIX: <要删除或替换的 finally 行说明>。\n\n```python\ndef read_value():\n try:\n return 42\n finally:\n return None\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "3cc6ae8fe4f15528225f040b759f5535e31e0ee3dab226c836ed3b9c19707a57", "started_at": "2026-06-23T12:40:49.331588+00:00", "latency_ms": 123095, "output": "### Bug 说明\nPython 的`try/finally`控制流语义规定：当`try`块执行到`return`时，会先暂存待返回值，再强制执行`finally`块逻辑；如果`finally`块中存在显式`return`语句，会直接丢弃`try`中暂存的返回值，将`finally`的返回值作为函数最终结果（还会吞掉`try`中未处理的异常）。本代码在`finally`中写了`return None`，直接覆盖了`try`块暂存的返回值42，因此始终返回None。\n### 最小修复\n移除`finally`块中会覆盖返回值的`return None`语句；由于Python语法要求`finally`后必须跟非空代码块，若无实际清理操作，将该行替换为`pass`占位即可（若有清理逻辑则编写无return的清理代码），此时执行完`finally`后会正常返回`try`中的42。\n\nFIX: 将finally块中的`return None`替换为`pass`（或无return的清理逻辑），移除finally内的return语句避免覆盖try块返回值", "usage": {"prompt_tokens": 133, "completion_tokens": 5994, "total_tokens": 6127, "completion_tokens_details": {"reasoning_tokens": 5742}}, "response_id": "resp_0217822184389107e77ed289647c83a4aab1892556246c59300cf", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"manual_review_override": true, "original_score": {"passed": false, "score": 0.0, "details": {"matched": []}}, "reason": "人工复核通过：模型明确指出 finally 中的 return None 会覆盖 try 的返回值，并给出将其替换为 pass/无 return 清理逻辑的最小修复；原 contains_any scorer 未覆盖该等价表达。"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl", "original_score": {"passed": false, "score": 0.0, "details": {"matched": []}}} {"case_id": "challenge-reasoning-001", "suite": "reasoning", "case_version": "0.2.0", "tags": ["arithmetic", "multi-step"], "scorer": "exact", "expected": {"answer": "125", "normalize": true}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只回答最终数字：有 A、B、C、D 四个服务。A 每分钟处理 6 个请求，B 比 A 少 2 个，C 是 B 的 2 倍，D 是 A 和 C 的平均值。四个服务 5 分钟一共处理多少请求？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "06f9a3eb3666e5854b2b742553674373622a42e326be166231a39c67b5604810", "started_at": "2026-06-23T12:42:52.432326+00:00", "latency_ms": 5931, "output": "125", "usage": {"prompt_tokens": 128, "completion_tokens": 230, "total_tokens": 358, "completion_tokens_details": {"reasoning_tokens": 227}}, "response_id": "resp_021782218562093d8d1efa022ab0c380e7ecdb91b19d20a69b67e", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"expected": "125", "actual": "125"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-reasoning-002", "suite": "reasoning", "case_version": "0.2.0", "tags": ["constraint-solving"], "scorer": "exact", "expected": {"answer": "D", "normalize": true}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只回答一个字母：五个任务 A、B、C、D、E 排队。A 必须在 B 前，B 必须在 C 前；D 不能在两端；E 必须紧挨在 C 后面。在所有满足条件的队列中，按字典序最小的队列第三个任务是谁？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "e273e5a0cd57aef19af9e4ccfcd445a7523180f5cab62526c14aa0e62d5c1cc9", "started_at": "2026-06-23T12:42:58.364682+00:00", "latency_ms": 171837, "output": "D", "usage": {"prompt_tokens": 128, "completion_tokens": 9271, "total_tokens": 9399, "completion_tokens_details": {"reasoning_tokens": 9270}}, "response_id": "resp_021782218567937b8b2db5ad8d9605f3baecb70a2647fd8dfb29c", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"expected": "D", "actual": "D"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-reasoning-003", "suite": "reasoning", "case_version": "0.2.0", "tags": ["binary", "symbolic"], "scorer": "exact", "expected": {"answer": "0", "normalize": true}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只回答最终数字：一个 4 位二进制数从 1011 开始，先左移一位并只保留低 4 位，再与 0110 做异或，最后转成十进制。结果是多少？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "813f5aead205f4aa7a9d2a1e7b7085daae002a54592edd8d8aad2e7d7551897c", "started_at": "2026-06-23T12:45:50.209480+00:00", "latency_ms": 47264, "output": "0", "usage": {"prompt_tokens": 115, "completion_tokens": 2424, "total_tokens": 2539, "completion_tokens_details": {"reasoning_tokens": 2423}}, "response_id": "resp_021782218740205e4feb8d226730241afb8b5c4c4c35b32c38eb7", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"expected": "0", "actual": "0"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-reasoning-004", "suite": "reasoning", "case_version": "0.2.0", "tags": ["set", "operations"], "scorer": "exact", "expected": {"answer": "18", "normalize": true}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只回答最终数字：一个集合最初为 {1,2,3,4,5}。依次执行：删除所有偶数；加入 6 和 7；把每个元素乘以 2；删除大于 10 的元素。最终集合元素之和是多少？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "3077e41c160e4ebe361234b76e647f42583619ef446b73e7c1e25858312f328e", "started_at": "2026-06-23T12:46:37.476714+00:00", "latency_ms": 11866, "output": "18", "usage": {"prompt_tokens": 117, "completion_tokens": 553, "total_tokens": 670, "completion_tokens_details": {"reasoning_tokens": 551}}, "response_id": "resp_021782218786907e04d0ce0cb7c9c575f1fe60733b212fd6c59fc", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"expected": "18", "actual": "18"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-reasoning-005", "suite": "reasoning", "case_version": "0.2.0", "tags": ["logic", "sat"], "scorer": "exact", "expected": {"answer": "可满足", "normalize": true}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只回答“可满足”或“不可满足”：有三个布尔变量 x、y、z。约束为：x 或 y 为真；如果 x 为真则 z 为假；如果 y 为真则 z 为真；x 和 y 不能同时为真。是否存在赋值满足全部约束？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "1e534bd21af5e75f0c9fd61e3d5a3f660355d4555e34e16a06744234da47e006", "started_at": "2026-06-23T12:46:49.343855+00:00", "latency_ms": 43535, "output": "可满足", "usage": {"prompt_tokens": 131, "completion_tokens": 2214, "total_tokens": 2345, "completion_tokens_details": {"reasoning_tokens": 2212}}, "response_id": "resp_021782218798783ca53c98b4a9847edf4b59a94723c611e1bef56", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"expected": "可满足", "actual": "可满足"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-001", "suite": "rag_long_context", "case_version": "0.2.0", "tags": ["rag", "grounded"], "scorer": "text_rules", "expected": {"required": ["人工抽检"], "required_any": [["默认使用", "默认判分", "默认采用", "作为默认判分"]], "forbidden": ["因为成本太高", "因为不支持中文"], "max_chars": 160}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料 A：DracoBench v0.2 的目标是提高区分度，优先增加 coding、debugging、RAG 和 instruction_following 题。\n资料 B：DracoBench v0.2 不会默认使用 LLM-as-judge；开放中文写作题只做少量规则校验和人工抽检。\n资料 C：v0.2 的 HTML 报告必须展示通过率、成本、reasoning tokens、失败样例和逐题明细。\n\n问题：v0.2 为什么没有把 LLM-as-judge 作为默认判分？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "7a40b395b42a59b621957d70755d39fca26dba23f4e0c46743018935350319d3", "started_at": "2026-06-23T12:47:32.881656+00:00", "latency_ms": 52396, "output": "现有给定资料未说明DracoBench v0.2未将LLM-as-judge作为默认判分方式的原因，仅明确提及两项相关规则：一是该版本不会默认使用LLM-as-judge；二是开放中文写作题只做少量规则校验和人工抽检。", "usage": {"prompt_tokens": 185, "completion_tokens": 2513, "total_tokens": 2698, "completion_tokens_details": {"reasoning_tokens": 2452}}, "response_id": "resp_0217822188425505c847dce3dfef19d8a8a6e4aeb40e68b1f8ac1", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 109, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-002", "suite": "rag_long_context", "case_version": "0.2.0", "tags": ["rag", "tradeoff", "abstain"], "scorer": "text_rules", "expected": {"required": ["Beta", "成本", "延迟", "判断"], "forbidden": ["Alpha 更适合", "多模态更强"], "max_chars": 180}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答，不要自行补充。\n\n资料：模型 Alpha 在 30 道 challenge 题中通过 24 道，成本 0.08 美元，平均延迟 9 秒。模型 Beta 通过 22 道，成本 0.02 美元，平均延迟 3 秒。资料没有给出两者在多模态任务上的表现。\n\n问题：如果要写低成本高吞吐摘要服务，资料更支持选择哪个模型？另外，多模态表现能否判断？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "b25dfce277314b219ab8ccef6f02fb644e23cd169765867993ae58ccd7258c4a", "started_at": "2026-06-23T12:48:25.281452+00:00", "latency_ms": 23968, "output": "1. 资料更支持选择模型Beta：模型Beta成本为0.02美元，低于模型Alpha的0.08美元，符合低成本要求；同时模型Beta平均延迟为3秒，短于模型Alpha的9秒，更适配高吞吐的服务需求。\n2. 两者的多模态表现无法判断，因为资料没有给出两个模型在多模态任务上的表现相关信息。", "usage": {"prompt_tokens": 171, "completion_tokens": 1171, "total_tokens": 1342, "completion_tokens_details": {"reasoning_tokens": 1084}}, "response_id": "resp_021782218894737f6bbcadc5ac62a9c6cfd677b3d413815901904", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 143, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-003", "suite": "rag_long_context", "case_version": "0.2.0", "tags": ["rag", "nuance"], "scorer": "text_rules", "expected": {"required": ["题面歧义"], "required_any": [["47/50", "47 题", "47题"], ["空输出", "输出为空", "未输出"]], "forbidden": ["满分", "完全失败"], "max_chars": 240}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料：一次评测中，K 模型在 smoke set 上 7/7，通过率 100%；在 hard set 上原始结果 47/50，其中 2 题后来被认定为题面歧义，单独复测通过。另有 1 题因 reasoning tokens 用尽而空输出。\n\n问题：更公平的文字结论应该如何描述 K 模型这次 hard set 表现？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "12db30955792c48368632f01aa0a9c06e9fc27004c4dbf12ce602cb8488494e6", "started_at": "2026-06-23T12:48:49.251557+00:00", "latency_ms": 142638, "output": "更公平的文字结论需完整披露全部客观评测信息，不片面截取单一原始结果，不添加无依据的主观推断，可描述为：\n本次hard set共50题，K模型原始作答正确47题；原始失分的3题中，2题后续被认定为题面存在歧义，模型在这2题的单独复测中均作答通过；剩余1题的失分原因为模型reasoning tokens用尽产生空输出。", "usage": {"prompt_tokens": 156, "completion_tokens": 6283, "total_tokens": 6439, "completion_tokens_details": {"reasoning_tokens": 6187}}, "response_id": "resp_0217822189189327b2c08adb457386e4dc9fc8de61d86c7fa8dbd", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"manual_review_override": true, "original_score": {"passed": false, "score": 0.0, "details": {"missing": ["题面歧义"], "missing_any": [], "present_forbidden": [], "char_count": 158, "length_ok": true, "length_scored": false}}, "reason": "人工复核通过：模型完整描述 hard set 原始 47/50、2 题题面存在歧义且单独复测通过、1 题 reasoning tokens 用尽空输出；原 text_rules 只匹配“题面歧义”字面短语，未覆盖“题面存在歧义”。"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl", "original_score": {"passed": false, "score": 0.0, "details": {"missing": ["题面歧义"], "missing_any": [], "present_forbidden": [], "char_count": 158, "length_ok": true, "length_scored": false}}} {"case_id": "challenge-rag-004", "suite": "rag_long_context", "case_version": "0.2.0", "tags": ["rag", "abstain"], "scorer": "text_rules", "expected": {"required": ["不能", "不包含", "利润率"], "forbidden": ["可以判断", "训练数据"], "max_chars": 100}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答，不要使用资料外知识。\n\n资料：报告字段包括 pass_rate、avg_latency_ms、total_cost、prompt_tokens、completion_tokens、reasoning_tokens。报告不包含模型训练数据来源，也不包含供应商真实利润率。\n\n问题：根据资料，能否从报告判断供应商利润率？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "cb2d9baeb98b0b4836972058965359b461f03aaba6cd096f40ffc813ce331880", "started_at": "2026-06-23T12:51:11.896248+00:00", "latency_ms": 7141, "output": "不能。根据资料，报告不包含供应商真实利润率，因此无法从报告判断供应商利润率。", "usage": {"prompt_tokens": 128, "completion_tokens": 143, "total_tokens": 271, "completion_tokens_details": {"reasoning_tokens": 123}}, "response_id": "resp_021782219061517cfcda9ed8a69b2edb3386bd623e2b8f0ac69aa", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 38, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-if-001", "suite": "instruction_following", "case_version": "0.2.0", "tags": ["json", "nested"], "scorer": "json_schema_lite", "expected": {"schema": {"type": "object", "required": ["status", "scores", "note"], "additionalProperties": false, "properties": {"status": {"type": "string", "const": "review"}, "scores": {"type": "object", "required": ["coding", "debugging", "rag"], "additionalProperties": false, "properties": {"coding": {"type": "integer"}, "debugging": {"type": "integer"}, "rag": {"type": "integer"}}}, "note": {"type": "string"}}}}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 JSON 对象，不要 Markdown。字段 status 必须是 \"review\"；字段 scores 必须是对象，包含 coding、debugging、rag 三个整数；字段 note 必须是不超过 12 个汉字的中文字符串。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "7f670ae329c67bf1957a977815256b403ae7965cb15f2d2db520256cc3f2eb77", "started_at": "2026-06-23T12:51:19.038713+00:00", "latency_ms": 8219, "output": "{\"status\":\"review\",\"scores\":{\"coding\":85,\"debugging\":82,\"rag\":88},\"note\":\"能力表现较为均衡\"}", "usage": {"prompt_tokens": 115, "completion_tokens": 312, "total_tokens": 427, "completion_tokens_details": {"reasoning_tokens": 281}}, "response_id": "resp_02178221906863314c42a60562dfb6c540ee32d577db546ee54bd", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"errors": [], "value": {"status": "review", "scores": {"coding": 85, "debugging": 82, "rag": 88}, "note": "能力表现较为均衡"}}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-if-002", "suite": "instruction_following", "case_version": "0.2.0", "tags": ["format", "line-control"], "scorer": "regex", "expected": {"pattern": "^\\s*BEGIN\\s*\\n[^\\n]*模型评测[^\\n]*\\n[^\\n]*HTML报告[^\\n]*\\nEND\\s*$", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 4 行，不要解释。第 1 行是 BEGIN，第 4 行是 END。第 2 行必须包含“模型评测”，第 3 行必须包含“HTML报告”。不能出现任何项目符号。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "773b2e7398be979e738e08c3c8486ca69bae0f71d9a2abe3430cef775f5a59fb", "started_at": "2026-06-23T12:51:27.258827+00:00", "latency_ms": 11911, "output": "BEGIN\n严谨规范的模型评测是衡量人工智能系统综合能力的重要依据\n待所有评测流程结束后系统会自动生成内容详实的HTML报告\nEND", "usage": {"prompt_tokens": 116, "completion_tokens": 377, "total_tokens": 493, "completion_tokens_details": {"reasoning_tokens": 343}}, "response_id": "resp_0217822190768656bc07008344feaee43478f55fd03b6b4ff6bbd", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "^\\s*BEGIN\\s*\\n[^\\n]*模型评测[^\\n]*\\n[^\\n]*HTML报告[^\\n]*\\nEND\\s*$"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-if-003", "suite": "instruction_following", "case_version": "0.2.0", "tags": ["constraint", "zh"], "scorer": "text_rules", "expected": {"required": ["难题", "区分度"], "forbidden": ["最强", "吊打", "遥遥领先"], "max_chars": 28}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出一个中文句子，不能超过 28 个汉字。必须同时包含“难题”和“区分度”，不能包含“最强”“吊打”“遥遥领先”。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "63d393d3332e09c8d34bfa5fceb4812734bf76c2333f7e6d1585cf977944f0db", "started_at": "2026-06-23T12:51:39.172383+00:00", "latency_ms": 25373, "output": "设置合理的难题可有效提升试题的区分度。", "usage": {"prompt_tokens": 100, "completion_tokens": 1141, "total_tokens": 1241, "completion_tokens_details": {"reasoning_tokens": 1129}}, "response_id": "resp_0217822190891312691676f614000b9b6d45a3dc59ec4583c7af2", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 19, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-zh-writing-001", "suite": "chinese_writing", "case_version": "0.2.0", "tags": ["wechat", "conclusion"], "scorer": "text_rules", "expected": {"required": ["简单题", "真实能力", "失败样例"], "forbidden": ["吊打"], "max_chars": 120}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "写一段不超过 120 个汉字的公众号结论，主题是“模型在简单题满分不代表真实强”。必须包含“简单题”“真实能力”“失败样例”，不能出现“吊打”。", "parameters": {"temperature": 0.2, "max_tokens": 16384, "seed": null}, "prompt_hash": "f330d4a7913b830c15df0d2a36cf0a9e5c07628ef6fa52408cc1bd2e90074762", "started_at": "2026-06-23T12:52:04.548495+00:00", "latency_ms": 42579, "output": "别被模型在简单题上拿满分的跑分迷惑，这从来不是硬实力的全部证明。要摸准模型的真实能力，就得跳出熟题舒适区，多挖掘隐蔽的失败样例做校验，才不会在实际落地时踩意料之外的坑。", "usage": {"prompt_tokens": 111, "completion_tokens": 2092, "total_tokens": 2203, "completion_tokens_details": {"reasoning_tokens": 2033}}, "response_id": "resp_02178221911400614c42a60562dfb6c540ee32d577db5465914aa", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 84, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-zh-writing-002", "suite": "chinese_writing", "case_version": "0.2.0", "tags": ["rewrite", "zh"], "scorer": "text_rules", "expected": {"required": ["复杂调试", "稳定性"], "forbidden": ["猛", "掉链子"], "max_chars": 80}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "把下面口语句改写成专业表达，不超过 80 个汉字：这个模型看起来很猛，但一遇到复杂调试就开始掉链子。必须包含“复杂调试”和“稳定性”，不能包含“猛”“掉链子”。", "parameters": {"temperature": 0.2, "max_tokens": 16384, "seed": null}, "prompt_hash": "1628d5e9cbea22d0750b547f0818eafa82d503dcad8fdc41ef2f523f9fbb55a1", "started_at": "2026-06-23T12:52:47.130183+00:00", "latency_ms": 48580, "output": "该模型表观性能优异，但在开展复杂调试时易出现运行异常，暴露出其稳定性存在明显短板，有待针对性优化提升。", "usage": {"prompt_tokens": 114, "completion_tokens": 2374, "total_tokens": 2488, "completion_tokens_details": {"reasoning_tokens": 2345}}, "response_id": "resp_02178221915680598c470d32e326dd7109413e831d170d8aa83a6", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 51, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-009", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "aggregation", "dirty-input"], "scorer": "code_python_tests", "expected": {"tests": ["events=[{'user':'b','type':'add','amount':'5'},{'user':'a','type':'add','amount':3},{'user':'b','type':'remove','amount':2},{'user':'a','type':'remove','amount':3}]\nassert solution.solve(events) == {'b': 3}", "assert solution.solve([]) == {}", "assert solution.solve([{'user':'x','type':'remove','amount':'2'}]) == {'x': -2}"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(events: list[dict]) -> dict[str, int]。每个事件包含 user、type、amount；type 为 \"add\" 或 \"remove\"。按 user 汇总 add 减 remove 后的库存，结果中去掉库存为 0 的用户，按 key 字典序返回普通 dict。amount 可能是字符串数字。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "9b81c7f8e34dbf12cfabd35ebb618802fe09200deca1d146558b4e1c12e6fd1e", "started_at": "2026-06-23T12:53:35.713842+00:00", "latency_ms": 36057, "output": "def solve(events: list[dict]) -> dict[str, int]:\n stock = {}\n for event in events:\n user = event[\"user\"]\n op_type = event[\"type\"]\n amount = int(event[\"amount\"])\n if user not in stock:\n stock[user] = 0\n if op_type == \"add\":\n stock[user] += amount\n elif op_type == \"remove\":\n stock[user] -= amount\n return {k: v for k, v in sorted(stock.items()) if v != 0}", "usage": {"prompt_tokens": 144, "completion_tokens": 1847, "total_tokens": 1991, "completion_tokens_details": {"reasoning_tokens": 1720}}, "response_id": "resp_021782219205293275a44fd862e8b57dcd33414ced196963ea8f8", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-010", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "parser", "expression"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve('1 + 2 - 3') == 0", "assert solution.solve('10 - (2 + 3) + 4') == 9", "assert solution.solve('(1+(4+5+2)-3)+(6+8)') == 23"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(expr: str) -> int。expr 只包含非负整数、+、-、空格和括号，计算表达式结果。不要使用 eval。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "662cadd9b4e3fdcfdf38e1d7338201fe3049d948576c89303695cff58950b3b4", "started_at": "2026-06-23T12:54:11.820716+00:00", "latency_ms": 186454, "output": "```python\ndef solve(expr: str) -> int:\n res = 0\n num = 0\n sign = 1\n stack = [1]\n for c in expr:\n if c.isdigit():\n num = num * 10 + int(c)\n elif c in '+-':\n res += sign * num\n num = 0\n sign = stack[-1] * (1 if c == '+' else -1)\n elif c == '(':\n stack.append(sign)\n elif c == ')':\n res += sign * num\n num = 0\n stack.pop()\n res += sign * num\n return res\n```", "usage": {"prompt_tokens": 102, "completion_tokens": 10238, "total_tokens": 10340, "completion_tokens_details": {"reasoning_tokens": 10081}}, "response_id": "resp_0217822192414563587b81e5d6fed80d7b8fbd8e8cb3fa6d58b51", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-011", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "sorting", "dedupe"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve(['alice 10','bob 12','alice 15','','bob 11']) == [('alice',15),('bob',12)]", "assert solution.solve(['x -1','a -1','b 0']) == [('b',0),('a',-1),('x',-1)]"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(rows: list[str]) -> list[tuple[str, int]]。每行格式为 \"name score\"，空白数量不定；同名保留最高分；输出按分数降序、同分按 name 升序。忽略空行。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "332bccf5652506856c633d7093431279279a22b2d3a99762f6cba55df7d884af", "started_at": "2026-06-23T12:57:18.321018+00:00", "latency_ms": 308206, "output": "```python\ndef solve(rows: list[str]) -> list[tuple[str, int]]:\n score_map = {}\n for line in rows:\n parts = line.split()\n if not parts:\n continue\n score = int(parts[-1])\n name = ' '.join(parts[:-1])\n if name not in score_map or score > score_map[name]:\n score_map[name] = score\n return sorted(score_map.items(), key=lambda x: (-x[1], x[0]))\n```", "usage": {"prompt_tokens": 119, "completion_tokens": 15533, "total_tokens": 15652, "completion_tokens_details": {"reasoning_tokens": 15413}}, "response_id": "resp_021782219427890689dd86c635c9e312c9248b59ca51b2016240b", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-012", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "wildcard", "dp"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve('a*b?d', 'axxxbc d'.replace(' ','')) is True", "assert solution.solve('a*c', 'ab') is False", "assert solution.solve('*', '') is True", "assert solution.solve('file-??.txt', 'file-ab.txt') is True"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(pattern: str, text: str) -> bool。pattern 支持 ? 匹配任意单字符，* 匹配任意长度字符串。必须匹配完整 text。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "0ac24584e5856726a6e962c43819ac539910b115bdabb48f973f04258dd562d6", "started_at": "2026-06-23T13:02:26.579678+00:00", "latency_ms": 147676, "output": "```python\ndef solve(pattern: str, text: str) -> bool:\n m, n = len(pattern), len(text)\n i = j = 0\n star = -1\n match = 0\n while j < n:\n if i < m and (pattern[i] == text[j] or pattern[i] == '?'):\n i += 1\n j += 1\n elif i < m and pattern[i] == '*':\n star = i\n match = j\n i += 1\n elif star != -1:\n i = star + 1\n match += 1\n j = match\n else:\n return False\n while i < m and pattern[i] == '*':\n i += 1\n return i == m\n```", "usage": {"prompt_tokens": 105, "completion_tokens": 8367, "total_tokens": 8472, "completion_tokens_details": {"reasoning_tokens": 8183}}, "response_id": "resp_021782219736204618be72f3907769bd1896ffa0351d57580eebe", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-013", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "recursion", "flatten"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve({'a': {'b': 1, 'c': 'x'}, 'd': 2}) == ['a.b=1','a.c=x','d=2']", "assert solution.solve({'z': {'a': {'b': 3}}}) == ['z.a.b=3']", "assert solution.solve({}) == []"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(tree: dict) -> list[str]。tree 是嵌套字典，叶子是字符串或数字。返回所有叶子的路径，路径用点连接，按字典序排序。例如 {'a': {'b': 1}} -> ['a.b=1']。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "e658784bcc7118c9f5e1375c7f1f92aa8147c38fb8790138603de7f2b3722c98", "started_at": "2026-06-23T13:04:54.307086+00:00", "latency_ms": 48141, "output": "```python\ndef solve(tree: dict) -> list[str]:\n res = []\n def dfs(node, path_parts):\n for key, value in node.items():\n new_parts = path_parts + [str(key)]\n if isinstance(value, dict):\n dfs(value, new_parts)\n else:\n res.append(f\"{'.'.join(new_parts)}={value}\")\n dfs(tree, [])\n return sorted(res)\n```", "usage": {"prompt_tokens": 126, "completion_tokens": 2390, "total_tokens": 2516, "completion_tokens_details": {"reasoning_tokens": 2282}}, "response_id": "resp_02178221988420651482108684de07484f06ebb12145571759f84", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-014", "suite": "coding", "case_version": "0.2.0", "tags": ["python", "dynamic-programming"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve([1,-2,0,3]) == 4", "assert solution.solve([1,-2,-2,3]) == 3", "assert solution.solve([-1,-1,-1]) == -1", "assert solution.solve([2,1,-2,3]) == 6"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(nums: list[int]) -> int。返回删除恰好一个元素后，剩余数组的最大连续子数组和。nums 长度至少为 2，可以全为负数。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "a7a5de11dacd1864c7d44a3ce983c2f127d1bcff88bec3b15f8727c513f56ea2", "started_at": "2026-06-23T13:05:42.488242+00:00", "latency_ms": 266526, "output": "", "usage": {"prompt_tokens": 108, "completion_tokens": 16384, "total_tokens": 16492, "completion_tokens_details": {"reasoning_tokens": 16384}}, "response_id": "resp_021782219932068e542ae8f7c04736102ead861a601340c2be258", "finish_reason": "incomplete", "score": {"passed": false, "score": 0.0, "details": {"returncode": 1, "stdout": "", "stderr": "Traceback (most recent call last):\n File \"/var/folders/x_/4cht1kfj1_j31gdv5w01sgmw0000gn/T/dracobench-code-2tl7e5mu/test_solution.py\", line 3, in \n assert solution.solve([1,-2,0,3]) == 4\n ^^^^^^^^^^^^^^\nAttributeError: module 'solution' has no attribute 'solve'\n"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-009", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "parsing", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*return\\s+\\[part\\.strip\\(\\)\\s+for\\s+part\\s+in\\s+s\\.split\\('\\,'\\)\\s+if\\s+part\\.strip\\(\\)\\]\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望把输入按逗号拆分并去掉空白，但会把空字段也保留下来。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 return 行>。\n\n```python\ndef parse(s):\n return [part.strip() for part in s.split(',')]\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "fc78c43e70a6a2fe59904be00b93d579300af6ba72fe5a9629bfe5bd82fad3d6", "started_at": "2026-06-23T13:10:09.079028+00:00", "latency_ms": 100174, "output": "### Bug 说明\n代码按逗号分割字符串后，仅对每个分段做了首尾空白去除，但没有过滤掉经`strip()`处理后为空字符串的分段。这类空分段来源于三种场景：字符串以逗号开头/结尾时`split`产生的首尾空串、连续逗号之间无内容产生的中间空串、逗号之间仅含空白字符经`strip()`后变为空串，最终导致结果中保留了不符合预期的空字段。\n### 最小修复\n仅需在原有列表推导式中增加过滤条件，保留`strip()`后非空的分段即可，无需改动原有分割、去空白的核心逻辑。\n\nFIX: return [part.strip() for part in s.split(',') if part.strip()]", "usage": {"prompt_tokens": 127, "completion_tokens": 5027, "total_tokens": 5154, "completion_tokens_details": {"reasoning_tokens": 4871}}, "response_id": "resp_0217822201986191b28b806a7bc9927371d1855db3264b920b779", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*return\\s+\\[part\\.strip\\(\\)\\s+for\\s+part\\s+in\\s+s\\.split\\('\\,'\\)\\s+if\\s+part\\.strip\\(\\)\\]\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-010", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "datetime", "debugging"], "scorer": "contains_any", "expected": {"answers": ["datetime.fromisoformat", "dateutil.parser", "解析 ts 后排序", "parse ts", "fromisoformat", "int(r['ts'])", "float(r['ts'])", "key=lambda r: int(r['ts'])", "key=lambda r: float(r['ts'])"]}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望用 UTC 时间戳排序，但字符串排序会错。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 sort 行>。\n\n```python\nrows.sort(key=lambda r: r['ts'])\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "c0d491a68f215902d8b314423a6ab9aea7ac0782e1a34e6d6997670c2ddb8458", "started_at": "2026-06-23T13:11:49.259309+00:00", "latency_ms": 163983, "output": "### Bug 说明\n当前排序直接使用`r['ts']`的原始字符串值作为排序键，按字符串字典序逐字符比较大小，而非按UTC时间戳对应的数值大小（即实际时间先后）比较。对于字符串形式的数值型UTC时间戳，当字符串长度不一致时（如2001年9月前的秒级时间戳为9位，之后为10位），字典序与数值序不匹配（例如字符串比较下`\"999999999\" > \"1700000000\"`，但数值上后者对应时间更晚），最终导致排序结果不符合时间先后预期。\n### 最小修复\n将排序键从原始字符串转换为整数（时间戳常规存储形式），通过数值大小比较实现正确的时间先后排序。\n\nFIX: rows.sort(key=lambda r: int(r['ts']))", "usage": {"prompt_tokens": 114, "completion_tokens": 9500, "total_tokens": 9614, "completion_tokens_details": {"reasoning_tokens": 9305}}, "response_id": "resp_021782220298835fba75098d9ee455dd9478ca57adcb3fdaa1687", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"matched": ["int(r['ts'])", "key=lambda r: int(r['ts'])"]}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-011", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "truthiness", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*(?:limit\\s*=\\s*10\\s+if\\s+limit\\s+is\\s+None\\s+else\\s+limit|limit\\s*=\\s*limit\\s+if\\s+limit\\s+is\\s+not\\s+None\\s+else\\s+10|if\\s+limit\\s+is\\s+None\\s*:\\s*limit\\s*=\\s*10)\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码在输入包含 0 时会误判为默认值 10。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 limit 赋值行>。\n\n```python\ndef query(limit=None):\n limit = limit or 10\n return list(range(limit))\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "81dfcde4613fe16d52b7386d005b50beaafcb9916e6cb6c0d326eab3c808a843", "started_at": "2026-06-23T13:14:33.249819+00:00", "latency_ms": 34026, "output": "### Bug 说明\n代码使用`limit or 10`设置默认值的逻辑存在真值判断缺陷：Python中`or`是短路逻辑运算符，左操作数为布尔假值时会直接返回右操作数。`0`是合法的limit输入，但属于布尔假值，因此显式传入`limit=0`时，`0 or 10`会错误返回默认值10，该逻辑无法区分「未传参（limit为默认值None）」和「传入有效假值0」的场景。\n### 最小修复思路\n放弃真值判断，显式检查参数是否为默认哨兵值`None`，仅当未传入参数（limit为None）时使用默认值10，其余所有显式传入的值（包括0）均保留用户输入。\n\nFIX: limit = 10 if limit is None else limit", "usage": {"prompt_tokens": 131, "completion_tokens": 1511, "total_tokens": 1642, "completion_tokens_details": {"reasoning_tokens": 1325}}, "response_id": "resp_021782220463069d5261995f9623e565b5d66b904398165eee27f", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*(?:limit\\s*=\\s*10\\s+if\\s+limit\\s+is\\s+None\\s+else\\s+limit|limit\\s*=\\s*limit\\s+if\\s+limit\\s+is\\s+not\\s+None\\s+else\\s+10|if\\s+limit\\s+is\\s+None\\s*:\\s*limit\\s*=\\s*10)\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-012", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "deepcopy", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*copied\\s*=\\s*(?:copy\\.deepcopy\\(config\\)|__import__\\(['\\\"]copy['\\\"]\\)\\.deepcopy\\(config\\))\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望复制嵌套配置后修改副本，但原配置也被修改。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 copied 赋值行>。\n\n```python\ndef update(config):\n copied = dict(config)\n copied['db']['host'] = 'localhost'\n return copied\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "b9ce1efc39e6db4861310a8b269b4f7a5f68b90016ca7ebbef71eb8b6b011913", "started_at": "2026-06-23T13:15:07.278859+00:00", "latency_ms": 49613, "output": "### Bug 原因\n`dict(config)` 执行的是**浅拷贝**：仅创建了新的顶层字典，顶层存储的嵌套可变对象（比如这里的 `db` 子字典）仍然是原配置中同一个对象的引用，副本和原配置共享所有内层嵌套结构。因此修改 `copied['db']['host']` 时，实际改动的是双方共享的子字典，导致原配置被意外篡改。\n### 修复说明\n使用标准库 `copy` 模块的 `deepcopy` 做递归深拷贝，完全复制所有层级的嵌套结构，让副本和原配置完全独立（需提前在文件头部添加 `import copy`）。\n\nFIX: copied = copy.deepcopy(config)", "usage": {"prompt_tokens": 132, "completion_tokens": 2404, "total_tokens": 2536, "completion_tokens_details": {"reasoning_tokens": 2247}}, "response_id": "resp_021782220496800226d55c1868491c89c6234734c8c3308934bfe", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*copied\\s*=\\s*(?:copy\\.deepcopy\\(config\\)|__import__\\(['\\\"]copy['\\\"]\\)\\.deepcopy\\(config\\))\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-013", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "generator", "debugging"], "scorer": "regex", "expected": {"pattern": "\\*{0,2}FIX\\*{0,2}:\\*{0,2}\\s*`?\\s*return\\s+next\\(\\(u\\s+for\\s+u\\s+in\\s+users\\s+if\\s+u\\['name'\\]\\s*==\\s*name\\),\\s*None\\)\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望找到第一个匹配项，但找不到时会抛 StopIteration。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 return 行>。\n\n```python\ndef find_user(users, name):\n return next(u for u in users if u['name'] == name)\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "3f10be3c52149f96743fe37c0fb7327abfb5c8d4b9310088d76b04282b552f5b", "started_at": "2026-06-23T13:15:56.895116+00:00", "latency_ms": 59067, "output": "### Bug 说明\n`next()` 函数在迭代器耗尽且未传入默认值参数时，会直接抛出迭代器协议层面的`StopIteration`异常。当前实现仅向`next()`传入了做筛选的生成器，未指定无匹配时的兜底返回值：当不存在名称匹配的用户时，生成器无元素可产出，会把底层迭代异常直接暴露给上层调用，不符合查找函数的预期行为（无匹配时应返回未找到标识，而非抛出迭代相关异常）。\n最小修复利用`next()`原生的默认值能力，指定无匹配时返回`None`（Python惯例的无结果标识）；由于`next()`此时需要传入两个参数，作为第一个参数的生成器表达式需要包裹括号避免语法错误，无需额外异常捕获，改动量最小。\n\nFIX: return next((u for u in users if u['name'] == name), None)", "usage": {"prompt_tokens": 129, "completion_tokens": 2813, "total_tokens": 2942, "completion_tokens_details": {"reasoning_tokens": 2618}}, "response_id": "resp_0217822205471841542bc6dcdb74c46c1e1efc6be331707933484", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "\\*{0,2}FIX\\*{0,2}:\\*{0,2}\\s*`?\\s*return\\s+next\\(\\(u\\s+for\\s+u\\s+in\\s+users\\s+if\\s+u\\['name'\\]\\s*==\\s*name\\),\\s*None\\)\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-014", "suite": "debugging", "case_version": "0.2.0", "tags": ["python", "mutable-default", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*(?:```(?:python)?\\s*)?`?\\s*def\\s+log_request\\(req,\\s*logs=None\\)\\s*:", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望按请求顺序记录日志，但默认参数导致多次调用共用同一个列表。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的函数定义行>。\n\n```python\ndef log_request(req, logs=[]):\n logs.append(req['id'])\n return logs\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "2131fc867d62d7093265e77811a401805ff12d38e7bbdc5e8797b3d01b4c8c59", "started_at": "2026-06-23T13:16:55.966416+00:00", "latency_ms": 118604, "output": "### Bug 说明\nPython 函数的默认参数是在**函数定义阶段**完成求值初始化的，整个程序生命周期内仅创建一次，不会在每次调用时重新生成默认值。本代码使用可变类型（空列表`[]`）作为`logs`的默认参数，导致所有未显式传入`logs`的调用会共享同一个列表实例：\n1. 跨请求状态污染：不同请求的日志会持续追加到共享列表中，出现历史日志残留，默认调用时无法得到仅包含当前请求ID的独立日志；\n2. 违背设计预期：函数本应支持调用者传入自定义日志列表、按请求调用顺序追加记录，而隐式共享的默认列表会让无参调用的日志意外混在一起，破坏日志隔离性。\n\n### 最小修复\n将默认参数替换为不可变哨兵值`None`，在函数体内判断：若未传入`logs`参数（即`logs is None`），则初始化一个新的空列表，既保留传入外部列表时按顺序追加日志的能力，也避免可变默认值的共享问题。\n\nFIX: def log_request(req, logs=None):", "usage": {"prompt_tokens": 127, "completion_tokens": 5927, "total_tokens": 6054, "completion_tokens_details": {"reasoning_tokens": 5677}}, "response_id": "resp_02178222060588587baed5f4f5afe28d98e4b551391fd70ad2006", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*(?:```(?:python)?\\s*)?`?\\s*def\\s+log_request\\(req,\\s*logs=None\\)\\s*:"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-reasoning-006", "suite": "reasoning", "case_version": "0.2.0", "tags": ["symbolic", "sequence"], "scorer": "exact", "expected": {"answer": "3", "normalize": true}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只回答最终数字：一个队列从左到右是 A,B,C,D,E。依次执行：把第 2 个移到末尾；删除第 3 个；在最前面插入 X；把最后两个反转。最终队列中 C 的位置是第几位？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "0defd3c3114d287777b39b65961dfa6fd7469da02e6c49a8c170b4f1be17f971", "started_at": "2026-06-23T13:18:54.576219+00:00", "latency_ms": 155199, "output": "3", "usage": {"prompt_tokens": 119, "completion_tokens": 7921, "total_tokens": 8040, "completion_tokens_details": {"reasoning_tokens": 7920}}, "response_id": "resp_0217822207243791ff0a97b687c5fa8cd89ea5bb2fbb8cb518290", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"expected": "3", "actual": "3"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-reasoning-007", "suite": "reasoning", "case_version": "0.2.0", "tags": ["percentage", "metrics"], "scorer": "exact", "expected": {"answer": "120", "normalize": true}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只回答最终数字：某缓存命中率从 60% 提升到 75%。如果总请求数为 800，未命中请求减少了多少个？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "b5b568242a1b6c56d53db2d69addca2d25e472dcc9fb5aa8669dfa273609c0d4", "started_at": "2026-06-23T13:21:29.781661+00:00", "latency_ms": 7242, "output": "120", "usage": {"prompt_tokens": 99, "completion_tokens": 269, "total_tokens": 368, "completion_tokens_details": {"reasoning_tokens": 266}}, "response_id": "resp_02178222087994377f9606cef83fa80b3d313b2e208ca1c68d05d", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"expected": "120", "actual": "120"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-005", "suite": "rag_long_context", "case_version": "0.2.0", "tags": ["rag", "mode-comparison"], "scorer": "text_rules", "expected": {"required": ["ability mode", "efficiency mode", "真实能力", "低预算"], "forbidden": ["只看一个模式"], "max_chars": 180}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料：A 模型在 ability mode 中通过 46/50，成本 0.12 美元；在 efficiency mode 中通过 34/50，成本 0.04 美元。报告建议把 ability mode 用于能力判断，把 efficiency mode 用于成本和稳定性判断。\n\n问题：如果文章讨论“模型真实能力”，应该优先引用哪个模式？如果讨论“低预算可用性”，应该优先引用哪个模式？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "0265f7540e44005627e16c6995d2a9cb458be65bb560cd1d5646c0435265c4e3", "started_at": "2026-06-23T13:21:37.024609+00:00", "latency_ms": 20604, "output": "1. 讨论“模型真实能力”应优先引用ability mode，根据资料给出的报告建议，ability mode的定位就是用于能力判断。\n2. 讨论“低预算可用性”应优先引用efficiency mode，根据资料给出的报告建议，efficiency mode的定位是用于成本相关判断，该模式下成本仅0.04美元，成本更低，契合低预算的评估维度。", "usage": {"prompt_tokens": 167, "completion_tokens": 913, "total_tokens": 1080, "completion_tokens_details": {"reasoning_tokens": 828}}, "response_id": "resp_0217822208868591773480a8a8832a2fb78799cf0b427337d0d49", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 172, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-006", "suite": "rag_long_context", "case_version": "0.2.0", "tags": ["rag", "failure-analysis"], "scorer": "text_rules", "expected": {"required": ["key"], "required_any": [["字符串", "类型不匹配"], ["整数", "int"], ["类型假设", "假设 key 为整数", "假设 key 是整数", "key 为整数类型"]], "forbidden": [], "max_chars": 120}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料：一次失败分析显示，模型输出了 key = int(parts[1])，但测试数据里的 key 是字符串 a、b、c。报告认为这是“擅自补类型假设”，不是 token 截断。\n\n问题：这次失败的根因是什么？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "a49a29fca4961af6e5e2b25c2e82cc0a7be110d48893fc55f7af30f6e1804362", "started_at": "2026-06-23T13:21:57.630776+00:00", "latency_ms": 14479, "output": "这次失败的根因是模型擅自补类型假设：模型擅自对key的类型做出了整型的假设，输出了`key = int(parts[1])`的处理逻辑，和测试数据中key为字符串a、b、c的实际情况不匹配，该问题并非token截断导致。", "usage": {"prompt_tokens": 126, "completion_tokens": 715, "total_tokens": 841, "completion_tokens_details": {"reasoning_tokens": 650}}, "response_id": "resp_02178222090698260fde4f85d3ced80a85c9996730272b2b3a1ff", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 110, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-007", "suite": "rag_long_context", "case_version": "0.2.0", "tags": ["rag", "abstain"], "scorer": "text_rules", "expected": {"required": ["图表", "资料"], "required_any": [["提到", "提及"]], "forbidden": ["已经支持", "折线图", "柱状图"], "max_chars": 100}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答，不要自行补充。\n\n资料：DracoBench 的 HTML 报告目前展示 Summary、By Suite、Failure Examples 和 Case Details。Failure Examples 中包括 Prompt、Mistake Analysis、Scorer Details 和 Output。资料没有提到图表功能。\n\n问题：报告是否已经支持图表？回答时说明依据。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "c47ee5e02396d5ba9d3852992bf82351aeea58cd22d81035d26a6bac283f07ba", "started_at": "2026-06-23T13:22:12.112148+00:00", "latency_ms": 51390, "output": "现有资料未表明DracoBench的HTML报告已支持图表功能。\n依据：资料明确说明，DracoBench的HTML报告目前展示Summary、By Suite、Failure Examples和Case Details，其中Failure Examples包括Prompt、Mistake Analysis、Scorer Details和Output，资料没有提到图表功能。", "usage": {"prompt_tokens": 130, "completion_tokens": 2650, "total_tokens": 2780, "completion_tokens_details": {"reasoning_tokens": 2586}}, "response_id": "resp_021782220922074ebdbbc37c8ece91f45ac418a6cca0631aa9fe1", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 188, "length_ok": false, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-if-004", "suite": "instruction_following", "case_version": "0.2.0", "tags": ["json", "nested", "strict"], "scorer": "json_schema_lite", "expected": {"schema": {"type": "object", "required": ["mode", "limits", "warnings"], "additionalProperties": false, "properties": {"mode": {"type": "string", "const": "ability"}, "limits": {"type": "object", "required": ["max_tokens", "temperature"], "additionalProperties": false, "properties": {"max_tokens": {"type": "integer", "const": 4096}, "temperature": {"type": "integer", "const": 0}}}, "warnings": {"type": "array", "minItems": 2, "maxItems": 2, "items": {"type": "string"}}}}}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 JSON 对象。字段 mode 必须是 \"ability\"；字段 limits 必须是对象，包含 max_tokens:4096 和 temperature:0；字段 warnings 必须是正好两个字符串组成的数组。不要 Markdown。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "14a17039d4b99be719ac311a5d824ceb40affbf22079ddbcf8bf33af6f04fb0a", "started_at": "2026-06-23T13:23:03.507397+00:00", "latency_ms": 6108, "output": "{\n \"mode\": \"ability\",\n \"limits\": {\n \"max_tokens\": 4096,\n \"temperature\": 0\n },\n \"warnings\": [\n \"输出内容需严格遵守相关法律法规及公序良俗要求\",\n \"需严格按照指定格式输出内容，不得额外添加无关信息\"\n ]\n}", "usage": {"prompt_tokens": 110, "completion_tokens": 307, "total_tokens": 417, "completion_tokens_details": {"reasoning_tokens": 224}}, "response_id": "resp_021782220973058d66ba69fb6c1ec845eae277b4137078e47c00a", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"errors": [], "value": {"mode": "ability", "limits": {"max_tokens": 4096, "temperature": 0}, "warnings": ["输出内容需严格遵守相关法律法规及公序良俗要求", "需严格按照指定格式输出内容，不得额外添加无关信息"]}}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-if-005", "suite": "instruction_following", "case_version": "0.2.0", "tags": ["markdown", "table"], "scorer": "regex", "expected": {"pattern": "^\\s*\\|\\s*模式\\s*\\|\\s*用途\\s*\\|\\s*风险\\s*\\|\\s*\\n\\s*\\|\\s*-+\\s*\\|\\s*-+\\s*\\|\\s*-+\\s*\\|[\\s\\S]*ability mode[\\s\\S]*efficiency mode[\\s\\S]*$", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Markdown 表格，必须有 3 列：模式、用途、风险。必须包含 ability mode 和 efficiency mode 两行。不能添加表格外说明。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "0a28a65a8a714a5c613c2e01f5bf2b72b7b18cdd17059ad9d38cea38a6e1ff4f", "started_at": "2026-06-23T13:23:09.617364+00:00", "latency_ms": 17474, "output": "| 模式 | 用途 | 风险 |\n| --- | --- | --- |\n| ability mode | 释放模型全量能力上限，适配复杂逻辑推理、多步任务规划、高精度专业内容生成、高难度问题求解等对输出质量要求严苛的场景，通过全链路深度推理保障结果的准确性、完整性与创造性 | 算力资源消耗量大，响应延迟较高，高并发场景下易出现服务排队拥堵，部分场景可能因过度关联推理产生冗余信息、内容发散偏离核心需求的问题 |\n| efficiency mode | 优先保障推理响应速度与资源利用效率，适配简单事实问答、实时交互反馈、批量低复杂度文本处理、高频轻量请求等对时延敏感的场景，通过轻量化推理链路压缩计算开销、缩短返回时长 | 复杂任务场景下推理精度明显下降，易出现事实错误、逻辑断层、内容细节缺失的问题，对多轮深度推理、专业级创作等高要求任务的支撑能力不足 |", "usage": {"prompt_tokens": 96, "completion_tokens": 790, "total_tokens": 886, "completion_tokens_details": {"reasoning_tokens": 583}}, "response_id": "resp_021782220979011b74211a9fa242456af57e687c336f2688cfd55", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "^\\s*\\|\\s*模式\\s*\\|\\s*用途\\s*\\|\\s*风险\\s*\\|\\s*\\n\\s*\\|\\s*-+\\s*\\|\\s*-+\\s*\\|\\s*-+\\s*\\|[\\s\\S]*ability mode[\\s\\S]*efficiency mode[\\s\\S]*$"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-zh-writing-003", "suite": "chinese_writing", "case_version": "0.2.0", "tags": ["wechat", "explanation"], "scorer": "text_rules", "expected": {"required": ["原题", "错误解释", "可复查"], "forbidden": ["玄学"], "max_chars": 140}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "写一段不超过 140 个汉字的公众号小结，主题是“为什么要同时展示原题、模型输出和错误解释”。必须包含“原题”“错误解释”“可复查”，不能出现“玄学”。", "parameters": {"temperature": 0.2, "max_tokens": 16384, "seed": null}, "prompt_hash": "f0853803e81259a7bee0074883cc95566fe68e7e6f9ce84b4c3bda73b8c0e638", "started_at": "2026-06-23T13:23:27.094946+00:00", "latency_ms": 46160, "output": "本期小结：我们坚持同步展示原题、模型输出与错误解释，既完整呈现模型作答的全逻辑链路，也把错漏根源拆解透亮，所有内容可复查、可对照，帮你清晰摸透模型能力边界，不被片面截取的信息误导，看得明白、学得扎实。", "usage": {"prompt_tokens": 111, "completion_tokens": 2674, "total_tokens": 2785, "completion_tokens_details": {"reasoning_tokens": 2610}}, "response_id": "resp_021782220996619ccf5b8319d03f62f675ca8fce44c02e7317c39", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 100, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-reasoning-008", "suite": "reasoning", "case_version": "0.3.0", "tags": ["state-tracking", "branching"], "scorer": "exact", "expected": {"answer": "15", "normalize": true}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只回答最终数字：变量初始为 x=2, y=5, z=1。若 x list[str]，解析一行简化 CSV：逗号分隔；字段可以用双引号包裹；引号内逗号保留；引号内两个连续双引号表示一个双引号；空字段保留；空格是普通字符，不要自动 trim。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "81b6be132a0538b92491e39887550c116b7f09684b97f8dd1ad7df5426e97adb", "started_at": "2026-06-23T13:50:27.552144+00:00", "latency_ms": 318984, "output": "", "usage": {"prompt_tokens": 124, "completion_tokens": 16384, "total_tokens": 16508, "completion_tokens_details": {"reasoning_tokens": 16384}}, "response_id": "resp_02178222261739604b7fd0428efab78207837433ca5fe67217930", "finish_reason": "incomplete", "score": {"passed": false, "score": 0.0, "details": {"returncode": 1, "stdout": "", "stderr": "Traceback (most recent call last):\n File \"/var/folders/x_/4cht1kfj1_j31gdv5w01sgmw0000gn/T/dracobench-code-_v5kirzd/test_solution.py\", line 3, in \n assert solution.solve('a,\"b,c\",d') == ['a', 'b,c', 'd']\n ^^^^^^^^^^^^^^\nAttributeError: module 'solution' has no attribute 'solve'\n"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-016", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "intervals", "sorting"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve([(5,7),(1,2),(3,4),(10,10)]) == [(1,7),(10,10)]", "assert solution.solve([(1,1),(3,3)]) == [(1,1),(3,3)]", "assert solution.solve([]) == []"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(intervals: list[tuple[int, int]]) -> list[tuple[int, int]]。intervals 是整数闭区间，先按起点排序；重叠或端点相邻的区间要合并，例如 [1,2] 和 [3,4] 合并为 [1,4]。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "6dc22c1e6ae004a46c47dc11d5cc9ed5d57f09ec77549ff0ada57b459929d814", "started_at": "2026-06-23T13:55:46.599960+00:00", "latency_ms": 130241, "output": "```python\ndef solve(intervals: list[tuple[int, int]]) -> list[tuple[int, int]]:\n if not intervals:\n return []\n sorted_intervals = sorted(intervals, key=lambda x: x[0])\n merged = [sorted_intervals[0]]\n for curr_s, curr_e in sorted_intervals[1:]:\n last_s, last_e = merged[-1]\n if curr_s <= last_e + 1:\n merged[-1] = (last_s, max(last_e, curr_e))\n else:\n merged.append((curr_s, curr_e))\n return merged\n```", "usage": {"prompt_tokens": 134, "completion_tokens": 7211, "total_tokens": 7345, "completion_tokens_details": {"reasoning_tokens": 7065}}, "response_id": "resp_0217822229361231f1c2502e2c4024a5d577df8bf366bf3dd3d96", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-017", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "rate-limit", "state"], "scorer": "code_python_tests", "expected": {"tests": ["events=[('a',1),('a',2),('a',3),('a',4),('a',8)]\nassert solution.solve(events, 5, 2) == [True, True, False, False, True]", "assert solution.solve([('x',10),('x',11),('x',16)], 5, 1) == [True, False, True]"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(events: list[tuple[str, int]], window: int, limit: int) -> list[bool]，模拟按用户限流。events 按时间非降序排列。某用户在当前事件时间 ts 的窗口 (ts-window, ts] 内，已放行事件数少于 limit 才放行；被拒绝的事件不计入窗口。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "1539217453d6af4bac3c7abfbd8df43ed131cd647e58a0ad2a5a670794a91f80", "started_at": "2026-06-23T13:57:56.887929+00:00", "latency_ms": 141880, "output": "```python\nfrom collections import deque\n\ndef solve(events: list[tuple[str, int]], window: int, limit: int) -> list[bool]:\n user_records = {}\n result = []\n for user_id, ts in events:\n if user_id not in user_records:\n user_records[user_id] = deque()\n dq = user_records[user_id]\n threshold = ts - window\n while dq and dq[0] <= threshold:\n dq.popleft()\n if len(dq) < limit:\n dq.append(ts)\n result.append(True)\n else:\n result.append(False)\n return result\n```", "usage": {"prompt_tokens": 145, "completion_tokens": 6686, "total_tokens": 6831, "completion_tokens_details": {"reasoning_tokens": 6525}}, "response_id": "resp_0217822230791480a5720dfd57fd83aed3bdbca02adba84152914", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-018", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "path", "stack"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve('/a//b/./c/../') == '/a/b'", "assert solution.solve('/../../x') == '/x'", "assert solution.solve('/') == '/'"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(path: str) -> str，规范化绝对 Unix 路径：处理重复斜杠、.、..；.. 在根目录时仍留在根目录；返回结果必须以 / 开头且除根目录外不能以 / 结尾。不要使用 os.path。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "92e706f1062d7652c2f14c514c44602c6708e65d87771889d3aff7450e33e85d", "started_at": "2026-06-23T14:00:18.817243+00:00", "latency_ms": 34626, "output": "```python\ndef solve(path: str) -> str:\n stack = []\n for part in path.split('/'):\n if part == '' or part == '.':\n continue\n elif part == '..':\n if stack:\n stack.pop()\n else:\n stack.append(part)\n return '/' if not stack else '/' + '/'.join(stack)\n```", "usage": {"prompt_tokens": 126, "completion_tokens": 1669, "total_tokens": 1795, "completion_tokens_details": {"reasoning_tokens": 1582}}, "response_id": "resp_021782223208378cd9ac3abd2c47e334a510ad7678f857ee8700f", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-019", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "grouping", "stable-order"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve(['eat','tea','tan','ate','nat','bat']) == [['eat','tea','ate'], ['tan','nat'], ['bat']]", "assert solution.solve(['b','B','bb']) == [['b'], ['B'], ['bb']]"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(words: list[str]) -> list[list[str]]，按变位词分组。大小写敏感；组内保持输入顺序；组的顺序按该组第一个单词在输入中出现的位置。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "0a957bf71a0be6fa198a854347b2fe5d2c5d5f9b9a8bd5041dac9cdb46ae06ce", "started_at": "2026-06-23T14:00:53.487585+00:00", "latency_ms": 86777, "output": "```python\ndef solve(words: list[str]) -> list[list[str]]:\n groups = {}\n result = []\n for word in words:\n key = ''.join(sorted(word))\n if key not in groups:\n groups[key] = []\n result.append(groups[key])\n groups[key].append(word)\n return result\n```", "usage": {"prompt_tokens": 113, "completion_tokens": 4688, "total_tokens": 4801, "completion_tokens_details": {"reasoning_tokens": 4605}}, "response_id": "resp_021782223243631cb43b51ad7a6f2b3d703638489b11c037d854f", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-020", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "parser", "front-matter"], "scorer": "code_python_tests", "expected": {"tests": ["text='---\\ntitle: Test\\ntags: [ai, bench, v0]\\ndraft: false\\n---\\nbody'\nassert solution.solve(text) == {'title':'Test','tags':['ai','bench','v0'],'draft':'false'}", "assert solution.solve('hello\\n---\\na: b') == {}", "assert solution.solve('---\\na: 1') == {}"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(text: str) -> dict，解析 Markdown front matter。只有第一行正好是 --- 时才解析，直到下一行 --- 结束；每行 key: value，key 和 value 两侧 trim；形如 [a, b] 的 value 返回字符串列表；其他 value 返回字符串。没有完整 front matter 返回 {}。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "4111f2228a7b621a743019a4cad1b5638c8a120765153c2d35f8db2aa136f79c", "started_at": "2026-06-23T14:02:20.309664+00:00", "latency_ms": 116686, "output": "```python\ndef solve(text: str) -> dict:\n lines = text.splitlines()\n if not lines or lines[0] != '---':\n return {}\n \n end_idx = None\n for i in range(1, len(lines)):\n if lines[i] == '---':\n end_idx = i\n break\n if end_idx is None:\n return {}\n \n result = {}\n for line in lines[1:end_idx]:\n if ':' not in line:\n continue\n key_part, value_part = line.split(':', 1)\n key = key_part.strip()\n value_str = value_part.strip()\n \n if value_str.startswith('[') and value_str.endswith(']'):\n inner = value_str[1:-1].strip()\n value = [item.strip() for item in inner.split(',')] if inner else []\n else:\n value = value_str\n \n result[key] = value\n return result\n```", "usage": {"prompt_tokens": 138, "completion_tokens": 5816, "total_tokens": 5954, "completion_tokens_details": {"reasoning_tokens": 5593}}, "response_id": "resp_021782223331981edfe17761d7699bd2e54408390fbabd188f32e", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-021", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "frequency", "stable-order"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve(['b','a','b','c','a','b'], 2) == ['b','a']", "assert solution.solve(['x','y','z','y','z','x'], 2) == ['x','y']", "assert solution.solve(['a'], 5) == ['a']"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(items: list[str], k: int) -> list[str]，返回出现次数最高的前 k 个元素；次数相同按第一次出现的先后排序；k 大于不同元素数量时返回全部。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "e73dccf0e991c296063c917fbe5a6b5f99f50fb3baed6d0820cfc35df858c663", "started_at": "2026-06-23T14:04:17.041854+00:00", "latency_ms": 48612, "output": "```python\ndef solve(items: list[str], k: int) -> list[str]:\n counts = {}\n first_occur = {}\n for idx, item in enumerate(items):\n if item not in counts:\n counts[item] = 0\n first_occur[item] = idx\n counts[item] += 1\n sorted_unique = sorted(counts.keys(), key=lambda x: (-counts[x], first_occur[x]))\n return sorted_unique[:k]\n```", "usage": {"prompt_tokens": 115, "completion_tokens": 2433, "total_tokens": 2548, "completion_tokens_details": {"reasoning_tokens": 2319}}, "response_id": "resp_0217822234508521bf7e6c6c13a047df19a5322aa31ba20c533f7", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-022", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "nested-dict", "copy"], "scorer": "code_python_tests", "expected": {"tests": ["base={'db':{'host':'old','port':5432},'feature':{'x':1}}\noverrides={'db.host':'new','feature.x':None,'api.timeout':30}\nres=solution.solve(base, overrides)\nassert res == {'db':{'host':'new','port':5432},'feature':{},'api':{'timeout':30}}\nassert base == {'db':{'host':'old','port':5432},'feature':{'x':1}}", "assert solution.solve({'a':1}, {'a.b':2}) == {'a': {'b': 2}}"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(config: dict, overrides: dict[str, object]) -> dict。overrides 的 key 是点号路径；值为 None 表示删除该路径的叶子 key；其他值写入路径，必要时创建中间 dict；不能修改输入 config。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "e21e5bbf42159ba49144d39d17824dcf011df026efab7a9bfb6179aed2bb558a", "started_at": "2026-06-23T14:05:05.693562+00:00", "latency_ms": 142557, "output": "```python\nimport copy\n\n\ndef solve(config: dict, overrides: dict[str, object]) -> dict:\n result = copy.deepcopy(config)\n for dot_path, value in overrides.items():\n parts = dot_path.split('.')\n if value is None:\n current = result\n valid = True\n for part in parts[:-1]:\n if not isinstance(current, dict) or part not in current:\n valid = False\n break\n current = current[part]\n if valid and isinstance(current, dict) and parts[-1] in current:\n del current[parts[-1]]\n else:\n current = result\n for part in parts[:-1]:\n if part not in current or not isinstance(current[part], dict):\n current[part] = {}\n current = current[part]\n current[parts[-1]] = value\n return result\n```", "usage": {"prompt_tokens": 124, "completion_tokens": 6937, "total_tokens": 7061, "completion_tokens_details": {"reasoning_tokens": 6732}}, "response_id": "resp_0217822234958411fabb13063b3e354601db29cbe6491e1bcb7a7", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-023", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "url", "parser"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve('a=1&b=two+words&a=3') == {'a':['1','3'], 'b':['two words']} ", "assert solution.solve('empty=&flag&x=a%2Bb') == {'empty':[''], 'flag':[''], 'x':['a+b']} ", "assert solution.solve('') == {}"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(query: str) -> dict[str, list[str]]，解析 URL query string。& 分隔参数；没有 = 的参数值为空字符串；+ 表示空格；%XX 需要按 UTF-8 百分号解码；同名 key 保留所有值的列表。可以使用标准库。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "cfe8be23444d346039a0740ab1568f54ed98bd44e5d63909779673da2207f393", "started_at": "2026-06-23T14:07:28.298519+00:00", "latency_ms": 271907, "output": "```python\nfrom urllib.parse import unquote_plus\n\ndef solve(query: str) -> dict[str, list[str]]:\n result = {}\n if not query:\n return result\n for part in query.split('&'):\n kv_parts = part.split('=', 1)\n key_raw = kv_parts[0]\n value_raw = kv_parts[1] if len(kv_parts) == 2 else ''\n key = unquote_plus(key_raw, encoding='utf-8')\n value = unquote_plus(value_raw, encoding='utf-8')\n if key not in result:\n result[key] = []\n result[key].append(value)\n return result\n```", "usage": {"prompt_tokens": 132, "completion_tokens": 14128, "total_tokens": 14260, "completion_tokens_details": {"reasoning_tokens": 13965}}, "response_id": "resp_0217822236390651b8c9f69a83502ac57f9e97e4a3cfb74c5983f", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-024", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "parser", "stack"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve('

x

') is True", "assert solution.solve('') is False", "assert solution.solve('text') is True"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(html: str) -> bool，校验简化 HTML 标签是否配对。标签形如 、、
，标签名只含小写字母；自闭合标签不入栈；普通文本忽略。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "088a9867c7f154cf1ee172830e13e8e49c53a8e572707d776f0cdea59f3555cb", "started_at": "2026-06-23T14:12:00.270847+00:00", "latency_ms": 75777, "output": "```python\nimport re\n\ndef solve(html: str) -> bool:\n stack = []\n tags = re.findall(r'<[^>]+>', html)\n for tag in tags:\n if tag.endswith('/>'):\n continue\n if tag.startswith(' int。grid 中 S 是起点，E 是终点，# 是墙，. 可走。每步上下左右移动一格，返回从 S 到 E 的最短步数；不可达返回 -1。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "8c6dc47fe32c2eef1e05bed2e6bd50520bb606e6965fe6c18e354a50a1e65085", "started_at": "2026-06-23T14:13:16.113904+00:00", "latency_ms": 53228, "output": "```python\nfrom collections import deque\n\ndef solve(grid: list[str]) -> int:\n rows = len(grid)\n if rows == 0:\n return -1\n cols = len(grid[0])\n start = None\n for i in range(rows):\n for j in range(cols):\n if grid[i][j] == 'S':\n start = (i, j)\n break\n if start:\n break\n if not start:\n return -1\n dirs = [(-1, 0), (1, 0), (0, -1), (0, 1)]\n q = deque([(start[0], start[1], 0)])\n visited = [[False] * cols for _ in range(rows)]\n visited[start[0]][start[1]] = True\n while q:\n x, y, dist = q.popleft()\n for dx, dy in dirs:\n nx, ny = x + dx, y + dy\n if 0 <= nx < rows and 0 <= ny < cols:\n if grid[nx][ny] == 'E':\n return dist + 1\n if grid[nx][ny] == '.' and not visited[nx][ny]:\n visited[nx][ny] = True\n q.append((nx, ny, dist + 1))\n return -1\n```", "usage": {"prompt_tokens": 125, "completion_tokens": 2654, "total_tokens": 2779, "completion_tokens_details": {"reasoning_tokens": 2332}}, "response_id": "resp_021782223986001d2117f1954c918d1636ca8e94ee30b50fb3fe4", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-026", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "sliding-window", "median"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve([1,3,-1,-3,5,3,6,7], 3) == [1, -1, -1, 3, 5, 6]", "assert solution.solve([1,2,3,4], 2) == [1.5, 2.5, 3.5]"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(nums: list[int], k: int) -> list[float]，返回每个长度为 k 的滑动窗口中位数。k 为奇数时返回中间值；k 为偶数时返回两个中间值平均数。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "53547e8e9f25c840b8a6248c7be329b1779ddc5a47484208f5c434403bd8228c", "started_at": "2026-06-23T14:14:09.384087+00:00", "latency_ms": 83534, "output": "```python\nimport bisect\n\ndef solve(nums: list[int], k: int) -> list[float]:\n if not nums or k <= 0 or len(nums) < k:\n return []\n window = sorted(nums[:k])\n res = []\n mid = k // 2\n if k % 2 == 1:\n res.append(float(window[mid]))\n else:\n res.append((window[mid - 1] + window[mid]) / 2.0)\n for i in range(k, len(nums)):\n left_val = nums[i - k]\n del_idx = bisect.bisect_left(window, left_val)\n window.pop(del_idx)\n bisect.insort(window, nums[i])\n if k % 2 == 1:\n res.append(float(window[mid]))\n else:\n res.append((window[mid - 1] + window[mid]) / 2.0)\n return res\n```", "usage": {"prompt_tokens": 117, "completion_tokens": 4476, "total_tokens": 4593, "completion_tokens_details": {"reasoning_tokens": 4249}}, "response_id": "resp_0217822240406191b28b806a7bc9927371d1855db3264b9f49aae", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-027", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "sorting", "semver"], "scorer": "code_python_tests", "expected": {"tests": ["assert solution.solve(['1.0.0','1.0.0-alpha','1.0.1','1.0.0-beta','0.9.9']) == ['0.9.9','1.0.0-alpha','1.0.0-beta','1.0.0','1.0.1']", "assert solution.solve(['1.0.10','1.0.2']) == ['1.0.2','1.0.10']"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(versions: list[str]) -> list[str]，按简化语义版本排序。版本为 major.minor.patch 或 major.minor.patch-label；major/minor/patch 按数字比较；同一数字版本下，带 label 的预发布版本低于正式版本；label 按字典序比较。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "f9b7324425840beb574b30433a6e004d34a834ef548c9cfa579b9b9b6950377d", "started_at": "2026-06-23T14:15:32.976957+00:00", "latency_ms": 98556, "output": "```python\ndef solve(versions: list[str]) -> list[str]:\n def version_key(v: str) -> tuple:\n core_part, *label_parts = v.split(\"-\", 1)\n major, minor, patch = map(int, core_part.split(\".\"))\n if label_parts:\n return (major, minor, patch, 0, label_parts[0])\n return (major, minor, patch, 1, \"\")\n return sorted(versions, key=version_key)\n```", "usage": {"prompt_tokens": 134, "completion_tokens": 5095, "total_tokens": 5229, "completion_tokens_details": {"reasoning_tokens": 4980}}, "response_id": "resp_02178222412287279e9bf0afca82c9100a3d2e8164ce29caab9d0", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-coding-028", "suite": "coding", "case_version": "0.3.0", "tags": ["python", "graph", "reverse-deps"], "scorer": "code_python_tests", "expected": {"tests": ["deps={'app':['lib','ui'],'ui':['core'],'lib':['core'],'core':[],'docs':[]}\nassert solution.solve(deps, ['core']) == ['app','core','lib','ui']", "assert solution.solve({'a':['x']}, ['x']) == ['a','x']", "assert solution.solve({'a':['b'], 'b':[]}, ['a']) == ['a']"], "timeout_seconds": 3}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 Python 代码，不要解释。实现 solve(deps: dict[str, list[str]], changed: list[str]) -> list[str]。deps 表示 package -> 直接依赖。返回所有 changed 包，以及直接或间接依赖 changed 包的包，按字典序排序；只出现在依赖列表中的包也要能作为 changed 包返回。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "2a0e51751556bc0e35f2a2a2aa4ed86810eacfc274bb9c5f48e8990611a9a473", "started_at": "2026-06-23T14:17:11.577202+00:00", "latency_ms": 161586, "output": "```python\nfrom collections import defaultdict, deque\ndef solve(deps: dict[str, list[str]], changed: list[str]) -> list[str]:\n reverse_deps = defaultdict(list)\n for pkg, dependencies in deps.items():\n for dep in dependencies:\n reverse_deps[dep].append(pkg)\n visited = set(changed)\n queue = deque(changed)\n while queue:\n current = queue.popleft()\n for dependent in reverse_deps.get(current, []):\n if dependent not in visited:\n visited.add(dependent)\n queue.append(dependent)\n return sorted(visited)\n```", "usage": {"prompt_tokens": 137, "completion_tokens": 8373, "total_tokens": 8510, "completion_tokens_details": {"reasoning_tokens": 8226}}, "response_id": "resp_021782224224662f17176ac9df90535d88cbb24b1f2709b2149e6", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"returncode": 0, "stdout": "", "stderr": ""}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-008", "suite": "rag_long_context", "case_version": "0.3.0", "tags": ["rag", "conflict", "workflow"], "scorer": "text_rules", "expected": {"required": ["50", "审题"], "required_any": [["不跑", "不要立刻", "先不跑", "不应立刻", "不应该立刻", "不应立即", "不应该立即", "不应立刻跑", "不应该立刻跑", "不应立即跑", "不应该立即跑"]], "forbidden": [], "max_chars": 160}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料 A：早期草案说 v0.3 只新增 20 道题。\n资料 B：2026-06-16 的确认方案是 v0.3-challenge100，在现有 50 题基础上新增 50 道题。\n资料 C：新增题应先生成审题 HTML，确认后再跑模型，不要立即消耗 API。\n\n问题：下一步应该新增多少题？是否应该立刻跑模型？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "c99da78812f6f4ab7745f793b9e3dc087033b0cce0591db49bf8642829324a81", "started_at": "2026-06-23T14:19:53.215277+00:00", "latency_ms": 41189, "output": "下一步应新增50道题：根据2026-06-16确认的v0.3-challenge100方案，需在现有50题基础上新增50道题，早期草案提及的新增20道题为早期未最终确定的内容。\n不应该立刻跑模型：新增题应先生成审题HTML，确认后再跑模型，不要立即消耗API。", "usage": {"prompt_tokens": 169, "completion_tokens": 1831, "total_tokens": 2000, "completion_tokens_details": {"reasoning_tokens": 1743}}, "response_id": "resp_021782224390119ca53c98b4a9847edf4b59a94723c611ee75631", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 130, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-009", "suite": "rag_long_context", "case_version": "0.3.0", "tags": ["rag", "abstain"], "scorer": "text_rules", "expected": {"required": ["不能", "用户满意度"], "required_any": [["没有包含", "没有提供", "资料没有", "未包含"]], "forbidden": ["可以判断"], "max_chars": 120}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答，不要自行补充。\n\n资料：报告展示 pass_rate、avg_latency_ms、total_cost、reasoning_tokens、失败样例和逐题明细。资料没有包含用户满意度调查，也没有包含模型训练数据来源。\n\n问题：能否根据这份报告判断用户满意度？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "c072287ee5dbe1d477ab21e37485fb72f07cd11a57b57f199f071287443f404e", "started_at": "2026-06-23T14:20:34.406832+00:00", "latency_ms": 6564, "output": "不能。该报告的资料没有包含用户满意度调查相关内容，无法根据这份报告判断用户满意度。", "usage": {"prompt_tokens": 124, "completion_tokens": 156, "total_tokens": 280, "completion_tokens_details": {"reasoning_tokens": 134}}, "response_id": "resp_021782224424021852c9ca059d0686f905b05cb05880f50751818", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 41, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-010", "suite": "rag_long_context", "case_version": "0.3.0", "tags": ["rag", "tradeoff"], "scorer": "text_rules", "expected": {"required": ["B", "延迟"], "required_any": [["没有超时", "无超时"]], "forbidden": ["选择 A", "选择 C"], "max_chars": 160}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料：模型 A 通过 48/50，平均延迟 20 秒，成本 0.20 美元，没有超时失败。模型 B 通过 44/50，平均延迟 4 秒，成本 0.05 美元，没有超时失败。模型 C 通过 46/50，平均延迟 9 秒，成本 0.04 美元，但有 3 次 API 超时。\n\n问题：如果要选低延迟且稳定的生产摘要模型，资料更支持选择哪个？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "a17bb8c56d524481eedc45557a1394b5b65b034af6847928cca16e2c8c2428c9", "started_at": "2026-06-23T14:20:40.972512+00:00", "latency_ms": 74000, "output": "资料更支持选择模型B，原因如下：\n1. 稳定性维度：模型A、B均无超时失败，模型C存在3次API超时，属于服务调用层面的稳定性故障，首先排除C。\n2. 低延迟维度：模型B平均延迟仅4秒，远低于模型C的9秒、模型A的20秒，是三个模型中延迟最低的，且无超时问题，服务运行稳定，符合低延迟且稳定的选型要求。", "usage": {"prompt_tokens": 185, "completion_tokens": 3769, "total_tokens": 3954, "completion_tokens_details": {"reasoning_tokens": 3664}}, "response_id": "resp_02178222443071115aaa5b88a002f341961ec0a179b9cfa6430af", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 153, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-011", "suite": "rag_long_context", "case_version": "0.3.0", "tags": ["rag", "scoring-policy"], "scorer": "text_rules", "expected": {"required": ["诊断"], "required_any": [["不扣分", "不会", "不会因长度", "不会因此被扣分", "不会直接导致扣分", "不会导致扣分", "不会导致失败或扣分"], ["length_scored", "长度"]], "forbidden": ["直接失败", "必须重跑"], "max_chars": 140}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料：DracoBench 当前 scoring policy 中，text_rules 的 max_chars/min_chars 只作为诊断信息记录，length_scored 为 false。缺少 required、required_any 或出现 forbidden 才会导致失败。\n\n问题：如果答案超过 max_chars，但 required 都满足且没有 forbidden，会因为长度直接扣分吗？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "1ff7f897fc3d3121f25358f9a8df2a2df2970bc5762ff1218dd63c3f42d7d598", "started_at": "2026-06-23T14:21:54.976355+00:00", "latency_ms": 8973, "output": "不会。根据资料，DracoBench当前scoring policy中，text_rules的max_chars/min_chars只作为诊断信息记录，length_scored为false，仅缺少required、required_any或出现forbidden才会导致失败，因此该情形不会因为长度直接扣分。", "usage": {"prompt_tokens": 141, "completion_tokens": 401, "total_tokens": 542, "completion_tokens_details": {"reasoning_tokens": 341}}, "response_id": "resp_0217822245044704d1a7bd4cdfcf1811296cf68db8d43d814973c", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 155, "length_ok": false, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-012", "suite": "rag_long_context", "case_version": "0.3.0", "tags": ["rag", "rescoring"], "scorer": "text_rules", "expected": {"required": ["46/50"], "forbidden": ["45/50 更合适", "原始结果为准"], "max_chars": 140}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料 A：某模型原始结果为 45/50。\n资料 B：修正一个同义表达评分规则后，rescored 结果为 46/50。\n资料 C：总览 index 使用 rescored JSONL 作为展示结果。\n\n问题：写文章引用总览结果时，应写 45/50 还是 46/50？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "274c0483d9fd5cae178e795e16add941178c14f5ae34a82f631655c000ff929a", "started_at": "2026-06-23T14:22:03.950498+00:00", "latency_ms": 13153, "output": "应写46/50。\n根据资料说明，总览index使用rescored JSONL作为展示结果，而修正评分规则后的rescored结果为46/50，因此引用总览结果时应使用46/50。", "usage": {"prompt_tokens": 150, "completion_tokens": 343, "total_tokens": 493, "completion_tokens_details": {"reasoning_tokens": 289}}, "response_id": "resp_021782224517238571e879a7e0eb1247cfcaaab487786a1948e85", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 90, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-013", "suite": "rag_long_context", "case_version": "0.3.0", "tags": ["rag", "reproducibility"], "scorer": "text_rules", "expected": {"required": ["model", "prompt_hash", "parameters", "scorer"], "forbidden": ["chain of thought", "训练数据"], "max_chars": 200}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料：为了复现一次 benchmark，记录中包含 model、case_id、prompt_hash、parameters、scorer details、output、latency_ms 和 usage。记录不包含 chain of thought，也不包含训练数据。\n\n问题：复现评测主要应该依赖哪些记录字段？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "7455c8e21abbb53f7cb11c9db29ed66ad1dfcc7e60138f6dfccb7cbcacc958c3", "started_at": "2026-06-23T14:22:17.105353+00:00", "latency_ms": 71951, "output": "复现评测主要依赖记录中为复现benchmark留存的以下字段：model、case_id、prompt_hash、parameters、scorer details、output、latency_ms、usage。\n记录中不包含的chain of thought（思维链）、训练数据不属于可依赖的复现记录字段。", "usage": {"prompt_tokens": 126, "completion_tokens": 3665, "total_tokens": 3791, "completion_tokens_details": {"reasoning_tokens": 3601}}, "response_id": "resp_02178222452755290b54e8a6ac9d8f31999a1fe47d3c2b0117c28", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"manual_review_override": true, "original_score": {"passed": false, "score": 0.0, "details": {"missing": [], "missing_any": [], "present_forbidden": ["chain of thought", "训练数据"], "char_count": 155, "length_ok": true, "length_scored": false}}, "reason": "人工复核通过：模型列出了 model、case_id/prompt_hash、parameters、scorer details、output、latency_ms、usage 等复现字段，并明确说明 chain of thought 和训练数据不属于可依赖字段；原 forbidden 规则没有处理否定语境。"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl", "original_score": {"passed": false, "score": 0.0, "details": {"missing": [], "missing_any": [], "present_forbidden": ["chain of thought", "训练数据"], "char_count": 155, "length_ok": true, "length_scored": false}}} {"case_id": "challenge-rag-014", "suite": "rag_long_context", "case_version": "0.3.0", "tags": ["rag", "mode"], "scorer": "text_rules", "expected": {"required": ["16384"], "forbidden": ["4096"], "max_chars": 100}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料 A：早期 ability mode 使用 4096 max_tokens。\n资料 B：后来为了公平，ability mode 统一放宽到 16384 max_tokens。\n资料 C：当前总览页只展示 v0.2-challenge50 且 ability16384 的结果。\n\n问题：当前总览页对应的 token limit 是多少？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "1d5b45892f4d0509e7805976e9bb60b3893e68657b40fcfa934db89a27eccc7b", "started_at": "2026-06-23T14:23:29.059713+00:00", "latency_ms": 6906, "output": "当前总览页对应的token limit是16384 max_tokens，依据是当前总览页只展示ability16384的结果，而该设置下ability mode的max_tokens为16384。", "usage": {"prompt_tokens": 148, "completion_tokens": 290, "total_tokens": 438, "completion_tokens_details": {"reasoning_tokens": 241}}, "response_id": "resp_0217822245992101fabb13063b3e354601db29cbe6491e157e7eb", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 99, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-015", "suite": "rag_long_context", "case_version": "0.3.0", "tags": ["rag", "failure-analysis", "scoring-policy"], "scorer": "text_rules", "expected": {"required": ["人工复查", "coding-004"], "required_any": [["rag-006", "规则", "评分规则", "同义表达"], ["不属于", "不是", "不需要"]], "max_chars": 180}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料：coding-004 的失败是代码运行时 ValueError，根因是把字符串 key 强转 int。rag-006 的失败曾经是规则没有把 int 类型视作“整数”的同义表达，后来通过人工复查修正规则。\n\n问题：哪类失败更需要人工复查评分规则？回答时也说明 coding-004 是否属于这种情况。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "d4725ef21042620eab8231d5671ac8f778c2b71f85afb72d592c3da2278cc9c4", "started_at": "2026-06-23T14:23:35.967547+00:00", "latency_ms": 24734, "output": "由评分规则本身不完善（如规则未将int类型视作“整数”的同义表达）引发的失败更需要人工复查评分规则，rag-006就属于这类需要人工复查修正规则的情况。\ncoding-004不属于这种情况：coding-004的失败是代码运行时触发ValueError，根因是代码编写中将字符串key强转为int，属于代码自身的运行错误，并非评分规则存在问题导致的失败。", "usage": {"prompt_tokens": 149, "completion_tokens": 1159, "total_tokens": 1308, "completion_tokens_details": {"reasoning_tokens": 1060}}, "response_id": "resp_0217822246063031f1c2502e2c4024a5d577df8bf366bf3b8068d", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 177, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-016", "suite": "rag_long_context", "case_version": "0.3.0", "tags": ["rag", "abstain", "cost"], "scorer": "text_rules", "expected": {"required": ["不能", "利润率"], "required_any": [["不包含", "没有"]], "forbidden": ["可以判断"], "max_chars": 120}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料：usage 字段包含 prompt_tokens、completion_tokens、reasoning_tokens 和 cost。报告没有供应商真实利润率，也没有供应商内部成本。\n\n问题：能否从 usage 字段判断供应商利润率？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "622c02bde50c8b232afe7bb574da76e2ed1e35d49a55a96f63b935dbbbb16175", "started_at": "2026-06-23T14:24:00.703372+00:00", "latency_ms": 6743, "output": "不能。usage字段仅包含prompt_tokens、completion_tokens、reasoning_tokens和cost相关内容，不具备判断供应商利润率所需的供应商真实利润率、供应商内部成本相关信息，无法据此判断供应商利润率。", "usage": {"prompt_tokens": 109, "completion_tokens": 246, "total_tokens": 355, "completion_tokens_details": {"reasoning_tokens": 202}}, "response_id": "resp_02178222463053906836341896f8585700baa60fd8566b55ede33", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"manual_review_override": true, "original_score": {"passed": false, "score": 0.0, "details": {"missing": [], "missing_any": [["不包含", "没有"]], "present_forbidden": [], "char_count": 119, "length_ok": true, "length_scored": false}}, "reason": "人工复核通过：模型明确回答不能从 usage 判断供应商利润率，并说明缺少真实利润率和内部成本；原 required_any 只接受“不包含/没有”的字面表达，未覆盖“不具备/无法据此判断”的等价表达。"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl", "original_score": {"passed": false, "score": 0.0, "details": {"missing": [], "missing_any": [["不包含", "没有"]], "present_forbidden": [], "char_count": 119, "length_ok": true, "length_scored": false}}} {"case_id": "challenge-rag-017", "suite": "rag_long_context", "case_version": "0.3.0", "tags": ["rag", "index-scope"], "scorer": "text_rules", "expected": {"required": ["v0.2-challenge50", "ability16384"], "required_any": [["不应该", "不展示", "未展示", "不显示", "均不显示", "不应展示"], ["旧版本", "之前的版本"]], "max_chars": 160}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料：当前 index 页只显示最后这版 50 道题的 v0.2-challenge50 结果，并且 token limit 都统一为 ability16384。之前的版本和 ability4096 结果不显示在 index 页。\n\n问题：index 页是否应该展示旧版本结果？当前展示范围是什么？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "84234aeba9875f705a8daae80d0f0a287d1cef74de57877dfe6fbd41e6d1e835", "started_at": "2026-06-23T14:24:07.447957+00:00", "latency_ms": 21011, "output": "根据资料明确的显示规则：\n1. index页不展示旧版本结果，之前的版本结果不在index页的显示范围内。\n2. 当前index页的展示范围为：仅显示最后这版50道题的v0.2-challenge50结果，所展示结果的token limit统一为ability16384；之前的版本结果、ability4096结果均不显示在index页。", "usage": {"prompt_tokens": 145, "completion_tokens": 988, "total_tokens": 1133, "completion_tokens_details": {"reasoning_tokens": 895}}, "response_id": "resp_0217822246372763e1d45c2e42a445361f900407e7e03eac05e10", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 169, "length_ok": false, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-rag-018", "suite": "rag_long_context", "case_version": "0.3.0", "tags": ["rag", "failure-analysis", "truncation"], "scorer": "text_rules", "expected": {"required": ["输出"], "required_any": [["截断", "length", "token"]], "forbidden": [], "max_chars": 140}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只根据资料回答。\n\n资料：某 coding 题中，模型响应的 finish_reason 为 length，output 为空，测试报错为 module has no attribute solve。报告认为这不是算法逻辑错误，而是输出被截断后没有形成可执行函数。\n\n问题：这次失败应如何归因？", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "f819b3ec516c1c77e6e0c1bbcece217d77fc5a4aaee6e7787be0b4cd5d97c9ff", "started_at": "2026-06-23T14:24:28.460793+00:00", "latency_ms": 8392, "output": "这次失败不属于算法逻辑错误，归因如下：模型响应的finish_reason为length，即输出因长度限制被截断，最终输出为空，没能形成可执行的solve函数，进而触发了\"module has no attribute solve\"的测试报错。", "usage": {"prompt_tokens": 125, "completion_tokens": 315, "total_tokens": 440, "completion_tokens_details": {"reasoning_tokens": 262}}, "response_id": "resp_021782224658348eaad90e368bae3f187447d9def39b907931c30", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 122, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-015", "suite": "debugging", "case_version": "0.3.0", "tags": ["python", "closure", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*funcs\\.append\\(lambda\\s+i=i\\s*:\\s*i\\)\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望 funcs[0]()、funcs[1]()、funcs[2]() 分别返回 0、1、2，但实际都会返回 2。请指出根因，并给出最小修复。最后单独一行写 FIX: <修复后的 append 行>。\n\n```python\nfuncs = []\nfor i in range(3):\n funcs.append(lambda: i)\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "e90d1ddfafd6f041b7be435f823b34b7ad74ddf85bb4dbb228db8e1183cd8843", "started_at": "2026-06-23T14:24:36.854222+00:00", "latency_ms": 27769, "output": "### 根因\nPython闭包采用**晚绑定（延迟绑定）**机制：lambda表达式中引用的自由变量`i`不会在函数定义时捕获循环迭代的瞬时值，而是在函数实际调用时才向外层作用域查找`i`的当前值。循环结束后外层作用域的`i`已固定为终值2，因此三个lambda调用时均获取到i=2，不符合预期。\n### 最小修复逻辑\n利用Python函数默认参数在**函数定义时立即求值**的特性，将每次循环的`i`瞬时值作为默认参数绑定到对应lambda上，让每个函数实例独立持有定义时刻的i值，避免共享循环终值。\n\nFIX: funcs.append(lambda i=i: i)", "usage": {"prompt_tokens": 146, "completion_tokens": 1218, "total_tokens": 1364, "completion_tokens_details": {"reasoning_tokens": 1055}}, "response_id": "resp_021782224668570d64c75fbaa5981adc97406384c968765fd0d52", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*funcs\\.append\\(lambda\\s+i=i\\s*:\\s*i\\)\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-016", "suite": "debugging", "case_version": "0.3.0", "tags": ["python", "context-manager", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*return(?:\\s+False|\\s+None)?\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面 context manager 期望异常继续向外抛出，但实际异常被吞掉。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 return 行>。\n\n```python\nclass Span:\n def __enter__(self):\n return self\n def __exit__(self, exc_type, exc, tb):\n return True\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "e7cbf0d9b2df76954e4b3aacf7d21f0a22c40f64ccc2432b115896cdb5a3f123", "started_at": "2026-06-23T14:25:04.625412+00:00", "latency_ms": 52774, "output": "### Bug 说明\nPython 上下文管理器协议规定：`__exit__` 方法返回**真值（truthy）**时，解释器会判定 with 块抛出的异常已被处理，会抑制异常向外传播；只有返回**假值（falsy，如`False`/`None`）**时，异常才会正常向外抛出。当前实现固定返回`True`，会吞掉 with 块内所有异常，不符合异常继续抛出的预期。\n### 最小修复\n仅需将返回值从`True`改为`False`即可：无异常场景下`__exit__`的返回值会被解释器忽略，无副作用；有异常时返回假值，异常会正常向外传播。\n\nFIX: return False", "usage": {"prompt_tokens": 138, "completion_tokens": 2742, "total_tokens": 2880, "completion_tokens_details": {"reasoning_tokens": 2579}}, "response_id": "resp_02178222469601256f99a772ec79dd45125e5081732d611a31eaa", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*return(?:\\s+False|\\s+None)?\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-017", "suite": "debugging", "case_version": "0.3.0", "tags": ["python", "class-variable", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*self\\.items\\s*=\\s*\\[\\]\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望每个 Bag 实例都有独立列表，但不同实例会共享 items。请指出 bug，并给出最小修复。最后单独一行写 FIX: <新增到 __init__ 中的赋值行>。\n\n```python\nclass Bag:\n items = []\n def add(self, x):\n self.items.append(x)\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "c2b320a4b54c57505deb9a6f14ebb63217e6c351a61dbfeed24131e848fe7553", "started_at": "2026-06-23T14:25:57.404068+00:00", "latency_ms": 61272, "output": "### Bug 说明\n原代码中`items`是定义在类作用域的**类属性**，归属于`Bag`类本身，会被所有`Bag`实例共享。当通过`self.items`访问列表时，若实例自身没有同名属性，就会直接引用这个类级别的共享列表，导致所有实例调用`add`方法时都修改同一个列表对象，不同实例的items互相干扰，无法实现每个实例持有独立列表的预期。\n### 最小修复\n为`Bag`添加`__init__`实例构造方法，在每次创建实例时，为当前实例绑定专属的独立空列表作为实例属性`items`，实例属性会优先于同名类属性被访问，从而实现实例间items隔离。（可删除类层面冗余的`items = []`以清理无用代码，不影响功能正确性）\nFIX: self.items = []", "usage": {"prompt_tokens": 133, "completion_tokens": 3141, "total_tokens": 3274, "completion_tokens_details": {"reasoning_tokens": 2952}}, "response_id": "resp_021782224749302fc7bcbb7d43fe183dbed5bfff3b26ba2ab02e0", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*self\\.items\\s*=\\s*\\[\\]\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-018", "suite": "debugging", "case_version": "0.3.0", "tags": ["python", "default-argument", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*def\\s+mark\\(ts=None\\)\\s*:", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望每次不传 ts 时使用当前时间，但默认时间只会在函数定义时计算一次。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的函数定义行>。\n\n```python\nfrom datetime import datetime\n\ndef mark(ts=datetime.utcnow()):\n return ts.isoformat()\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "96674f1c8091e56110c21b7b057178e05bd5225812543e3101e86fc6c5d2dba7", "started_at": "2026-06-23T14:26:58.680802+00:00", "latency_ms": 74267, "output": "### Bug 说明\nPython 函数默认参数采用**定义时求值**机制：默认参数值会在函数定义（通常是模块加载阶段）时仅计算一次，之后绑定到函数对象的`__defaults__`属性中，后续调用若不传对应参数会直接复用这个提前计算好的固定值，不会在每次调用时重新求值。\n本代码中`ts=datetime.utcnow()`仅会在`mark`函数定义时执行一次，得到一个固定的时间对象，后续所有不传`ts`的调用都会永久复用这个定义时刻的时间，完全无法实现“不传参时使用调用时刻当前时间”的预期。\n\n### 最小修复\n使用`None`作为默认参数的哨兵值，在函数内部增加判断：当`ts`为`None`（即调用方未传参）时，实时调用`datetime.utcnow()`获取当前UTC时间，原有返回逻辑保持不变即可。\n\nFIX: def mark(ts=None):", "usage": {"prompt_tokens": 134, "completion_tokens": 3713, "total_tokens": 3847, "completion_tokens_details": {"reasoning_tokens": 3503}}, "response_id": "resp_021782224808508c1c101af9eebb417c253ad03e8acb9419180a5", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*def\\s+mark\\(ts=None\\)\\s*:"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-debugging-019", "suite": "debugging", "case_version": "0.3.0", "tags": ["python", "method-call", "debugging"], "scorer": "regex", "expected": {"pattern": "FIX:\\s*`?\\s*return\\s+s\\.strip\\(\\)\\.lower\\(\\)\\s*`?", "flags": "is"}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "下面代码期望返回小写字符串，但实际返回的是方法对象。请指出 bug，并给出最小修复。最后单独一行写 FIX: <修复后的 return 行>。\n\n```python\ndef normalize(s):\n return s.strip().lower\n```", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "e9d483dafd7852b396cee6b27c6c25c888983460dba83c871f7f9a239dd601ef", "started_at": "2026-06-23T14:28:12.951239+00:00", "latency_ms": 12784, "output": "### Bug 说明\n`lower` 是 Python 字符串的内置实例方法，代码中仅引用了该方法对象本身，没有添加`()`执行方法调用，因此最终返回的是绑定方法对象，而非调用方法后得到的小写字符串结果。\n\nFIX: return s.strip().lower()", "usage": {"prompt_tokens": 113, "completion_tokens": 383, "total_tokens": 496, "completion_tokens_details": {"reasoning_tokens": 319}}, "response_id": "resp_0217822248860384e907dae862d8462a9d84f058f87472e537dcc", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"pattern": "FIX:\\s*`?\\s*return\\s+s\\.strip\\(\\)\\.lower\\(\\)\\s*`?"}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-if-006", "suite": "instruction_following", "case_version": "0.3.0", "tags": ["json", "nested", "strict"], "scorer": "json_schema_lite", "expected": {"schema": {"type": "object", "required": ["version", "ready", "cases"], "additionalProperties": false, "properties": {"version": {"type": "string", "const": "v0.3"}, "ready": {"type": "boolean", "const": true}, "cases": {"type": "array", "minItems": 2, "maxItems": 2, "items": {"type": "object", "required": ["id", "weight", "active"], "additionalProperties": false, "properties": {"id": {"type": "string", "pattern": "^case-[12]$"}, "weight": {"type": "integer"}, "active": {"type": "boolean"}}}}}}}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "只输出 JSON 对象，不要 Markdown。字段 version 必须是 \"v0.3\"；字段 ready 必须是 true；字段 cases 必须是正好两个对象组成的数组。cases 中每个对象只能包含 id、weight、active 三个字段；id 是字符串且匹配 case-1 或 case-2；weight 是整数；active 是布尔值。", "parameters": {"temperature": 0, "max_tokens": 16384, "seed": null}, "prompt_hash": "7aa2b7b97ec6a230b8e08a0949edc3f573a3979a488949456e1106dcbad11695", "started_at": "2026-06-23T14:28:25.737430+00:00", "latency_ms": 12357, "output": "{\"version\":\"v0.3\",\"ready\":true,\"cases\":[{\"id\":\"case-1\",\"weight\":1,\"active\":true},{\"id\":\"case-2\",\"weight\":2,\"active\":false}]}", "usage": {"prompt_tokens": 142, "completion_tokens": 339, "total_tokens": 481, "completion_tokens_details": {"reasoning_tokens": 295}}, "response_id": "resp_021782224895501dd8ef7af441b125f2150dd3f48013f2a692433", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"errors": [], "value": {"version": "v0.3", "ready": true, "cases": [{"id": "case-1", "weight": 1, "active": true}, {"id": "case-2", "weight": 2, "active": false}]}}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"} {"case_id": "challenge-zh-writing-004", "suite": "chinese_writing", "case_version": "0.3.0", "tags": ["wechat", "summary"], "scorer": "text_rules", "expected": {"required": ["推理题", "边界条件", "人工复查"], "forbidden": ["排行榜"], "max_chars": 160}, "model": "doubao-seed-2-1-pro-260628", "provider": null, "backend": "volcengine-ark", "prompt": "写一段不超过 160 个汉字的公众号小结，主题是“为什么 v0.3 要扩到 100 题”。必须包含“推理题”“边界条件”“人工复查”，不能出现“排行榜”。", "parameters": {"temperature": 0.2, "max_tokens": 16384, "seed": null}, "prompt_hash": "2e406239f22e7454c39c9cc970d8388b92c83522c1edacf056286806f5933790", "started_at": "2026-06-23T14:28:38.097031+00:00", "latency_ms": 59548, "output": "本周更新的v0.3版本，我们把题库扩容到了100题：新增了大量考察完整逻辑链路的推理题，补齐了容易让模型“翻车”的各类边界条件场景，所有题目都经过多轮交叉人工复查校准，只为更精准锚定模型真实能力，减少单题偏差，给大家更靠谱的实测参考。", "usage": {"prompt_tokens": 116, "completion_tokens": 3477, "total_tokens": 3593, "completion_tokens_details": {"reasoning_tokens": 3399}}, "response_id": "resp_021782224907619df6479734761ee7b3ca927c635915b31794a24", "finish_reason": "stop", "score": {"passed": true, "score": 1.0, "details": {"missing": [], "missing_any": [], "present_forbidden": [], "char_count": 117, "length_ok": true, "length_scored": false}}, "error": null, "rescored_from": "/Users/dracohu/REPO/dracobench/runs/v0.3-challenge100-doubao-seed-2-1-pro-260628-ability16384.jsonl"}