DracoBench Report: z-ai/glm-5.1

z-ai/glm-5.1
Pass rate
95.0%
Cases
95/100
Avg latency
23829 ms
Total time
39m 45s
Total cost
$0.311060
Prompt tokens
9373
Completion tokens
76288
Reasoning tokens
65021
Errors
1

By Suite

SuiteCasesPassedPass rateAvg latencyCost
chinese_writing44100.0%36906 ms$0.013554
coding282589.3%44448 ms$0.176809
debugging1919100.0%12510 ms$0.024289
instruction_following66100.0%15014 ms$0.012982
rag_long_context181794.4%12896 ms$0.027728
reasoning252496.0%17232 ms$0.055699

Failure Examples

challenge-coding-003

Suite: coding
Finish: None
Prompt
只输出 Python 代码,不要解释。实现 solve(deps: dict[str, list[str]]) -> list[list[str]]。deps 表示 task -> prerequisites。返回可并行执行的拓扑层,每层任务按字典序排序;如果存在环,返回空列表。所有只出现在 prerequisites 中的任务也要包含。
Standard Answer / Scoring Expectation
Reference tests:
deps = {'build': ['lint', 'test'], 'test': ['install'], 'lint': ['install']}
assert solution.solve(deps) == [['install'], ['lint', 'test'], ['build']]
assert solution.solve({'a': ['b'], 'b': ['a']}) == []
assert solution.solve({'deploy': ['build'], 'build': []}) == [['build'], ['deploy']]
Mistake Analysis
Failure type: api_timeout_no_answer
OpenRouter 调用超过 300 秒后被运行器中止,模型没有返回可评分正文。后续代码测试只能拿到空文件,因此报找不到 `solve`。根因是 API/模型响应超时,不是该题算法逻辑被判错。
Scorer Details
{
  "type": "api_wall_timeout",
  "message": "OpenRouter request exceeded 300 seconds"
}
Output

challenge-coding-004

Suite: coding
Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(capacity: int, ops: list[str]) -> list[int],模拟 LRU cache。ops 包含 "put key value" 和 "get key"。get 返回 value,不存在返回 -1。put 已存在 key 要更新并变成最近使用。容量为 0 时所有 get 都返回 -1。
Standard Answer / Scoring Expectation
Reference tests:
ops = ['put a 1', 'put b 2', 'get a', 'put c 3', 'get b', 'get a', 'get c']
assert solution.solve(2, ops) == [1, -1, 1, 3]
assert solution.solve(0, ['put a 1', 'get a']) == [-1]
assert solution.solve(1, ['put a 1', 'put a 2', 'get a']) == [2]
Mistake Analysis
Failure type: runtime_type_assumption
题目里的 cache key 是字符串,例如 `a`、`b`、`c`,但模型把 key 强行写成 `int(parts[1])`。测试一遇到 `put a 1` 就在解析 `a` 时抛出 ValueError。根因是模型擅自补了“key 是整数”的类型假设。
Scorer Details
{
  "returncode": 1,
  "stdout": "",
  "stderr": "Traceback (most recent call last):\n  File \"/var/folders/x_/4cht1kfj1_j31gdv5w01sgmw0000gn/T/dracobench-code-rfcv_osl/test_solution.py\", line 4, in <module>\n    assert solution.solve(2, ops) == [1, -1, 1, 3]\n           ^^^^^^^^^^^^^^^^^^^^^^\n  File \"/private/var/folders/x_/4cht1kfj1_j31gdv5w01sgmw0000gn/T/dracobench-code-rfcv_osl/solution.py\", line 11, in solve\n    k, v = int(parts[1]), int(parts[2])\n           ^^^^^^^^^^^^^\nValueError: invalid literal for int() with base 10: 'a'\n"
}
Output
```python
from collections import OrderedDict

def solve(capacity: int, ops: list[str]) -> list[int]:
    cache = OrderedDict()
    res = []
    for op in ops:
        parts = op.split()
        if parts[0] == "put":
            if capacity == 0:
                continue
            k, v = int(parts[1]), int(parts[2])
            if k in cache:
                cache.move_to_end(k)
            cache[k] = v
            if len(cache) > capacity:
                cache.popitem(last=False)
        else:
            k = int(parts[1])
            if k in cache:
                cache.move_to_end(k)
                res.append(cache[k])
            else:
                res.append(-1)
    return res
```

challenge-coding-007

Suite: coding
Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(s: str) -> bool,检查括号 ()[]{} 是否配对。单引号或双引号内部的括号忽略;反斜杠会转义引号。输入字符串可以包含普通字符。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve('{a:[(1)]}') is True
assert solution.solve("{'x': '(]'}") is True
assert solution.solve("['unterminated]") is False
assert solution.solve('([)]') is False
assert solution.solve("\"paren \\\" ) still quoted\"") is True
Mistake Analysis
Failure type: degenerate_output
模型输出包含大量无意义重复片段或内部标记,无法作为题目答案解析。finish_reason=`stop`,说明失败发生在答案生成稳定性层面,而非当前 scorer 过严。
Scorer Details
{
  "returncode": 1,
  "stdout": "",
  "stderr": "Traceback (most recent call last):\n  File \"/var/folders/x_/4cht1kfj1_j31gdv5w01sgmw0000gn/T/dracobench-code-cwsyuip9/test_solution.py\", line 1, in <module>\n    import solution\n  File \"/private/var/folders/x_/4cht1kfj1_j31gdv5w01sgmw0000gn/T/dracobench-code-cwsyuip9/solution.py\", line 1\n    lue></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>\n     ^\nSyntaxError: unexpected character after line continuation character\n"
}
Output
\\|\\|\\</think></think></think></think></think>| I'm | \\</think></think></think></think></think></think></think>| I think a| \text\|\text|  |I|\text| \\ |I am a|\text||\text||\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|ted</think></think></think></think></think></think></think>|I \\||\|\|||\text||\</think></think></think></think></think></think></think>|I ||</arg_value>|\|\</arg_value></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></arg_value>|\||\|\|\|\</arg_value></arg_value></arg_value></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></arg_value>||\|\|\|\|\</think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></arg_value></arg_value></arg_value></arg_value></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>|\|</arg_value></arg_value></arg_value></think></think></think></think></think></think></think></think></think></think></think></think></think></thi

challenge-rag-001

Suite: rag_long_context
Finish: stop
Prompt
只根据资料回答。

资料 A:DracoBench v0.2 的目标是提高区分度,优先增加 coding、debugging、RAG 和 instruction_following 题。
资料 B:DracoBench v0.2 不会默认使用 LLM-as-judge;开放中文写作题只做少量规则校验和人工抽检。
资料 C:v0.2 的 HTML 报告必须展示通过率、成本、reasoning tokens、失败样例和逐题明细。

问题:v0.2 为什么没有把 LLM-as-judge 作为默认判分?
Standard Answer / Scoring Expectation
Required phrases: 人工抽检
At least one from each group: 默认使用 / 默认判分 / 默认采用 / 作为默认判分
Forbidden phrases: 因为成本太高、因为不支持中文
Length guidance (diagnostic only): min=n/a, max=160
Mistake Analysis
Failure type: missing_required_info
问题问的是“为什么 v0.2 不把 LLM-as-judge 作为默认判分”。资料 B 给出的关键依据是:开放中文写作题只做少量规则校验和人工抽检。模型回答 `根据提供的资料,没有说明 v0.2 为什么没有把 LLM-as-judge 作为默认判分的原因。` 只是复述“不会默认使用”或说资料未说明,没有说出替代判分方式里的“人工抽检”,因此缺少真正解释原因的证据链。
Scorer Details
{
  "missing": [
    "人工抽检"
  ],
  "missing_any": [],
  "present_forbidden": [],
  "char_count": 48,
  "length_ok": true,
  "length_scored": false
}
Output
根据提供的资料,没有说明 v0.2 为什么没有把 LLM-as-judge 作为默认判分的原因。

challenge-reasoning-015

Suite: reasoning
Finish: length
Prompt
只回答最终数字:集合 S 初始为 {a,b,c,d}。删除字母序在 c 之前的元素;加入 e 和 b;再删除所有元音字母。最终集合有几个元素?
Standard Answer / Scoring Expectation
Answer: 3
Mistake Analysis
Failure type: degenerate_output
这道集合题本身很短,正确答案是 3,但模型没有给出数字,而是输出了大量跨语言碎片、内部样式词和伪代码片段,直到 finish_reason=`length`。根因是生成阶段退化/失控,不是集合操作推理失败。
Scorer Details
{
  "expected": "3",
  "actual": "_FS\n\n 'https Ri cries_processesaabddb'https họ'https miserリー họ\n\n'https OMG'https'https OMG ''ils'em prosperity '_firestore Wilde Cooperative ' Zoies'https Everyday cooperative\n\n'httpsrü.bio Wilde人寿 你们CCC'L Они Yours007007'https Они\n\npsc ''https'https Cooperative ' nonsense_redis\n\n wildly rhythms_processes_startup '_redis_FSоторые miser.bio_redis miser nonsense  the ''https'https'https_processes ' Они\n\n.psc '_redis powering 'psc miser\n\n'https.'https they OMG'https'https shouts loudly cries Č cries cries https Yours'https loudly shouts'https miser họ.bio.fhir'https. OMG pioneers cries_glob miser'n OMG Everyday. the'https Yours\n\n'https 'https'httpspsc'https melanch'https\n\n'https'https珂 Они_redis.bio.bio\"?\n\n؟ loudly=https OMG\n\n\n\npsc\n\n ' họ声器oggles_redis\\Doctrinepsc cries cries OMG họ_poll'https'https loudly_startup Yours –\n\n miser OMG_FS\n\n the Yours ''https Yours_workflow\n\npsc'https'https'https'httpsidataЯ  Yours Zo'https Yours_redis\n\npsc_processes'https ' Yours Они'https'https loudly 'https'httpsЯ Yours_glob            \n\n miser\n\n 'https'https'https\n\n ''https'https.fhir века\n\npsc '\n\n'https_processes loudly Yoursidata\n\n'https\n\n OMG'https ' Она '\n\nzshtt miser_processes\n\n OMG'n OMG\n\n họorganisation nonsense'https loudly'https she loudly silently=https'https Онаribbon'ils.biopsctis  họ ' phenomena 'https rebellion.bio отчет.runners.bio.loader miser_workflow\n\n'httpsspecs'httpsilos nonsense\n\n\n\n_pollždždlockerstylοιItemListItemList mastur pz magas cooperative她又”。\n\n_publisherebo_textsПО소οιstylstylònpoi_interactioncolapiglölö 出版的图书Х loudly-parseТОПО소mdllöVolumespulХХХ nerlölockerPPPنبПОХ소pulstyllö_pollХТО_poll сталоoguiПОaszt книгиfunkVolumes tn-dat192pul нейfunk_texts_docs,...\n_docsMALlö_reading_texts소pliMAL 有利于Parsed195 _textsăn läs-exc\trbdna raispoi_readingПО бог소esti nó_textsIgطل_docs소opf소отворithefavporta소 bola 出版的图书ăn nó 有利于 jb_pipeline Letters [,scientific002 poi tá142РО_ta ela-eventsArchiveVintage-pe неб 有利于 有利于-dat meteor002소MAL jb192رضVintageVintage風182192ху restaithe MALMALgolden 有利于 lại_critical-launchотвор débMAL 今 (? lạiổi-evvat-newsstylита tựhoz_textsInitialStateотворpli002 AppearsInitialStateыс zdeطلотвор 当时 появ ” hads and_launcher Она_resolver FogХhä:,Vintage,on 有利于192porta-ro,eppompul :\n\n\n\n :\n\n\n\n-bookspompom’elleănestiVolumesstyl_launcherعل-ro-java_textsfunk-books-ro;.raisVolumes-booksRaised_texts19_docsMAL\trb gwХstylpli_pythonruby_readingхуobiejournal002-ro\trbJsonObjectlaisParsedstylpliobieVolumes-ro_py_taVintage Appears Gale_texts eo-news tome houobie eodbgstyl소MAL gw hone ellaMAL Galeнарод gwhomes hone Mercury läs_prep 出版的图书  онаLi\trb 出版的图书和人cycl_mc ejerreadsăn masturstylJsonObject xb hone hone hone_docsMALIg ejerănMALнародJsonObjectПОJsonObjectMALReadableFo Mercury трурошMAL GorIg소 gv elecTs-roPoseRadJsonObjectJsonObjectLead tesLi_docs_widgets stamMALVintage gwTs gwRadХTsmaidsJsonObject  ”MALlömaids_docs Нов_docs_radicubeestiVolumes -._widgetspill tnopo marspliVintageJsonObjectjournaljournalVolumesVintageРАодарOVERRad10_docs_docsХ Gor Mercury standardized-eventsIg_poll_widgets-eventsозит 出版的图书Volumesstyl ner.PO 听 Wilde\"、cubepig gw-books ейNEG pz был のХstylJsonObjectlö_docsедиTslö nó tôiPoseIg ZahIg\trbstyl bä bä_docsJsonObject_docs_efХ biedХ ZahCubeJsonObject,key:,GRPодарCube-eventscube gwcube_widgetsTsTsVolumespulJsonObjectLead ejerstylchriftХmaidsFoХlöJsonObjectstyl ejerGRPabomaidsnjDataManager_docsХlömaids_docsJsonObjectитационpisJsonObjectTsодарJsonObject MercuryVolumes_pipeline_widgetsStoneХpul_shellJsonObjectGlobalsolle-roLiobieοι nfpoi ejerIg소journalmaidsItal dn MercurylöодарpliLilölö_docsMALCubeMAL Gale dor tôiJsonObject такой_pipeline_docspulJsonObjectطلMALpul_docsХJsonObjectTs_docsLiabo nguobie ей vg nó_docs-lapomХХита [,_docs GaleIxlöăncubeVolumes-ro былитаPoseХοι\tgui_reading_googleоди _docsJsonObjectULA MAL-la_community_docs_docs_docs ,\r\n195Volumes книги_docs_docs_docs русTs-eventsди_docsJsonObjectроз_texts года_docsХ 当时;._docs magas elapom  ” года ]\r\n_docs книги_docs_widgetsdatal было ей tongues_docs_docs_docs_docsРАfunk ей_docs_docs -.calculator-parseJsonObjectLilöХpsc_docsReadable_docs _docs_widgetsidentiись 当时_docs_docs-events_docsfunk_docs;. Collector_launcher aurcubeTsJsonObject_launcher_docsIg_community-events自已_docs_docs_beta_docscalculatorCollectorRad_docs стала_docs было_docsди dwarf ей было_starts_widgetsди-events wl ola\tgtk_docsись -._docs ” 出版的图书_docs_betarais ей было_docs Wilde_docsسیPose-booksvista_docsstyl_docsцион 丁_docs-events книги_docscalculator_docs_docscollections_widgets_docs_docs-ro ela人会-laгиcube_docs Gale_docs_docs_widgets_docs_docs_bo_widgets -._docs стар深化改革_docs_docs_docsцион_calendar было_docs_docs ейказа_beta_docs было_widgetsХ_widgets казVolumes_docs_docs_docs_docs_docspsc_docs_beta мая_docspois_launcher_docsись_docs珂_docs помог-roди-eventsVolumesletters_docsозит_docs_widgets_docsCollector_launcherάλcube-editor_docscubecalculator_docs -. 丁диfunkabo-librarystylfunkabo_launcherТО_widgetscalculator_docs_docs_docs_beta_docs элем_docsCollectorlettersдиcalculator Gale_docs_docs_docsReadable_docs_docsReadableStone-events_launcherцион百余_launcherCollector_docs_docsVolumesNEG-eventspscRadCollectorVolumesCollector-books_docsCollector_docs-eventsNEG水利工程_docs_docsХ_widgets_docsportaVolumes -.JsonObjectolle-editor -._docsCollectorCollector_docsди_docs -.Х\teventscollectionscalculatorJsonObjectХCollectorдиCollectorCollectorCalculatorХCollectorCollectorCube -.Collector_docsди_docscubeди-library_docsfricalculator_docs_docs (?_docs_docs_docsSounds дан_docs_docs_docsRad_docs_finderCollector.poifondcalculator_docs_docs_docsReadableCollector_docsCollector-library_docsди_docs_texts_docsjb Appears\trenderer года ей-events McGr-events_docs-library_docs-booksRaisedReadableCollections_docs:, magas_docsVolumesдиstylди_docs dor_docs -,-la tes_docs -._docs-la_docs-eventsCollectoraso -.ди_docs :\n\n\n\n Rising_dictionaryитаCollector-books_docsCollector_baseline_docsХ MercuryCubeCollector-booksReadable -._docsThemesVolumesVolumes_docsReadable_docscalculator_docsRad_docs_docs_docs_docs_docs_docs Gould_docs_docsCollector_docs_docsfond ông-libraryReadable-la\trb_docs_docs_widgets_docs_widgetscollections_docs_docscubecalculator_docs-booksCollector Gould;. tesAud_docsReadable-events winesCollectorReadable :\n\n\n\nRaisedCollectorcalculator_docs_launcherCalculatorLiver-bookscalculator_docs _docs ellatourubycollector theCollectorReadablelacReadablefond_docs_docs_launcher_docsCollectorVolumes_pipelinefondcollectionsοι 字_docs sounding_docslö_docs_docs_docsbucketsReadableди_docs_docscollections_docs-eventsVolumesди_docsди-libraryRad_docsFetcher学和диReadablecalculator_dictionary_docs débди_docsFetcherVolumesгиcollections_docs_docs_docs-libraryRaised-books.PO_docs_launcherFoХ_docs_launcherCollector_launcher ola_docsaboabo_docsReadable_docs стар dor_docs为王_launcherVolumes_launcher_fm_docsfond_docsReadablecalculator-books_docs_docsRaised_docscalculator-books winesVolumes_docs dor-la McGr_docs材料和_docsдиCollector_docs_launcher_docs_launcher_docsabopsc_docs_docs-booksfond_docs_docsCollector_docs_launchercalculatorCube_docs_ta_docsGlass見_docs läsдиCollector_docs ola-ro:l_docs_docsCollector-booksCalculatorCollector-booksfond_launcher perse_docs_docs_docs_docs_docs和老_docsозит酒的_launcher원_docs_docsfond和老_docs_docs звLauncherVolumes_launcherGlass_launcher_launcher старcalculator_launcherVolumes_launcherVolumesabo дня_launcherVolumes_launcherodi:\"._raiseLauncher stylackers_launcher wines_launcher見_launcher.digital град winescalculator wines_launcherIgIg.magっ wines\":\n\nClub года 村 годаcalculatorNEG的事儿Ο wines\tbtn和老_launcher_launcher\":\n\n家和_launcher_launcher года стали_radi bulbs_launcher数次 ae\").\n\n”),calculator_launcherVolumes람 wines可不是 the_launcher’ мая_launcher -._launcher_launcher ):\n\nstyl bulbscalculator GouldΡ_launcher_launcher Gould ella_launcher ):\n\n_launcherVolumes годаYEAR Ведь мы ner.calendar мая_launcher;-_launcher器和Launcher_launcher wines_launcher_launcher珂Launcher_finder.sound_launchercalculator_launcher什么事情_launcher. ella_launcher.loadData_launcherstyl 丁_launcher_launcher –\n\n_launcher мы_launcher _launcherΟcalculator bao ella ella_launcher годаLauncher  –\n\nги”.\n\n_launcher_launcher года ).\n\nгиBeer ).\n\ncalculator_launcher_launcher_launchercalculator_launcher_launcher_launchercalculator_launchersscdacVolumes铎calculator再多 ella_launcher года_launcher wines_launcher сама_lm ).\n\n_docs_collections даcalculatorabo铎aboFx_FS_launcher года_launcher器和HF_raise_launchercalculatorcalculator_launcherVolumes.digital_launcher –\n\n я ):\n\n ):\n\n ):\n\n ):\n\n_launcher вред годаwins女郎 ).\n\n_launcher_docs_launcher \"\");\n\n;-_launcher ):\n\n ):\n\nwins_launcher –\n\n_launcher.calendar_launcher.RESET года_launcher\").\n\n_launcher года –\n\n_launcher:\"._launcher_launcher_launcher ):\n\n_launcher –\n\n ):\n\nLauncher ):\n\n_shellLaunchercalculator ):\n\n这些年的事儿_raise_launcher_launcherLauncher_launcher_launcher_launcher спрос_launcher_launcher_launcher ):\n\n_launcher ner _launcherassel.bio_launcher_launcher –\n\n_launcherIncomplete новостиcalculator_launcher_launcherLauncher –\n\nItalstyl ner_launcher ner_launcherΟRace_launcher_launcher годаCollectorLauncherNEG\trenderer _launcher_launcher года_launcher汞_launcher_findercalculator года товаровLauncher ).\n\n_launcherYEAR_launcher_launcherΟLauncherLauncher再多 ):\n\n_launcherΟYears –\n\n_launcher ella_launcher_launcher там'elle ellaги莺_launcher –\n\n_launcher_launcher стала_launcher.;- bao ):\n\n_launcher года_launcherLauncher ella там там_launcher года;-_launcher bao_fm_launcher_launcher_launcher ella.runners_launcherbeercalculator ):\n\n ):\n\nulled –\n\n   да_launcher_radi каз да液化_launcherги.students да_launcher_launcher да да_launcher_launcher-la года tá_docs-pill.mag_launcherstyl-eventspoisХcalculatorCollectorга_launcherRaisedита 曹calculatorFetcherCollector McGr-el неб-ft débVolumes-filesVolumesRaised\tevents-booksRaised-pill-ro_textsfunkstyl新闻报道_fmитаCollectorитаOVERХ ella_reading -,_docsWiki -. dare”.\n\n ella_raise dareaboabo”.\n\n да_launcher ζ.party珂 practise_FS rising practise_FS (?_FSFinderGa обще_FS_mr_FSWalker_launcher practiseAppear_FS_FS practise_FS_RBWalker -.aboWalker_finder_FS practise_FS_RB_FStober_FS_FS practisetoberpps practise-IS_FS practise人会_RBFinderGoods_RBFinderGoods_FSGoodsWalker-IS practise人会_FStober_FSWalker_FS-IS_FSGoods_RB_FSFH人会GoodsισGa_FSWalker_MR_FS人会Walker_Detailspsc PickertoberWalkerodie_FSGoodsGoodsViaWalkPractice_RB_FSPractice_mrinsk practise_FS Wilde_FS Picker.idea_RBGoodsGoods人会 WwwFinder Gould Sounds_launcherGoods_FS.debита Voll inex余生_FS_RB_RB VollWalkerichi_FS Picker_FS Remed practisePracticeGoods_FS_FS_FS_FS_FSGoodsGoodsFHograd余生tober_FS Remed haitoberFHogradFHtoberaboFH_FS_FSFinderFinderFindertoberpps琥珀WBFHFinder和睦 FinderFinder FlintbalancedFinderPracticeWBFinder_FSinsk_CIoux人会tober_FSersteowoFindertober人会 practisetoberFHppsobot_RBGoods practiseGrammar.Ver_FS Flint人会WBPracticetober practisetoberppstoberFinder Flint_FS_FS RemedPracticefierWB人会_FStoberppsograd.whichPractice_FS_FS porterGoods人会艺术学院人会WB_FS_FSFinder_FS practise SignsFHWARtobertoberercippstoberFHFinder_FStober WI好啊PracticeGHoux娘娘WB_FS余生娘娘Practice_FSGuesstobertober_FSGoodsWB Picker_FS好啊tobertoberFHFinder/cc珂_FS_FSresses RemedFHbalanced声器Finder_FSFHVerifier списка람_FSFHPILE_FStober_FS_FS_FStoberFinderinsktober Remed_FS_FSigurFH_FS_FS_FS_FS_FSGa_FSGaFH_FSigurtober好啊_FSerna SignsodiaGHGHpps_FSigurigurigurppsISONppsFindertoberFinderpps inexcollectorPracticetoresrobiFHFinder_FS_FSFH Remed_FS_FStoberaday_FSigurtoberbalanced_FSFH人会Walk好啊igur_FS人会PracticePracticeogradtoberigurGoodsGHtober人会GH_FS Flint_FSPracticetober_FStober_FSbalancedolith callableFHtober声器FHGHGoodsGoodsmailer人会итаISONigur不明人会GoodsFH_FS人会igurigurpps_FSppsита_FS_FS_FSFHBinderGoods_FS Picker_FSigurtoberGoods_FSFH_FSGoods inexigur_FSFH Finderpps_FSernaianne Finderpsc:listCollector FinderFinder Pickerikersianne.calendar.calendarelight_sunelight.matcher.WRITE,_launcher sorterď_RBigur_FS-launchbalanced_FS_FS_FSGoodsGoodsinsktober_FSigurGoods余生ograd_FS_FSGa_FSigurilverppsGoods_FS_FS RemedGH.matcher_FS_FSigurskie_FSGHigurGoodstobertoberFHNit_FSGoodsGoodsograd_FS VIIussenPractice娘娘 GregolianGoodsGoods practiseGoods Colleg Remed Sonnikers Flintigur FlintGoodsigur_FS Ridertober Ridertober Remed_FS吉尔FH_FS_FS艺术学院_FS所欲Finder-ISита.VerPractice.debitel好啊WBита.debmailerGoods practise.Ver Remed_FStobertober_FSbalancedinsen.dx_FS_FS艺术学院 Remed Flint Wilde Remed人会AIR_FS_FStober RemedGoods好啊НовGoods七天MillerFH_FS_FSита.dxigurtober RemedGoods人会igurита Flint余生_FSGoodsoux人会_FS_FS Flintigur_FSGoods人会_FS_FSGoodsGoodsита.dxVerifier_FSGoodspickerGoods PickerVerifiergeist_FSmailer余生AIRmailer inexGoods_FS-_PSVerifier VollPractice Remedツ谤Verifier_FSmailer余生余生 SignsbaiGoodsитаGoods Signsigur_FSитаGoods_FS FinderAIRVerifierosteigur.VerGoodsGoods_FSigurGoods余生 inexgeistigur numa complimentaryynaVerifierGoods_FSď.Ver PickertoberVerifier.dx.dx Flint Remed.Ver所欲geistVerifier Signs_FSskiepickerFH Pickergeist Flint_FS_FSigur_FSVerifier.debigurigur_FSGoodsigur.VerGoods Voll所欲geistigur余生WB RiderGoodsGoodsgeist Signsinskbalancedtober所欲GHGoods inex reimbgeistoda所欲 scatterGHianne所欲pps_FS_FS娘娘ита_FSppsGoods inex_FS_localeigurodaigur声器.deb Picker所欲_salt所欲七天 Remed_FS WI_FSigur_FSigurитаита所欲 Guessigur Signs Signs所欲所欲178GrammarVerifier inex Wilde.dx inex Flintigur_FSskie inexита.dx inex_FS inexитаWB Flint inex_FS_FS scatterGoods_FS余生_FSigur_FS Signs Signsigur_FS Signs inex_FStober_FSpps SignsAIR inexитаFinderита Signs_FS.dxbalancedитаита_FSGoods所欲ита所欲七天geist Picker inex.dxigurucha私立ianne Goosetobertober吉尔pps_localeAIR.dxgeist七天_FS_saltppsита inexитаppsFileSystem Examiner所欲igur Signs所欲ianneigurAIR所欲tober Sonn所欲.va.va inexbalanced SonnVerifierFindergeist所欲ianne所欲ouxFinderbalancedитаVerifier Examinerита所欲 Signssalt-bootgeistiannegeist Sonn.va不言geisttober.va_sun底气geist_FS sorterianne七天GH mareigurgeistigur trophies Via Signsigur inex pinnedletteianneigurFH扫码skiefier不明 Signs_FStober inexpicker Picker inexитаigur_counters_FSikkaickers dockstoberGoodsита/ccитаikkapicker艺术学院igurGoods_FS_FS inex所欲ита_saltigurigurodiaGoods inex Findermailerита Examinerigur_salt Finder_FS Signs Remed扫码 inex inex WI.vaigur.vatoberGoodstober所欲geist所欲_saltGoods PickerFH Pickeritizenianne所欲_FS所欲 Pickertobertober.va_dictsFileSystem_saltandex.son.va inextober_salt Remedigur.va inex所欲Verifierigur Rider Signstober所欲_FS Signsikkaresses_salt所欲所欲geist_salt Signsigur_FS服务中心geist_salttoberigurgeist inex.Verитаgeistigur所欲 Sonn余生igurgeistoux所欲 inex.Ver inex余生 inex Remedianne.VerVerifiergeist inex inex_salt余生igur.vaoux Wildeianne Signs_salt inex余生 Wilde inexmailer_salt.MSG艺术学院 Remed_FS.vaigurVerifierigurPDabeebalancedитаilorigurPD Signs Signsgeistigurskie不明geist FinderPDgeist Examiner艺术学院itelgeist Examiner xcラー списка xcАрАрита списка_letterАритаАр+._letter_sunАрита спискаитаита tempListItemList_sun Sonn_sun-languageАр-language_sunита_sun.arm_locale-languageАр.arm.armickers:list-language.armАрАр\txtype.calendar.arm.arm.arm trophies..:listАр.arm.arm trophies_locale trophies.arm trophies_lc(lista:listитаитаита.arm trophies.arm.armilor.arm.arm:list.arm trophies.arm:list_locale.arm trophies.arm:listАр trophies.arm.armита.arm.arm声器.arm.arm.arm trophies.arm trophies.armита..armАр1ilor.calendar.calendar:list.arm.calendar.calendar trophiesита.arm声器ита.arm Ар..arm.arm trophiesلف.arm.calendarита.arm меньше_lcита trophies声器ита.calendar trophies.less trophies:listита.less声器.less:list.calendar.calendar trophies声器 trophies.calendar.calendar:list.arm.arm\tassertFalse.arm trophiesита trophies.calendar trophies+t_localeита+t.arm_lcита trophiesАр Аритаulg trophies.calendar..arm trophiesитаита:list спискаazo声器_lc.calendar trophies. trophiesита trophies trophies气和. trophies_locale声器.calendar.armulg_lc.arm trophies.artıl.armfier.less.less trophies trophies.less.art声器_sun.calendarulg.calendar trophiesickersита.calendar.calendar списка.less_collections.less trophies.calendar..less:list trophies_ENTRIES trophies trophies trophiesита trophies声器_lc.calendar trophies.calendarfier trophies.calendar_launcher.bio.armfier声器 trophies trophies.calendar trophies.less:list trophies.calendar_lc_wall_collections.arm.calendarита trophies trophiesита.lessfier.calendar:listلس Ар.armита.less.bio声器.arm.calendar.calendar.art.less trophies.arm.calendar_collectionsита.less.less(lista.arm_collections.arm:listласьLauncher.less►.bio声器_launcher Ар.bio_launcher_collections.armита.calendar_launcher.calendar_launcher.armитаита_launcher_launcheritta arte.bio_launcher_launcher арАр气和.lessickers arteickers_lcCollector钊 арickersАр.art.bio arte arte_audit:listCollector искус气和气和Collector.bio.less ар艺术Scanner ар_audit.bioلس Arte:listilliilliittel imassc_challenge.bio начал:listildailliCollector_launcher.less lvl气和Collectoralisation callersollo气和.lessisses арcollector launcherschein Wie为难氣为王珂太郎.artieux气和elightmounted.lessWalk maskedSStiva.lessinhaarkigurucha_FS_visitor.OS为王 checkeriesen искус.diffitestVERIFYrosse-sk_lc.diffickersApolloickers.less艺术学院schein Picker Flood람_lc艺术学院看不起ош Flood.axes_tools Daniels inex yourselves所欲 tenta.bio MaoternaAPS換Mailerlearner/errors ар艺术学院hallWalk SPELL дух_lc paralle. Rider marsussen.less不明/errors-training气和/art crosses.vanier искус_visitoritel气和砚 complementary yourselvesbagsMessenger crossesikers Rideroux換 искус氣珂 искус SPELLumasussen气和itelilda complimentaryikkageistMailer七天_visitor.va气和 Goose scareoya Auroraoux Aurora paralleodia/photoikkaipa气和msgigur_auditза.va_auditolaАр Danorsa 听ilda_sku_audit_auditorta.less humour Dana Arte dn.ser.infinity hoa气итаclimateapsienne9曜195气isciΟoux\tADD"
}
Output
_FS

 'https Ri cries_processesaabddb'https họ'https miserリー họ

'https OMG'https'https OMG ''ils'em prosperity '_firestore Wilde Cooperative ' Zoies'https Everyday cooperative

'httpsrü.bio Wilde人寿 你们CCC'L Они Yours007007'https Они

psc ''https'https Cooperative ' nonsense_redis

 wildly rhythms_processes_startup '_redis_FSоторые miser.bio_redis miser nonsense  the ''https'https'https_processes ' Они

.psc '_redis powering 'psc miser

'https.'https they OMG'https'https shouts loudly cries Č cries cries https Yours'https loudly shouts'https miser họ.bio.fhir'https. OMG pioneers cries_glob miser'n OMG Everyday. the'https Yours

'https 'https'httpspsc'https melanch'https

'https'https珂 Они_redis.bio.bio"?

؟ loudly=https OMG



psc

 ' họ声器oggles_redis\Doctrinepsc cries cries OMG họ_poll'https'https loudly_startup Yours –

 miser OMG_FS

 the Yours ''https Yours_workflow

psc'https'https'https'httpsidataЯ  Yours Zo'https Yours_redis

psc_processes'https ' Yours Они'https'https loudly 'https'httpsЯ Yours_glob            

 miser

 'https'https'https

 ''https'https.fhir века

psc '

'https_processes loudly Yoursidata

'https

 OMG'https ' Она '

zshtt miser_processes

 OMG'n OMG

 họorganisation nonsense'https loudly'https she loudly silently=https'https Онаribbon'ils.biopsctis  họ ' phenomena 'https rebellion.bio отчет.runners.bio.loader miser_workflow

'httpsspecs'httpsilos nonsense



_pollždždlockerstylοιItemListItemList mastur pz magas cooperative她又”。

_publisherebo_textsПО소οιstylstylònpoi_interactioncolapiglölö 出版的图书Х loudly-parseТОПО소mdllöVolumespulХХХ nerlölockerPPPنبПОХ소pulstyllö_pollХТО_poll сталоoguiПОaszt книгиfunkVolumes tn-dat192pul нейfunk_texts_docs,...
_docsMALlö_reading_texts소pliMAL 有利于Parsed195 _textsăn läs-exc	rbdna raispoi_readingПО бог소esti nó_textsIg

All Questions & Answers

challenge-coding-001 PASS
Suite: coding · Latency: 33196 ms · Cost: $0.008464 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(lines: list[str]) -> dict。lines 是简化 INI:空行和以 # 开头的行忽略;[section] 开始新 section;key=value 写入当前 section;key/value 两侧空白要去掉;同一 section 内重复 key 后者覆盖前者。返回嵌套 dict。
Standard Answer / Scoring Expectation
Reference tests:
lines = ['# comment', '[db]', ' host = localhost ', 'port=5432', '', '[feature]', 'enabled=true', 'enabled = false']
assert solution.solve(lines) == {'db': {'host': 'localhost', 'port': '5432'}, 'feature': {'enabled': 'false'}}
assert solution.solve(['[x]', 'a=1', 'a=2', '[y]', 'b = hi']) == {'x': {'a': '2'}, 'y': {'b': 'hi'}}
Output
```python
def solve(lines: list[str]) -> dict:
    result = {}
    current_section = ""
    for line in lines:
        line = line.strip()
        if not line or line.startswith('#'):
            continue
        if line.startswith('[') and line.endswith(']'):
            current_section = line[1:-1].strip()
        elif '=' in line:
            key, value = line.split('=', 1)
            result.setdefault(current_section, {})[key.strip()] = value.strip()
    return result
```
challenge-coding-002 PASS
Suite: coding · Latency: 30751 ms · Cost: $0.005914 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(events: list[str]) -> int。事件格式:"<id> credit <amount>" 增加余额,"<id> debit <amount>" 减少余额,"rev <id>" 撤销此前尚未撤销过的同 id 事件。重复交易 id 只保留第一次,未知撤销忽略。返回最终余额。
Standard Answer / Scoring Expectation
Reference tests:
events = ['a credit 100', 'b debit 40', 'a credit 999', 'rev b', 'c debit 10']
assert solution.solve(events) == 90
events = ['x debit 5', 'rev y', 'rev x', 'rev x', 'z credit 7']
assert solution.solve(events) == 7
Output
```python
def solve(events: list[str]) -> int:
    balance = 0
    processed = {}
    for event in events:
        parts = event.split()
        if parts[0] == "rev":
            tid = parts[1]
            if tid in processed and not processed[tid][2]:
                amt, typ, _ = processed[tid]
                balance += -amt if typ == "credit" else amt
                processed[tid] = (amt, typ, True)
        else:
            tid, typ, amt = parts[0], parts[1], int(parts[2])
            if tid not in processed:
                balance += amt if typ == "credit" else -amt
                processed[tid] = (amt, typ, False)
    return balance
```
challenge-coding-003 FAIL
Suite: coding · Latency: 300003 ms · Cost: $0.000000 · Finish: None
Prompt
只输出 Python 代码,不要解释。实现 solve(deps: dict[str, list[str]]) -> list[list[str]]。deps 表示 task -> prerequisites。返回可并行执行的拓扑层,每层任务按字典序排序;如果存在环,返回空列表。所有只出现在 prerequisites 中的任务也要包含。
Standard Answer / Scoring Expectation
Reference tests:
deps = {'build': ['lint', 'test'], 'test': ['install'], 'lint': ['install']}
assert solution.solve(deps) == [['install'], ['lint', 'test'], ['build']]
assert solution.solve({'a': ['b'], 'b': ['a']}) == []
assert solution.solve({'deploy': ['build'], 'build': []}) == [['build'], ['deploy']]
Mistake Analysis
Failure type: api_timeout_no_answer
OpenRouter 调用超过 300 秒后被运行器中止,模型没有返回可评分正文。后续代码测试只能拿到空文件,因此报找不到 `solve`。根因是 API/模型响应超时,不是该题算法逻辑被判错。
Output

  
challenge-coding-004 FAIL
Suite: coding · Latency: 99770 ms · Cost: $0.005782 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(capacity: int, ops: list[str]) -> list[int],模拟 LRU cache。ops 包含 "put key value" 和 "get key"。get 返回 value,不存在返回 -1。put 已存在 key 要更新并变成最近使用。容量为 0 时所有 get 都返回 -1。
Standard Answer / Scoring Expectation
Reference tests:
ops = ['put a 1', 'put b 2', 'get a', 'put c 3', 'get b', 'get a', 'get c']
assert solution.solve(2, ops) == [1, -1, 1, 3]
assert solution.solve(0, ['put a 1', 'get a']) == [-1]
assert solution.solve(1, ['put a 1', 'put a 2', 'get a']) == [2]
Mistake Analysis
Failure type: runtime_type_assumption
题目里的 cache key 是字符串,例如 `a`、`b`、`c`,但模型把 key 强行写成 `int(parts[1])`。测试一遇到 `put a 1` 就在解析 `a` 时抛出 ValueError。根因是模型擅自补了“key 是整数”的类型假设。
Output
```python
from collections import OrderedDict

def solve(capacity: int, ops: list[str]) -> list[int]:
    cache = OrderedDict()
    res = []
    for op in ops:
        parts = op.split()
        if parts[0] == "put":
            if capacity == 0:
                continue
            k, v = int(parts[1]), int(parts[2])
            if k in cache:
                cache.move_to_end(k)
            cache[k] = v
            if len(cache) > capacity:
                cache.popitem(last=False)
        else:
            k = int(parts[1])
            if k in cache:
                cache.move_to_end(k)
                res.append(cache[k])
            else:
                res.append(-1)
    return res
```
challenge-coding-005 PASS
Suite: coding · Latency: 7696 ms · Cost: $0.001103 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(base: dict, patch: dict) -> dict,执行 JSON Merge Patch:patch 中值为 None 表示删除该 key;两个值都是 dict 时递归合并;其他值覆盖。不能修改输入对象。
Standard Answer / Scoring Expectation
Reference tests:
base = {'a': 1, 'b': {'x': 1, 'y': 2}, 'c': 3}
patch = {'b': {'y': None, 'z': 4}, 'c': None}
assert solution.solve(base, patch) == {'a': 1, 'b': {'x': 1, 'z': 4}}
assert base == {'a': 1, 'b': {'x': 1, 'y': 2}, 'c': 3}
assert solution.solve({'a': {'b': 1}}, {'a': 5}) == {'a': 5}
Output
```python
def solve(base: dict, patch: dict) -> dict:
    result = dict(base)
    for key, value in patch.items():
        if value is None:
            result.pop(key, None)
        elif key in result and isinstance(result[key], dict) and isinstance(value, dict):
            result[key] = solve(result[key], value)
        else:
            result[key] = value
    return result
```
challenge-coding-006 PASS
Suite: coding · Latency: 34369 ms · Cost: $0.002339 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(text: str) -> int,把时长字符串转换成毫秒。支持整数单位 d、h、m、s、ms,可有空格,单位可重复,顺序任意。例如 "1h 30m 500ms"。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve('1h 30m 500ms') == 5400500
assert solution.solve('2d 1h 2s') == 176402000
assert solution.solve('100ms 2s 3ms') == 2103
Output
```python
import re

def solve(text: str) -> int:
    unit_to_ms = {
        'ms': 1,
        's': 1000,
        'm': 60000,
        'h': 3600000,
        'd': 86400000
    }
    total_ms = 0
    for val, unit in re.findall(r'(\d+)\s*(ms|s|m|h|d)', text):
        total_ms += int(val) * unit_to_ms[unit]
    return total_ms
```
challenge-coding-007 FAIL
Suite: coding · Latency: 27096 ms · Cost: $0.005129 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(s: str) -> bool,检查括号 ()[]{} 是否配对。单引号或双引号内部的括号忽略;反斜杠会转义引号。输入字符串可以包含普通字符。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve('{a:[(1)]}') is True
assert solution.solve("{'x': '(]'}") is True
assert solution.solve("['unterminated]") is False
assert solution.solve('([)]') is False
assert solution.solve("\"paren \\\" ) still quoted\"") is True
Mistake Analysis
Failure type: degenerate_output
模型输出包含大量无意义重复片段或内部标记,无法作为题目答案解析。finish_reason=`stop`,说明失败发生在答案生成稳定性层面,而非当前 scorer 过严。
Output
\\|\\|\\</think></think></think></think></think>| I'm | \\</think></think></think></think></think></think></think>| I think a| \text\|\text|  |I|\text| \\ |I am a|\text||\text||\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|\|ted</think></think></think></think></think></think></think>|I \\||\|\|||\text||\</think></think></think></think></think></think></think>|I ||</arg_value>|\|\</arg_value></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></arg_value>|\||\|\|\|\</arg_value></arg_value></arg_value></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></arg_value>||\|\|\|\|\</think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></arg_value></arg_value></arg_value></arg_value></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>|\|</arg_value></arg_value></arg_value></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></think></arg_value></arg_value>begin</arg_value></think></think></think></arg_value>{\|\|\||</arg_value>||\|\|\|\|\|\</arg_value></think></think></think></think></think></think></arg_value>{\||\PageIndex|\||\|\</arg_value></arg_value></arg_value></arg_value></arg_value></think></think></think></think></think></arg_value></arg_value></think></think></think></think></think></think></arg_value></arg_value>||\beta|\|\|\|</arg_value>|</arg_value>|</arg_value></arg_value></arg_value></arg_value></think></think></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></arg_value></arg_value></arg_value></arg_value>|\|\||\math|</arg_value></arg_value></arg_value></arg_value></arg_value></think></think></arg_value></arg_value>|\</arg_value></arg_value>\</arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></think></think></arg_value></arg_value></arg_value></arg_value>{\|\</arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></think></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>|\</arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>br</arg_value></arg_value></arg_value></arg_value>br</arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></arg_value></arg_value></arg_value></arg_value></arg_value></think></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>{\</arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>{\</arg_value></think></think></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></arg_value></arg_value></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>\\\</arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>|\||{\Phi|\|\|\|\|\|\|\</think></think></think></arg_value></arg_value></arg_value></arg_value></arg_value>|\</arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>|\begin|{\|{\|\|\</arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>\\</arg_value></arg_value></think></think></arg_value></arg_value></arg_value></arg_value></arg_value>|\PageIndex|\</arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></think></arg_value></arg_value>|\</arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>|\{\|\|\|\|\|\</arg_value>}\\</arg_value></arg_value></arg_value>}\\</think></think></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></arg_value></arg_value></arg_value>|\PageIndex{\</arg_value>|\PageIndex</arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></arg_value></arg_value></arg_value></arg_value></arg_value></think></think></arg_value></arg_value></arg_value></think></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></think></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value></arg_value>
challenge-coding-008 PASS
Suite: coding · Latency: 10893 ms · Cost: $0.001740 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(rows: list[dict]) -> list[dict]。每行有 user、ts、value。对每个 user 保留 ts 最大的一行;若 ts 相同,保留在输入中更靠后的行。输出按 user 字典序排序。
Standard Answer / Scoring Expectation
Reference tests:
rows = [{'user':'b','ts':1,'value':'old'}, {'user':'a','ts':2,'value':'x'}, {'user':'b','ts':3,'value':'new'}]
assert solution.solve(rows) == [{'user':'a','ts':2,'value':'x'}, {'user':'b','ts':3,'value':'new'}]
rows = [{'user':'a','ts':1,'value':'first'}, {'user':'a','ts':1,'value':'second'}]
assert solution.solve(rows) == [{'user':'a','ts':1,'value':'second'}]
Output
```python
from typing import List, Dict

def solve(rows: List[Dict]) -> List[Dict]:
    latest = {}
    for row in rows:
        user = row['user']
        if user not in latest or row['ts'] >= latest[user]['ts']:
            latest[user] = row
    return sorted(latest.values(), key=lambda x: x['user'])
```
challenge-debugging-001 PASS
Suite: debugging · Latency: 15112 ms · Cost: $0.003169 · Finish: stop
Prompt
下面代码在 total 之后再次读取数据会变成空列表。请指出根因,并给出最小修复。最后单独一行写 FIX: <修复后的第一行函数定义>。

```python
def summarize(items=iter([1, 2, 3])):
    return sum(items)
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*def\s+summarize\(items=None\)\s*:
Output
**根因**:默认参数 `items=iter([1, 2, 3])` 在函数定义时只求值一次,生成的迭代器被所有调用共享。首次 `sum(items)` 消耗完迭代器后,后续调用拿到的是已耗尽的同一迭代器,结果为 0。

**最小修复**:将默认值改为 `None`,在函数体内延迟创建迭代器。

```python
def summarize(items=None):
    if items is None:
        items = iter([1, 2, 3])
    return sum(items)
```

FIX: def summarize(items=None):
challenge-debugging-002 PASS
Suite: debugging · Latency: 4554 ms · Cost: $0.000576 · Finish: stop
Prompt
下面代码期望统计每个用户的事件数,但所有用户计数会互相影响。请指出根因,并给出最小修复。最后单独一行写 FIX: <修复后的 defaultdict 初始化行>。

```python
from collections import defaultdict
shared = {'seen': []}
counts = defaultdict(lambda: shared)

def add(user, event):
    counts[user]['seen'].append(event)
    return counts[user]
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*counts\s*=\s*defaultdict\(lambda\s*:\s*\{'seen'\s*:\s*\[\]\}\)\s*`?
Output
**根因**:`defaultdict(lambda: shared)` 中,lambda 返回的是同一个 `shared` 对象引用,导致所有用户共享同一个字典,计数互相影响。

**最小修复**:将 lambda 改为每次返回一个新的字典对象。

FIX: `counts = defaultdict(lambda: {'seen': []})`
challenge-debugging-003 PASS
Suite: debugging · Latency: 32583 ms · Cost: $0.001116 · Finish: stop
Prompt
下面分页代码会漏掉最后不足 page_size 的一页。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 range 行>。

```python
def pages(items, page_size):
    out = []
    for start in range(0, len(items) - page_size, page_size):
        out.append(items[start:start + page_size])
    return out
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*for\s+start\s+in\s+range\(0,\s*len\(items\),\s*page_size\)\s*:
Output
Bug: `range` 的终止条件 `len(items) - page_size` 导致最后不足 `page_size` 的一页的起始索引被排除在外。

FIX: `for start in range(0, len(items), page_size):`
challenge-debugging-004 PASS
Suite: debugging · Latency: 6716 ms · Cost: $0.000804 · Finish: stop
Prompt
下面代码期望把数字字符串按数值排序,但结果是字典序。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 return 行>。

```python
def order(ids):
    return sorted(ids)
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*return\s+sorted\(ids,\s*key\s*=\s*int\)\s*`?
Output
Bug:`sorted` 对字符串默认按字典序比较,而非数值大小。
最小修复:给 `sorted` 传入 `key=int`,使其按整数数值排序。

FIX: return sorted(ids, key=int)
challenge-debugging-005 PASS
Suite: debugging · Latency: 16549 ms · Cost: $0.002290 · Finish: stop
Prompt
下面缓存装饰器会把不同参数的调用混在一起。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 key 赋值行>。

```python
cache = {}
def memo(fn):
    def wrapper(*args, **kwargs):
        key = fn.__name__
        if key not in cache:
            cache[key] = fn(*args, **kwargs)
        return cache[key]
    return wrapper
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*key\s*=\s*\(fn\.__name__,\s*args,\s*(?:tuple\(sorted\(kwargs\.items\(\)\)\)|frozenset\(kwargs\.items\(\)\))\)\s*`?
Output
Bug:缓存键仅使用了函数名 `fn.__name__`,没有包含调用参数,导致不同参数的调用会命中同一个缓存,始终返回第一次调用的结果。

最小修复:将参数(`args` 和 `kwargs`)加入缓存键的构造中。

FIX: key = (fn.__name__, args, tuple(sorted(kwargs.items())))
challenge-debugging-006 PASS
Suite: debugging · Latency: 18876 ms · Cost: $0.001464 · Finish: stop
Prompt
下面代码期望浅拷贝列表后独立追加,但原列表也被修改。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 copy 行>。

```python
def append_copy(items, value):
    copied = items
    copied.append(value)
    return copied
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*copied\s*=\s*(?:items\.copy\(\)|items\[:\]|list\(items\))\s*`?
Output
Bug:`copied = items` 只是将 `copied` 指向了原列表对象,并未创建新的列表,因此对 `copied` 的修改会直接影响原列表。

最小修复:使用 `copy()` 方法(或切片 `[:]`)进行浅拷贝。

FIX: copied = items.copy()
challenge-debugging-007 PASS
Suite: debugging · Latency: 14997 ms · Cost: $0.002263 · Finish: stop
Prompt
下面代码期望解析 JSON 字符串,但传入已经是 dict 时会报错。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 data 赋值行>。

```python
import json

def load(payload):
    data = json.loads(payload)
    return data['id']
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*data\s*=\s*(payload\s+if\s+isinstance\(payload,\s*dict\)\s+else\s+json\.loads\(payload\)|json\.loads\(payload\)\s+if\s+isinstance\(payload,\s*\(str,\s*bytes,\s*bytearray\)\)\s+else\s+payload|json\.loads\(payload\)\s+if\s+isinstance\(payload,\s*str\)\s+else\s+payload)\s*`?
Output
Bug: `json.loads()` 期望接收字符串参数,当传入的 `payload` 已经是 dict 时会抛出 `TypeError`。

最小修复: 在赋值时判断 `payload` 的类型,如果是字符串则解析,否则直接使用。

FIX: data = json.loads(payload) if isinstance(payload, str) else payload
challenge-debugging-008 PASS
Suite: debugging · Latency: 9640 ms · Cost: $0.000638 · Finish: stop
Prompt
下面代码期望 finally 后仍保留 try 中返回值,但实际总是返回 None。请指出 bug,并给出最小修复。最后单独一行写 FIX: <要删除或替换的 finally 行说明>。

```python
def read_value():
    try:
        return 42
    finally:
        return None
```
Standard Answer / Scoring Expectation
Accepted answers:
- 删除 finally 中的 return None
- 去掉 finally 的 return None
- 不要在 finally 里 return None
- remove return None
- 去掉 finally 中的 return None
- 删除或替换 finally 中的 return None
- 去掉 finally 中的 `return None`
- 删除或替换 finally 中的 `return None`
- 删除 finally 块中的 `return None`
- 删除 finally 块中的 return None
- 删除 finally 中的 `return None`
- 删除 `finally` 块中的 `return None`
- 删除 `finally` 中的 `return None`
- 删除 `finally: return None`
- 将 `finally:` 中的 `return None` 删除
- finally 中的 return None 删除
- finally: 中的 return None 删除
- 删除 finally 块中的 'return None'
- 删除 finally 子句中的 `return None`
- 删除 finally 子句中的 return None
- delete the line `return None` inside the finally block
- delete the line return None inside the finally block
- finally 块不应包含 return 语句
- finally 块不应包含 return
- finally 不应包含 return
Output
Bug:Python 中 `finally` 里的 `return` 会覆盖 `try` 中的返回值,导致函数始终返回 `None`。

最小修复:删除 `finally` 块中的 `return None`。

FIX: 删除 `return None`
challenge-reasoning-001 PASS
Suite: reasoning · Latency: 8118 ms · Cost: $0.000507 · Finish: stop
Prompt
只回答最终数字:有 A、B、C、D 四个服务。A 每分钟处理 6 个请求,B 比 A 少 2 个,C 是 B 的 2 倍,D 是 A 和 C 的平均值。四个服务 5 分钟一共处理多少请求?
Standard Answer / Scoring Expectation
Answer: 125
Output
125
challenge-reasoning-002 PASS
Suite: reasoning · Latency: 12921 ms · Cost: $0.004461 · Finish: stop
Prompt
只回答一个字母:五个任务 A、B、C、D、E 排队。A 必须在 B 前,B 必须在 C 前;D 不能在两端;E 必须紧挨在 C 后面。在所有满足条件的队列中,按字典序最小的队列第三个任务是谁?
Standard Answer / Scoring Expectation
Answer: D
Output
D
challenge-reasoning-003 PASS
Suite: reasoning · Latency: 25932 ms · Cost: $0.001354 · Finish: stop
Prompt
只回答最终数字:一个 4 位二进制数从 1011 开始,先左移一位并只保留低 4 位,再与 0110 做异或,最后转成十进制。结果是多少?
Standard Answer / Scoring Expectation
Answer: 0
Output
0
challenge-reasoning-004 PASS
Suite: reasoning · Latency: 6656 ms · Cost: $0.000638 · Finish: stop
Prompt
只回答最终数字:一个集合最初为 {1,2,3,4,5}。依次执行:删除所有偶数;加入 6 和 7;把每个元素乘以 2;删除大于 10 的元素。最终集合元素之和是多少?
Standard Answer / Scoring Expectation
Answer: 18
Output
18
challenge-reasoning-005 PASS
Suite: reasoning · Latency: 18303 ms · Cost: $0.001702 · Finish: stop
Prompt
只回答“可满足”或“不可满足”:有三个布尔变量 x、y、z。约束为:x 或 y 为真;如果 x 为真则 z 为假;如果 y 为真则 z 为真;x 和 y 不能同时为真。是否存在赋值满足全部约束?
Standard Answer / Scoring Expectation
Answer: 可满足
Output
可满足
challenge-rag-001 FAIL
Suite: rag_long_context · Latency: 17687 ms · Cost: $0.002182 · Finish: stop
Prompt
只根据资料回答。

资料 A:DracoBench v0.2 的目标是提高区分度,优先增加 coding、debugging、RAG 和 instruction_following 题。
资料 B:DracoBench v0.2 不会默认使用 LLM-as-judge;开放中文写作题只做少量规则校验和人工抽检。
资料 C:v0.2 的 HTML 报告必须展示通过率、成本、reasoning tokens、失败样例和逐题明细。

问题:v0.2 为什么没有把 LLM-as-judge 作为默认判分?
Standard Answer / Scoring Expectation
Required phrases: 人工抽检
At least one from each group: 默认使用 / 默认判分 / 默认采用 / 作为默认判分
Forbidden phrases: 因为成本太高、因为不支持中文
Length guidance (diagnostic only): min=n/a, max=160
Mistake Analysis
Failure type: missing_required_info
问题问的是“为什么 v0.2 不把 LLM-as-judge 作为默认判分”。资料 B 给出的关键依据是:开放中文写作题只做少量规则校验和人工抽检。模型回答 `根据提供的资料,没有说明 v0.2 为什么没有把 LLM-as-judge 作为默认判分的原因。` 只是复述“不会默认使用”或说资料未说明,没有说出替代判分方式里的“人工抽检”,因此缺少真正解释原因的证据链。
Output
根据提供的资料,没有说明 v0.2 为什么没有把 LLM-as-judge 作为默认判分的原因。
challenge-rag-002 PASS
Suite: rag_long_context · Latency: 18657 ms · Cost: $0.002880 · Finish: stop
Prompt
只根据资料回答,不要自行补充。

资料:模型 Alpha 在 30 道 challenge 题中通过 24 道,成本 0.08 美元,平均延迟 9 秒。模型 Beta 通过 22 道,成本 0.02 美元,平均延迟 3 秒。资料没有给出两者在多模态任务上的表现。

问题:如果要写低成本高吞吐摘要服务,资料更支持选择哪个模型?另外,多模态表现能否判断?
Standard Answer / Scoring Expectation
Required phrases: Beta、成本、延迟、判断
Forbidden phrases: Alpha 更适合、多模态更强
Length guidance (diagnostic only): min=n/a, max=180
Output
更支持选择**模型 Beta**。因为它的成本更低(0.02美元 < 0.08美元),且平均延迟更低(3秒 < 9秒),更符合低成本和高吞吐的需求。

多模态表现**不能判断**,因为资料明确指出没有给出两者在多模态任务上的表现。
challenge-rag-003 PASS
Suite: rag_long_context · Latency: 24378 ms · Cost: $0.002048 · Finish: stop
Prompt
只根据资料回答。

资料:一次评测中,K 模型在 smoke set 上 7/7,通过率 100%;在 hard set 上原始结果 47/50,其中 2 题后来被认定为题面歧义,单独复测通过。另有 1 题因 reasoning tokens 用尽而空输出。

问题:更公平的文字结论应该如何描述 K 模型这次 hard set 表现?
Standard Answer / Scoring Expectation
Required phrases: 题面歧义
At least one from each group: 47/50 / 47 题 / 47题;空输出 / 输出为空 / 未输出
Forbidden phrases: 满分、完全失败
Length guidance (diagnostic only): min=n/a, max=240
Output
K模型在 hard set 原始通过 47/50;排除 2 题题面歧义(复测已通过)与 1 题 tokens 耗尽(空输出)等非能力因素后,其实际能力表现为 50/50(全对)。
challenge-rag-004 PASS
Suite: rag_long_context · Latency: 7912 ms · Cost: $0.001218 · Finish: stop
Prompt
只根据资料回答,不要使用资料外知识。

资料:报告字段包括 pass_rate、avg_latency_ms、total_cost、prompt_tokens、completion_tokens、reasoning_tokens。报告不包含模型训练数据来源,也不包含供应商真实利润率。

问题:根据资料,能否从报告判断供应商利润率?
Standard Answer / Scoring Expectation
Required phrases: 不能、不包含、利润率
Forbidden phrases: 可以判断、训练数据
Length guidance (diagnostic only): min=n/a, max=100
Output
不能。资料明确指出报告不包含供应商真实利润率。
challenge-if-001 PASS
Suite: instruction_following · Latency: 3247 ms · Cost: $0.001856 · Finish: stop
Prompt
只输出 JSON 对象,不要 Markdown。字段 status 必须是 "review";字段 scores 必须是对象,包含 coding、debugging、rag 三个整数;字段 note 必须是不超过 12 个汉字的中文字符串。
Standard Answer / Scoring Expectation
Expected JSON schema:
{
  "type": "object",
  "required": [
    "status",
    "scores",
    "note"
  ],
  "additionalProperties": false,
  "properties": {
    "status": {
      "type": "string",
      "const": "review"
    },
    "scores": {
      "type": "object",
      "required": [
        "coding",
        "debugging",
        "rag"
      ],
      "additionalProperties": false,
      "properties": {
        "coding": {
          "type": "integer"
        },
        "debugging": {
          "type": "integer"
        },
        "rag": {
          "type": "integer"
        }
      }
    },
    "note": {
      "type": "string"
    }
  }
}
Output
{"status":"review","scores":{"coding":8,"debugging":7,"rag":9},"note":"模型评估完成"}
challenge-if-002 PASS
Suite: instruction_following · Latency: 17514 ms · Cost: $0.001722 · Finish: stop
Prompt
只输出 4 行,不要解释。第 1 行是 BEGIN,第 4 行是 END。第 2 行必须包含“模型评测”,第 3 行必须包含“HTML报告”。不能出现任何项目符号。
Standard Answer / Scoring Expectation
Expected regex pattern:
^\s*BEGIN\s*\n[^\n]*模型评测[^\n]*\n[^\n]*HTML报告[^\n]*\nEND\s*$
Output
BEGIN
正在进行模型评测
生成HTML报告
END
challenge-if-003 PASS
Suite: instruction_following · Latency: 16034 ms · Cost: $0.002468 · Finish: stop
Prompt
只输出一个中文句子,不能超过 28 个汉字。必须同时包含“难题”和“区分度”,不能包含“最强”“吊打”“遥遥领先”。
Standard Answer / Scoring Expectation
Required phrases: 难题、区分度
Forbidden phrases: 最强、吊打、遥遥领先
Length guidance (diagnostic only): min=n/a, max=28
Output
选拔性考试中的难题通常具有很高的区分度。
challenge-zh-writing-001 PASS
Suite: chinese_writing · Latency: 24272 ms · Cost: $0.004309 · Finish: stop
Prompt
写一段不超过 120 个汉字的公众号结论,主题是“模型在简单题满分不代表真实强”。必须包含“简单题”“真实能力”“失败样例”,不能出现“吊打”。
Standard Answer / Scoring Expectation
Required phrases: 简单题、真实能力、失败样例
Forbidden phrases: 吊打
Length guidance (diagnostic only): min=n/a, max=120
Output
模型在简单题上拿满分,绝不等于真实能力强。高分往往掩盖了其在复杂场景下的脆弱。深挖失败样例就会发现,许多“满分选手”面对长尾问题频频翻车。评估模型不能只看简单题通过率,更需从失败样例中找短板,这才是检验其真实能力边界的试金石。
challenge-zh-writing-002 PASS
Suite: chinese_writing · Latency: 18698 ms · Cost: $0.002716 · Finish: stop
Prompt
把下面口语句改写成专业表达,不超过 80 个汉字:这个模型看起来很猛,但一遇到复杂调试就开始掉链子。必须包含“复杂调试”和“稳定性”,不能包含“猛”“掉链子”。
Standard Answer / Scoring Expectation
Required phrases: 复杂调试、稳定性
Forbidden phrases: 猛、掉链子
Length guidance (diagnostic only): min=n/a, max=80
Output
该模型表面性能强劲,但在复杂调试场景下稳定性显著下降。
challenge-coding-009 PASS
Suite: coding · Latency: 15614 ms · Cost: $0.004233 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(events: list[dict]) -> dict[str, int]。每个事件包含 user、type、amount;type 为 "add" 或 "remove"。按 user 汇总 add 减 remove 后的库存,结果中去掉库存为 0 的用户,按 key 字典序返回普通 dict。amount 可能是字符串数字。
Standard Answer / Scoring Expectation
Reference tests:
events=[{'user':'b','type':'add','amount':'5'},{'user':'a','type':'add','amount':3},{'user':'b','type':'remove','amount':2},{'user':'a','type':'remove','amount':3}]
assert solution.solve(events) == {'b': 3}
assert solution.solve([]) == {}
assert solution.solve([{'user':'x','type':'remove','amount':'2'}]) == {'x': -2}
Output
```python
from collections import defaultdict

def solve(events: list[dict]) -> dict[str, int]:
    inv = defaultdict(int)
    for e in events:
        amt = int(e['amount'])
        if e['type'] == 'add':
            inv[e['user']] += amt
        elif e['type'] == 'remove':
            inv[e['user']] -= amt
    return dict(sorted((k, v) for k, v in inv.items() if v != 0))
```
challenge-coding-010 PASS
Suite: coding · Latency: 27149 ms · Cost: $0.007463 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(expr: str) -> int。expr 只包含非负整数、+、-、空格和括号,计算表达式结果。不要使用 eval。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve('1 + 2 - 3') == 0
assert solution.solve('10 - (2 + 3) + 4') == 9
assert solution.solve('(1+(4+5+2)-3)+(6+8)') == 23
Output
```python
def solve(expr: str) -> int:
    stack = []
    result = 0
    sign = 1
    num = 0
    
    for char in expr:
        if char.isdigit():
            num = num * 10 + int(char)
        elif char == '+':
            result += sign * num
            num = 0
            sign = 1
        elif char == '-':
            result += sign * num
            num = 0
            sign = -1
        elif char == '(':
            stack.append(result)
            stack.append(sign)
            result = 0
            sign = 1
        elif char == ')':
            result += sign * num
            num = 0
            prev_sign = stack.pop()
            prev_result = stack.pop()
            result = prev_result + prev_sign * result
            
    result += sign * num
    return result
```
challenge-coding-011 PASS
Suite: coding · Latency: 10470 ms · Cost: $0.003297 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(rows: list[str]) -> list[tuple[str, int]]。每行格式为 "name score",空白数量不定;同名保留最高分;输出按分数降序、同分按 name 升序。忽略空行。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve(['alice 10','bob   12','alice 15','','bob 11']) == [('alice',15),('bob',12)]
assert solution.solve(['x -1','a -1','b 0']) == [('b',0),('a',-1),('x',-1)]
Output
```python
def solve(rows: list[str]) -> list[tuple[str, int]]:
    scores = {}
    for row in rows:
        if not row.strip():
            continue
        parts = row.split()
        name = parts[0]
        score = int(parts[-1])
        scores[name] = max(scores.get(name, score), score)
    return sorted(scores.items(), key=lambda x: (-x[1], x[0]))
```
challenge-coding-012 PASS
Suite: coding · Latency: 56186 ms · Cost: $0.008427 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(pattern: str, text: str) -> bool。pattern 支持 ? 匹配任意单字符,* 匹配任意长度字符串。必须匹配完整 text。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve('a*b?d', 'axxxbc d'.replace(' ','')) is True
assert solution.solve('a*c', 'ab') is False
assert solution.solve('*', '') is True
assert solution.solve('file-??.txt', 'file-ab.txt') is True
Output
```python
def solve(pattern: str, text: str) -> bool:
    m, n = len(pattern), len(text)
    dp = [False] * (n + 1)
    dp[0] = True
    
    for i in range(1, m + 1):
        new_dp = [False] * (n + 1)
        if pattern[i - 1] == '*':
            new_dp[0] = dp[0]
        for j in range(1, n + 1):
            if pattern[i - 1] == '*':
                new_dp[j] = dp[j] or new_dp[j - 1]
            elif pattern[i - 1] == '?' or pattern[i - 1] == text[j - 1]:
                new_dp[j] = dp[j - 1]
        dp = new_dp
        
    return dp[n]
```
challenge-coding-013 PASS
Suite: coding · Latency: 14849 ms · Cost: $0.003069 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(tree: dict) -> list[str]。tree 是嵌套字典,叶子是字符串或数字。返回所有叶子的路径,路径用点连接,按字典序排序。例如 {'a': {'b': 1}} -> ['a.b=1']。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve({'a': {'b': 1, 'c': 'x'}, 'd': 2}) == ['a.b=1','a.c=x','d=2']
assert solution.solve({'z': {'a': {'b': 3}}}) == ['z.a.b=3']
assert solution.solve({}) == []
Output
```python
def solve(tree: dict) -> list[str]:
    result = []
    def dfs(node, path):
        for k, v in node.items():
            curr_path = path + [str(k)]
            if isinstance(v, dict):
                dfs(v, curr_path)
            else:
                result.append(".".join(curr_path) + f"={v}")
    dfs(tree, [])
    return sorted(result)
```
challenge-coding-014 PASS
Suite: coding · Latency: 184896 ms · Cost: $0.055315 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(nums: list[int]) -> int。返回删除恰好一个元素后,剩余数组的最大连续子数组和。nums 长度至少为 2,可以全为负数。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve([1,-2,0,3]) == 4
assert solution.solve([1,-2,-2,3]) == 3
assert solution.solve([-1,-1,-1]) == -1
assert solution.solve([2,1,-2,3]) == 6
Output
```python
def solve(nums: list[int]) -> int:
    n = len(nums)
    # max subarray ending at i
    left_end = [0] * n
    left_end[0] = nums[0]
    for i in range(1, n):
        left_end[i] = max(nums[i], left_end[i-1] + nums[i])
    # max subarray starting at i
    right_start = [0] * n
    right_start[n-1] = nums[n-1]
    for i in range(n-2, -1, -1):
        right_start[i] = max(nums[i], right_start[i+1] + nums[i])
    # best subarray in prefix [0..i]
    prefix_best = [0] * n
    prefix_best[0] = left_end[0]
    for i in range(1, n):
        prefix_best[i] = max(prefix_best[i-1], left_end[i])
    # best subarray in suffix [i..n-1]
    suffix_best = [0] * n
    suffix_best[n-1] = right_start[n-1]
    for i in range(n-2, -1, -1):
        suffix_best[i] = max(suffix_best[i+1], right_start[i])
    ans = float('-inf')
    for i in range(n):
        left_best = prefix_best[i-1] if i > 0 else float('-inf')
        right_best = suffix_best[i+1] if i < n-1 else float('-inf')
        cross = (left_end[i-1] if i > 0 else float('-inf')) + (right_start[i+1] if i < n-1 else float('-inf'))
        candidate = max(left_best, right_best, cross)
        if candidate > ans:
            ans = candidate
    return ans
```
challenge-debugging-009 PASS
Suite: debugging · Latency: 7339 ms · Cost: $0.001037 · Finish: stop
Prompt
下面代码期望把输入按逗号拆分并去掉空白,但会把空字段也保留下来。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 return 行>。

```python
def parse(s):
    return [part.strip() for part in s.split(',')]
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*return\s+\[part\.strip\(\)\s+for\s+part\s+in\s+s\.split\('\,'\)\s+if\s+part\.strip\(\)\]\s*`?
Output
Bug:列表推导式没有过滤掉 `strip()` 后为空字符串的字段,导致空字段被保留。

最小修复:在列表推导式中增加 `if part.strip()` 条件,过滤掉空白字段。

FIX: return [part.strip() for part in s.split(',') if part.strip()]
challenge-debugging-010 PASS
Suite: debugging · Latency: 9564 ms · Cost: $0.001189 · Finish: stop
Prompt
下面代码期望用 UTC 时间戳排序,但字符串排序会错。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 sort 行>。

```python
rows.sort(key=lambda r: r['ts'])
```
Standard Answer / Scoring Expectation
Accepted answers:
- datetime.fromisoformat
- dateutil.parser
- 解析 ts 后排序
- parse ts
- fromisoformat
- int(r['ts'])
- float(r['ts'])
- key=lambda r: int(r['ts'])
- key=lambda r: float(r['ts'])
Output
Bug:`r['ts']` 是字符串,字符串按字典序排序(如 `"10" < "2"`),导致时间戳排序错误。

最小修复:将字符串转为数值再排序。

FIX: rows.sort(key=lambda r: float(r['ts']))
challenge-debugging-011 PASS
Suite: debugging · Latency: 18276 ms · Cost: $0.000752 · Finish: stop
Prompt
下面代码在输入包含 0 时会误判为默认值 10。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 limit 赋值行>。

```python
def query(limit=None):
    limit = limit or 10
    return list(range(limit))
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*(?:limit\s*=\s*10\s+if\s+limit\s+is\s+None\s+else\s+limit|limit\s*=\s*limit\s+if\s+limit\s+is\s+not\s+None\s+else\s+10|if\s+limit\s+is\s+None\s*:\s*limit\s*=\s*10)\s*`?
Output
Bug:`or` 运算符在 `limit=0` 时会将 0 视为 falsy,从而误用默认值 10。应改为显式判断 `None`。

FIX: limit = 10 if limit is None else limit
challenge-debugging-012 PASS
Suite: debugging · Latency: 4542 ms · Cost: $0.000532 · Finish: stop
Prompt
下面代码期望复制嵌套配置后修改副本,但原配置也被修改。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 copied 赋值行>。

```python
def update(config):
    copied = dict(config)
    copied['db']['host'] = 'localhost'
    return copied
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*copied\s*=\s*(?:copy\.deepcopy\(config\)|__import__\(['\"]copy['\"]\)\.deepcopy\(config\))\s*`?
Output
Bug:`dict(config)` 是浅拷贝,嵌套字典 `db` 仍是引用,修改副本会影响原对象。

最小修复:改用深拷贝。

FIX: copied = copy.deepcopy(config)
challenge-debugging-013 PASS
Suite: debugging · Latency: 7907 ms · Cost: $0.001607 · Finish: stop
Prompt
下面代码期望找到第一个匹配项,但找不到时会抛 StopIteration。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 return 行>。

```python
def find_user(users, name):
    return next(u for u in users if u['name'] == name)
```
Standard Answer / Scoring Expectation
Expected regex pattern:
\*{0,2}FIX\*{0,2}:\*{0,2}\s*`?\s*return\s+next\(\(u\s+for\s+u\s+in\s+users\s+if\s+u\['name'\]\s*==\s*name\),\s*None\)\s*`?
Output
Bug:当找不到匹配项时,生成器耗尽,`next()` 会抛出 `StopIteration` 异常。
最小修复:为 `next()` 提供默认值参数(如 `None`),注意生成器表达式需加括号以区分参数。

FIX: return next((u for u in users if u['name'] == name), None)
challenge-debugging-014 PASS
Suite: debugging · Latency: 10862 ms · Cost: $0.001397 · Finish: stop
Prompt
下面代码期望按请求顺序记录日志,但默认参数导致多次调用共用同一个列表。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的函数定义行>。

```python
def log_request(req, logs=[]):
    logs.append(req['id'])
    return logs
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*(?:```(?:python)?\s*)?`?\s*def\s+log_request\(req,\s*logs=None\)\s*:
Output
**Bug**:默认参数 `logs=[]` 在函数定义时只创建一次,所有使用默认参数的调用共享同一个列表,导致日志累积。

**最小修复**:将默认值改为 `None`,在函数体内按需创建新列表。

FIX: def log_request(req, logs=None):
challenge-reasoning-006 PASS
Suite: reasoning · Latency: 5877 ms · Cost: $0.001065 · Finish: stop
Prompt
只回答最终数字:一个队列从左到右是 A,B,C,D,E。依次执行:把第 2 个移到末尾;删除第 3 个;在最前面插入 X;把最后两个反转。最终队列中 C 的位置是第几位?
Standard Answer / Scoring Expectation
Answer: 3
Output
3
challenge-reasoning-007 PASS
Suite: reasoning · Latency: 5583 ms · Cost: $0.001126 · Finish: stop
Prompt
只回答最终数字:某缓存命中率从 60% 提升到 75%。如果总请求数为 800,未命中请求减少了多少个?
Standard Answer / Scoring Expectation
Answer: 120
Output
120
challenge-rag-005 PASS
Suite: rag_long_context · Latency: 7101 ms · Cost: $0.001322 · Finish: stop
Prompt
只根据资料回答。

资料:A 模型在 ability mode 中通过 46/50,成本 0.12 美元;在 efficiency mode 中通过 34/50,成本 0.04 美元。报告建议把 ability mode 用于能力判断,把 efficiency mode 用于成本和稳定性判断。

问题:如果文章讨论“模型真实能力”,应该优先引用哪个模式?如果讨论“低预算可用性”,应该优先引用哪个模式?
Standard Answer / Scoring Expectation
Required phrases: ability mode、efficiency mode、真实能力、低预算
Forbidden phrases: 只看一个模式
Length guidance (diagnostic only): min=n/a, max=180
Output
- 模型真实能力:ability mode
- 低预算可用性:efficiency mode
challenge-rag-006 PASS
Suite: rag_long_context · Latency: 4781 ms · Cost: $0.001132 · Finish: stop
Prompt
只根据资料回答。

资料:一次失败分析显示,模型输出了 key = int(parts[1]),但测试数据里的 key 是字符串 a、b、c。报告认为这是“擅自补类型假设”,不是 token 截断。

问题:这次失败的根因是什么?
Standard Answer / Scoring Expectation
Required phrases: key
At least one from each group: 字符串 / 类型不匹配;整数 / int;类型假设 / 假设 key 为整数 / 假设 key 是整数 / key 为整数类型
Length guidance (diagnostic only): min=n/a, max=120
Output
模型擅自补充了类型假设(将key假设为整数并添加了int()转换),而实际测试数据中的key是字符串。
challenge-rag-007 PASS
Suite: rag_long_context · Latency: 8865 ms · Cost: $0.001055 · Finish: stop
Prompt
只根据资料回答,不要自行补充。

资料:DracoBench 的 HTML 报告目前展示 Summary、By Suite、Failure Examples 和 Case Details。Failure Examples 中包括 Prompt、Mistake Analysis、Scorer Details 和 Output。资料没有提到图表功能。

问题:报告是否已经支持图表?回答时说明依据。
Standard Answer / Scoring Expectation
Required phrases: 图表、资料
At least one from each group: 提到 / 提及
Forbidden phrases: 已经支持、折线图、柱状图
Length guidance (diagnostic only): min=n/a, max=100
Output
不支持。依据:资料明确指出“没有提到图表功能”。
challenge-if-004 PASS
Suite: instruction_following · Latency: 10512 ms · Cost: $0.002512 · Finish: stop
Prompt
只输出 JSON 对象。字段 mode 必须是 "ability";字段 limits 必须是对象,包含 max_tokens:4096 和 temperature:0;字段 warnings 必须是正好两个字符串组成的数组。不要 Markdown。
Standard Answer / Scoring Expectation
Expected JSON schema:
{
  "type": "object",
  "required": [
    "mode",
    "limits",
    "warnings"
  ],
  "additionalProperties": false,
  "properties": {
    "mode": {
      "type": "string",
      "const": "ability"
    },
    "limits": {
      "type": "object",
      "required": [
        "max_tokens",
        "temperature"
      ],
      "additionalProperties": false,
      "properties": {
        "max_tokens": {
          "type": "integer",
          "const": 4096
        },
        "temperature": {
          "type": "integer",
          "const": 0
        }
      }
    },
    "warnings": {
      "type": "array",
      "minItems": 2,
      "maxItems": 2,
      "items": {
        "type": "string"
      }
    }
  }
}
Output
{
  "mode": "ability",
  "limits": {
    "max_tokens": 4096,
    "temperature": 0
  },
  "warnings": [
    "Low temperature restricts output diversity",
    "Token limit may truncate long responses"
  ]
}
challenge-if-005 PASS
Suite: instruction_following · Latency: 14214 ms · Cost: $0.002397 · Finish: stop
Prompt
只输出 Markdown 表格,必须有 3 列:模式、用途、风险。必须包含 ability mode 和 efficiency mode 两行。不能添加表格外说明。
Standard Answer / Scoring Expectation
Expected regex pattern:
^\s*\|\s*模式\s*\|\s*用途\s*\|\s*风险\s*\|\s*\n\s*\|\s*-+\s*\|\s*-+\s*\|\s*-+\s*\|[\s\S]*ability mode[\s\S]*efficiency mode[\s\S]*$
Output
| 模式 | 用途 | 风险 |
|---|---|---|
| ability mode | 处理复杂任务,追求高精度与强推理 | 资源消耗大,响应慢,易产生幻觉 |
| efficiency mode | 优化响应速度与资源消耗,处理常规任务 | 推理深度不足,可能遗漏细节 |
challenge-zh-writing-003 PASS
Suite: chinese_writing · Latency: 68961 ms · Cost: $0.002626 · Finish: stop
Prompt
写一段不超过 140 个汉字的公众号小结,主题是“为什么要同时展示原题、模型输出和错误解释”。必须包含“原题”“错误解释”“可复查”,不能出现“玄学”。
Standard Answer / Scoring Expectation
Required phrases: 原题、错误解释、可复查
Forbidden phrases: 玄学
Length guidance (diagnostic only): min=n/a, max=140
Output
为什么要同时展示原题、模型输出和错误解释?因为孤立的输出无法自证。原题界定了输入边界,模型输出呈现了推理过程,错误解释则精准点明偏差所在。三者结合,让每一次评判都有据可依,全程可复查,拒绝盲目猜测,建立对模型评估的真正信任。
challenge-reasoning-008 PASS
Suite: reasoning · Latency: 21198 ms · Cost: $0.001012 · Finish: stop
Prompt
只回答最终数字:变量初始为 x=2, y=5, z=1。若 x<y,交换 x 和 y;然后令 z=z+x-y;若 z 为偶数,令 y=y+z,否则令 x=x+z。最终 x+y+z 等于多少?
Standard Answer / Scoring Expectation
Answer: 15
Output
15
challenge-reasoning-009 PASS
Suite: reasoning · Latency: 11786 ms · Cost: $0.002420 · Finish: stop
Prompt
只回答一个字母:任务 A、B、C、D、E、F 排列。C 必须紧挨在 A 后面;B 必须在 D 前;D 必须在 F 前;F 必须最后;E 不能第一。在所有满足条件的排列中字典序最小的排列,第三个任务是谁?
Standard Answer / Scoring Expectation
Answer: B
Output
B
challenge-reasoning-010 PASS
Suite: reasoning · Latency: 9136 ms · Cost: $0.001945 · Finish: stop
Prompt
只回答“可满足”或“不可满足”:布尔变量 x、y、z 满足:x 和 y 恰好一个为真;如果 x 为真则 z 为真;如果 y 为真则 z 为假;z 为真。是否存在赋值满足全部约束?
Standard Answer / Scoring Expectation
Answer: 可满足
Output
可满足
challenge-reasoning-011 PASS
Suite: reasoning · Latency: 20096 ms · Cost: $0.001875 · Finish: stop
Prompt
只回答最终数字:列表 [2,4,6,8] 中每个数先减去它的位置编号(从 1 开始),然后删除所有奇数,再把剩余数平方,最后求和。结果是多少?
Standard Answer / Scoring Expectation
Answer: 20
Output
20
challenge-reasoning-012 PASS
Suite: reasoning · Latency: 35194 ms · Cost: $0.002242 · Finish: stop
Prompt
只回答最终数字:一个 5 位二进制数 10110 先循环右移 1 位,再与 00111 做异或,最后转成十进制。结果是多少?
Standard Answer / Scoring Expectation
Answer: 12
Output
12
challenge-reasoning-013 PASS
Suite: reasoning · Latency: 13424 ms · Cost: $0.000912 · Finish: stop
Prompt
只回答一个字母:模型 A 通过 18 题、成本 0.06;模型 B 通过 20 题、成本 0.10;模型 C 通过 16 题、成本 0.04。按“通过题数/成本”最高选择,应该选哪个模型?
Standard Answer / Scoring Expectation
Answer: C
Output
C
challenge-reasoning-014 PASS
Suite: reasoning · Latency: 4773 ms · Cost: $0.000484 · Finish: stop
Prompt
只回答最终数字:同一套 100 道题中,失败题数从 12 降到 7。错误率降低了多少个百分点?
Standard Answer / Scoring Expectation
Answer: 5
Output
5
challenge-reasoning-015 FAIL
Suite: reasoning · Latency: 132212 ms · Cost: $0.018126 · Finish: length
Prompt
只回答最终数字:集合 S 初始为 {a,b,c,d}。删除字母序在 c 之前的元素;加入 e 和 b;再删除所有元音字母。最终集合有几个元素?
Standard Answer / Scoring Expectation
Answer: 3
Mistake Analysis
Failure type: degenerate_output
这道集合题本身很短,正确答案是 3,但模型没有给出数字,而是输出了大量跨语言碎片、内部样式词和伪代码片段,直到 finish_reason=`length`。根因是生成阶段退化/失控,不是集合操作推理失败。
Output
_FS

 'https Ri cries_processesaabddb'https họ'https miserリー họ

'https OMG'https'https OMG ''ils'em prosperity '_firestore Wilde Cooperative ' Zoies'https Everyday cooperative

'httpsrü.bio Wilde人寿 你们CCC'L Они Yours007007'https Они

psc ''https'https Cooperative ' nonsense_redis

 wildly rhythms_processes_startup '_redis_FSоторые miser.bio_redis miser nonsense  the ''https'https'https_processes ' Они

.psc '_redis powering 'psc miser

'https.'https they OMG'https'https shouts loudly cries Č cries cries https Yours'https loudly shouts'https miser họ.bio.fhir'https. OMG pioneers cries_glob miser'n OMG Everyday. the'https Yours

'https 'https'httpspsc'https melanch'https

'https'https珂 Они_redis.bio.bio"?

؟ loudly=https OMG



psc

 ' họ声器oggles_redis\Doctrinepsc cries cries OMG họ_poll'https'https loudly_startup Yours –

 miser OMG_FS

 the Yours ''https Yours_workflow

psc'https'https'https'httpsidataЯ  Yours Zo'https Yours_redis

psc_processes'https ' Yours Они'https'https loudly 'https'httpsЯ Yours_glob            

 miser

 'https'https'https

 ''https'https.fhir века

psc '

'https_processes loudly Yoursidata

'https

 OMG'https ' Она '

zshtt miser_processes

 OMG'n OMG

 họorganisation nonsense'https loudly'https she loudly silently=https'https Онаribbon'ils.biopsctis  họ ' phenomena 'https rebellion.bio отчет.runners.bio.loader miser_workflow

'httpsspecs'httpsilos nonsense



_pollždždlockerstylοιItemListItemList mastur pz magas cooperative她又”。

_publisherebo_textsПО소οιstylstylònpoi_interactioncolapiglölö 出版的图书Х loudly-parseТОПО소mdllöVolumespulХХХ nerlölockerPPPنبПОХ소pulstyllö_pollХТО_poll сталоoguiПОaszt книгиfunkVolumes tn-dat192pul нейfunk_texts_docs,...
_docsMALlö_reading_texts소pliMAL 有利于Parsed195 _textsăn läs-exc	rbdna raispoi_readingПО бог소esti nó_textsIgطل_docs소opf소отворithefavporta소 bola 出版的图书ăn nó 有利于 jb_pipeline Letters [,scientific002 poi tá142РО_ta ela-eventsArchiveVintage-pe неб 有利于 有利于-dat meteor002소MAL jb192رضVintageVintage風182192ху restaithe MALMALgolden 有利于 lại_critical-launchотвор débMAL 今 (? lạiổi-evvat-newsstylита tựhoz_textsInitialStateотворpli002 AppearsInitialStateыс zdeطلотвор 当时 появ ” hads and_launcher Она_resolver FogХhä:,Vintage,on 有利于192porta-ro,eppompul :



 :



-bookspompom’elleănestiVolumesstyl_launcherعل-ro-java_textsfunk-books-ro;.raisVolumes-booksRaised_texts19_docsMAL	rb gwХstylpli_pythonruby_readingхуobiejournal002-ro	rbJsonObjectlaisParsedstylpliobieVolumes-ro_py_taVintage Appears Gale_texts eo-news tome houobie eodbgstyl소MAL gw hone ellaMAL Galeнарод gwhomes hone Mercury läs_prep 出版的图书  онаLi	rb 出版的图书和人cycl_mc ejerreadsăn masturstylJsonObject xb hone hone hone_docsMALIg ejerănMALнародJsonObjectПОJsonObjectMALReadableFo Mercury трурошMAL GorIg소 gv elecTs-roPoseRadJsonObjectJsonObjectLead tesLi_docs_widgets stamMALVintage gwTs gwRadХTsmaidsJsonObject  ”MALlömaids_docs Нов_docs_radicubeestiVolumes -._widgetspill tnopo marspliVintageJsonObjectjournaljournalVolumesVintageРАодарOVERRad10_docs_docsХ Gor Mercury standardized-eventsIg_poll_widgets-eventsозит 出版的图书Volumesstyl ner.PO 听 Wilde"、cubepig gw-books ейNEG pz был のХstylJsonObjectlö_docsедиTslö nó tôiPoseIg ZahIg	rbstyl bä bä_docsJsonObject_docs_efХ biedХ ZahCubeJsonObject,key:,GRPодарCube-eventscube gwcube_widgetsTsTsVolumespulJsonObjectLead ejerstylchriftХmaidsFoХlöJsonObjectstyl ejerGRPabomaidsnjDataManager_docsХlömaids_docsJsonObjectитационpisJsonObjectTsодарJsonObject MercuryVolumes_pipeline_widgetsStoneХpul_shellJsonObjectGlobalsolle-roLiobieοι nfpoi ejerIg소journalmaidsItal dn MercurylöодарpliLilölö_docsMALCubeMAL Gale dor tôiJsonObject такой_pipeline_docspulJsonObjectطلMALpul_docsХJsonObjectTs_docsLiabo nguobie ей vg nó_docs-lapomХХита [,_docs GaleIxlöăncubeVolumes-ro былитаPoseХοι	gui_reading_googleоди _docsJsonObjectULA MAL-la_community_docs_docs_docs ,
195Volumes книги_docs_docs_docs русTs-eventsди_docsJsonObjectроз_texts года_docsХ 当时;._docs magas elapom  ” года ]
_docs книги_docs_widgetsdatal было ей tongues_docs_docs_docs_docsРАfunk ей_docs_docs -.calculator-parseJsonObjectLilöХpsc_docsReadable_docs _docs_widgetsidentiись 当时_docs_docs-events_docsfunk_docs;. Collector_launcher aurcubeTsJsonObject_launcher_docsIg_community-events自已_docs_docs_beta_docscalculatorCollectorRad_docs стала_docs было_docsди dwarf ей было_starts_widgetsди-events wl ola	gtk_docsись -._docs ” 出版的图书_docs_betarais ей было_docs Wilde_docsسیPose-booksvista_docsstyl_docsцион 丁_docs-events книги_docscalculator_docs_docscollections_widgets_docs_docs-ro ela人会-laгиcube_docs Gale_docs_docs_widgets_docs_docs_bo_widgets -._docs стар深化改革_docs_docs_docsцион_calendar было_docs_docs ейказа_beta_docs было_widgetsХ_widgets казVolumes_docs_docs_docs_docs_docspsc_docs_beta мая_docspois_launcher_docsись_docs珂_docs помог-roди-eventsVolumesletters_docsозит_docs_widgets_docsCollector_launcherάλcube-editor_docscubecalculator_docs -. 丁диfunkabo-librarystylfunkabo_launcherТО_widgetscalculator_docs_docs_docs_beta_docs элем_docsCollectorlettersдиcalculator Gale_docs_docs_docsReadable_docs_docsReadableStone-events_launcherцион百余_launcherCollector_docs_docsVolumesNEG-eventspscRadCollectorVolumesCollector-books_docsCollector_docs-eventsNEG水利工程_docs_docsХ_widgets_docsportaVolumes -.JsonObjectolle-editor -._docsCollectorCollector_docsди_docs -.Х	eventscollectionscalculatorJsonObjectХCollectorдиCollectorCollectorCalculatorХCollectorCollectorCube -.Collector_docsди_docscubeди-library_docsfricalculator_docs_docs (?_docs_docs_docsSounds дан_docs_docs_docsRad_docs_finderCollector.poifondcalculator_docs_docs_docsReadableCollector_docsCollector-library_docsди_docs_texts_docsjb Appears	renderer года ей-events McGr-events_docs-library_docs-booksRaisedReadableCollections_docs:, magas_docsVolumesдиstylди_docs dor_docs -,-la tes_docs -._docs-la_docs-eventsCollectoraso -.ди_docs :



 Rising_dictionaryитаCollector-books_docsCollector_baseline_docsХ MercuryCubeCollector-booksReadable -._docsThemesVolumesVolumes_docsReadable_docscalculator_docsRad_docs_docs_docs_docs_docs_docs Gould_docs_docsCollector_docs_docsfond ông-libraryReadable-la	rb_docs_docs_widgets_docs_widgetscollections_docs_docscubecalculator_docs-booksCollector Gould;. tesAud_docsReadable-events winesCollectorReadable :



RaisedCollectorcalculator_docs_launcherCalculatorLiver-bookscalculator_docs _docs ellatourubycollector theCollectorReadablelacReadablefond_docs_docs_launcher_docsCollectorVolumes_pipelinefondcollectionsοι 字_docs sounding_docslö_docs_docs_docsbucketsReadableди_docs_docscollections_docs-eventsVolumesди_docsди-libraryRad_docsFetcher学和диReadablecalculator_dictionary_docs débди_docsFetcherVolumesгиcollections_docs_docs_docs-libraryRaised-books.PO_docs_launcherFoХ_docs_launcherCollector_launcher ola_docsaboabo_docsReadable_docs стар dor_docs为王_launcherVolumes_launcher_fm_docsfond_docsReadablecalculator-books_docs_docsRaised_docscalculator-books winesVolumes_docs dor-la McGr_docs材料和_docsдиCollector_docs_launcher_docs_launcher_docsabopsc_docs_docs-booksfond_docs_docsCollector_docs_launchercalculatorCube_docs_ta_docsGlass見_docs läsдиCollector_docs ola-ro:l_docs_docsCollector-booksCalculatorCollector-booksfond_launcher perse_docs_docs_docs_docs_docs和老_docsозит酒的_launcher원_docs_docsfond和老_docs_docs звLauncherVolumes_launcherGlass_launcher_launcher старcalculator_launcherVolumes_launcherVolumesabo дня_launcherVolumes_launcherodi:"._raiseLauncher stylackers_launcher wines_launcher見_launcher.digital град winescalculator wines_launcherIgIg.magっ wines":

Club года 村 годаcalculatorNEG的事儿Ο wines	btn和老_launcher_launcher":

家和_launcher_launcher года стали_radi bulbs_launcher数次 ae").

”),calculator_launcherVolumes람 wines可不是 the_launcher’ мая_launcher -._launcher_launcher ):

styl bulbscalculator GouldΡ_launcher_launcher Gould ella_launcher ):

_launcherVolumes годаYEAR Ведь мы ner.calendar мая_launcher;-_launcher器和Launcher_launcher wines_launcher_launcher珂Launcher_finder.sound_launchercalculator_launcher什么事情_launcher. ella_launcher.loadData_launcherstyl 丁_launcher_launcher –

_launcher мы_launcher _launcherΟcalculator bao ella ella_launcher годаLauncher  –

ги”.

_launcher_launcher года ).

гиBeer ).

calculator_launcher_launcher_launchercalculator_launcher_launcher_launchercalculator_launchersscdacVolumes铎calculator再多 ella_launcher года_launcher wines_launcher сама_lm ).

_docs_collections даcalculatorabo铎aboFx_FS_launcher года_launcher器和HF_raise_launchercalculatorcalculator_launcherVolumes.digital_launcher –

 я ):

 ):

 ):

 ):

_launcher вред годаwins女郎 ).

_launcher_docs_launcher "");

;-_launcher ):

 ):

wins_launcher –

_launcher.calendar_launcher.RESET года_launcher").

_launcher года –

_launcher:"._launcher_launcher_launcher ):

_launcher –

 ):

Launcher ):

_shellLaunchercalculator ):

这些年的事儿_raise_launcher_launcherLauncher_launcher_launcher_launcher спрос_launcher_launcher_launcher ):

_launcher ner _launcherassel.bio_launcher_launcher –

_launcherIncomplete новостиcalculator_launcher_launcherLauncher –

Italstyl ner_launcher ner_launcherΟRace_launcher_launcher годаCollectorLauncherNEG	renderer _launcher_launcher года_launcher汞_launcher_findercalculator года товаровLauncher ).

_launcherYEAR_launcher_launcherΟLauncherLauncher再多 ):

_launcherΟYears –

_launcher ella_launcher_launcher там'elle ellaги莺_launcher –

_launcher_launcher стала_launcher.;- bao ):

_launcher года_launcherLauncher ella там там_launcher года;-_launcher bao_fm_launcher_launcher_launcher ella.runners_launcherbeercalculator ):

 ):

ulled –

   да_launcher_radi каз да液化_launcherги.students да_launcher_launcher да да_launcher_launcher-la года tá_docs-pill.mag_launcherstyl-eventspoisХcalculatorCollectorга_launcherRaisedита 曹calculatorFetcherCollector McGr-el неб-ft débVolumes-filesVolumesRaised	events-booksRaised-pill-ro_textsfunkstyl新闻报道_fmитаCollectorитаOVERХ ella_reading -,_docsWiki -. dare”.

 ella_raise dareaboabo”.

 да_launcher ζ.party珂 practise_FS rising practise_FS (?_FSFinderGa обще_FS_mr_FSWalker_launcher practiseAppear_FS_FS practise_FS_RBWalker -.aboWalker_finder_FS practise_FS_RB_FStober_FS_FS practisetoberpps practise-IS_FS practise人会_RBFinderGoods_RBFinderGoods_FSGoodsWalker-IS practise人会_FStober_FSWalker_FS-IS_FSGoods_RB_FSFH人会GoodsισGa_FSWalker_MR_FS人会Walker_Detailspsc PickertoberWalkerodie_FSGoodsGoodsViaWalkPractice_RB_FSPractice_mrinsk practise_FS Wilde_FS Picker.idea_RBGoodsGoods人会 WwwFinder Gould Sounds_launcherGoods_FS.debита Voll inex余生_FS_RB_RB VollWalkerichi_FS Picker_FS Remed practisePracticeGoods_FS_FS_FS_FS_FSGoodsGoodsFHograd余生tober_FS Remed haitoberFHogradFHtoberaboFH_FS_FSFinderFinderFindertoberpps琥珀WBFHFinder和睦 FinderFinder FlintbalancedFinderPracticeWBFinder_FSinsk_CIoux人会tober_FSersteowoFindertober人会 practisetoberFHppsobot_RBGoods practiseGrammar.Ver_FS Flint人会WBPracticetober practisetoberppstoberFinder Flint_FS_FS RemedPracticefierWB人会_FStoberppsograd.whichPractice_FS_FS porterGoods人会艺术学院人会WB_FS_FSFinder_FS practise SignsFHWARtobertoberercippstoberFHFinder_FStober WI好啊PracticeGHoux娘娘WB_FS余生娘娘Practice_FSGuesstobertober_FSGoodsWB Picker_FS好啊tobertoberFHFinder/cc珂_FS_FSresses RemedFHbalanced声器Finder_FSFHVerifier списка람_FSFHPILE_FStober_FS_FS_FStoberFinderinsktober Remed_FS_FSigurFH_FS_FS_FS_FS_FSGa_FSGaFH_FSigurtober好啊_FSerna SignsodiaGHGHpps_FSigurigurigurppsISONppsFindertoberFinderpps inexcollectorPracticetoresrobiFHFinder_FS_FSFH Remed_FS_FStoberaday_FSigurtoberbalanced_FSFH人会Walk好啊igur_FS人会PracticePracticeogradtoberigurGoodsGHtober人会GH_FS Flint_FSPracticetober_FStober_FSbalancedolith callableFHtober声器FHGHGoodsGoodsmailer人会итаISONigur不明人会GoodsFH_FS人会igurigurpps_FSppsита_FS_FS_FSFHBinderGoods_FS Picker_FSigurtoberGoods_FSFH_FSGoods inexigur_FSFH Finderpps_FSernaianne Finderpsc:listCollector FinderFinder Pickerikersianne.calendar.calendarelight_sunelight.matcher.WRITE,_launcher sorterď_RBigur_FS-launchbalanced_FS_FS_FSGoodsGoodsinsktober_FSigurGoods余生ograd_FS_FSGa_FSigurilverppsGoods_FS_FS RemedGH.matcher_FS_FSigurskie_FSGHigurGoodstobertoberFHNit_FSGoodsGoodsograd_FS VIIussenPractice娘娘 GregolianGoodsGoods practiseGoods Colleg Remed Sonnikers Flintigur FlintGoodsigur_FS Ridertober Ridertober Remed_FS吉尔FH_FS_FS艺术学院_FS所欲Finder-ISита.VerPractice.debitel好啊WBита.debmailerGoods practise.Ver Remed_FStobertober_FSbalancedinsen.dx_FS_FS艺术学院 Remed Flint Wilde Remed人会AIR_FS_FStober RemedGoods好啊НовGoods七天MillerFH_FS_FSита.dxigurtober RemedGoods人会igurита Flint余生_FSGoodsoux人会_FS_FS Flintigur_FSGoods人会_FS_FSGoodsGoodsита.dxVerifier_FSGoodspickerGoods PickerVerifiergeist_FSmailer余生AIRmailer inexGoods_FS-_PSVerifier VollPractice Remedツ谤Verifier_FSmailer余生余生 SignsbaiGoodsитаGoods Signsigur_FSитаGoods_FS FinderAIRVerifierosteigur.VerGoodsGoods_FSigurGoods余生 inexgeistigur numa complimentaryynaVerifierGoods_FSď.Ver PickertoberVerifier.dx.dx Flint Remed.Ver所欲geistVerifier Signs_FSskiepickerFH Pickergeist Flint_FS_FSigur_FSVerifier.debigurigur_FSGoodsigur.VerGoods Voll所欲geistigur余生WB RiderGoodsGoodsgeist Signsinskbalancedtober所欲GHGoods inex reimbgeistoda所欲 scatterGHianne所欲pps_FS_FS娘娘ита_FSppsGoods inex_FS_localeigurodaigur声器.deb Picker所欲_salt所欲七天 Remed_FS WI_FSigur_FSigurитаита所欲 Guessigur Signs Signs所欲所欲178GrammarVerifier inex Wilde.dx inex Flintigur_FSskie inexита.dx inex_FS inexитаWB Flint inex_FS_FS scatterGoods_FS余生_FSigur_FS Signs Signsigur_FS Signs inex_FStober_FSpps SignsAIR inexитаFinderита Signs_FS.dxbalancedитаита_FSGoods所欲ита所欲七天geist Picker inex.dxigurucha私立ianne Goosetobertober吉尔pps_localeAIR.dxgeist七天_FS_saltppsита inexитаppsFileSystem Examiner所欲igur Signs所欲ianneigurAIR所欲tober Sonn所欲.va.va inexbalanced SonnVerifierFindergeist所欲ianne所欲ouxFinderbalancedитаVerifier Examinerита所欲 Signssalt-bootgeistiannegeist Sonn.va不言geisttober.va_sun底气geist_FS sorterianne七天GH mareigurgeistigur trophies Via Signsigur inex pinnedletteianneigurFH扫码skiefier不明 Signs_FStober inexpicker Picker inexитаigur_counters_FSikkaickers dockstoberGoodsита/ccитаikkapicker艺术学院igurGoods_FS_FS inex所欲ита_saltigurigurodiaGoods inex Findermailerита Examinerigur_salt Finder_FS Signs Remed扫码 inex inex WI.vaigur.vatoberGoodstober所欲geist所欲_saltGoods PickerFH Pickeritizenianne所欲_FS所欲 Pickertobertober.va_dictsFileSystem_saltandex.son.va inextober_salt Remedigur.va inex所欲Verifierigur Rider Signstober所欲_FS Signsikkaresses_salt所欲所欲geist_salt Signsigur_FS服务中心geist_salttoberigurgeist inex.Verитаgeistigur所欲 Sonn余生igurgeistoux所欲 inex.Ver inex余生 inex Remedianne.VerVerifiergeist inex inex_salt余生igur.vaoux Wildeianne Signs_salt inex余生 Wilde inexmailer_salt.MSG艺术学院 Remed_FS.vaigurVerifierigurPDabeebalancedитаilorigurPD Signs Signsgeistigurskie不明geist FinderPDgeist Examiner艺术学院itelgeist Examiner xcラー списка xcАрАрита списка_letterАритаАр+._letter_sunАрита спискаитаита tempListItemList_sun Sonn_sun-languageАр-language_sunита_sun.arm_locale-languageАр.arm.armickers:list-language.armАрАр	xtype.calendar.arm.arm.arm trophies..:listАр.arm.arm trophies_locale trophies.arm trophies_lc(lista:listитаитаита.arm trophies.arm.armilor.arm.arm:list.arm trophies.arm:list_locale.arm trophies.arm:listАр trophies.arm.armита.arm.arm声器.arm.arm.arm trophies.arm trophies.armита..armАр1ilor.calendar.calendar:list.arm.calendar.calendar trophiesита.arm声器ита.arm Ар..arm.arm trophiesلف.arm.calendarита.arm меньше_lcита trophies声器ита.calendar trophies.less trophies:listита.less声器.less:list.calendar.calendar trophies声器 trophies.calendar.calendar:list.arm.arm	assertFalse.arm trophiesита trophies.calendar trophies+t_localeита+t.arm_lcита trophiesАр Аритаulg trophies.calendar..arm trophiesитаита:list спискаazo声器_lc.calendar trophies. trophiesита trophies trophies气和. trophies_locale声器.calendar.armulg_lc.arm trophies.artıl.armfier.less.less trophies trophies.less.art声器_sun.calendarulg.calendar trophiesickersита.calendar.calendar списка.less_collections.less trophies.calendar..less:list trophies_ENTRIES trophies trophies trophiesита trophies声器_lc.calendar trophies.calendarfier trophies.calendar_launcher.bio.armfier声器 trophies trophies.calendar trophies.less:list trophies.calendar_lc_wall_collections.arm.calendarита trophies trophiesита.lessfier.calendar:listلس Ар.armита.less.bio声器.arm.calendar.calendar.art.less trophies.arm.calendar_collectionsита.less.less(lista.arm_collections.arm:listласьLauncher.less►.bio声器_launcher Ар.bio_launcher_collections.armита.calendar_launcher.calendar_launcher.armитаита_launcher_launcheritta arte.bio_launcher_launcher арАр气和.lessickers arteickers_lcCollector钊 арickersАр.art.bio arte arte_audit:listCollector искус气和气和Collector.bio.less ар艺术Scanner ар_audit.bioلس Arte:listilliilliittel imassc_challenge.bio начал:listildailliCollector_launcher.less lvl气和Collectoralisation callersollo气和.lessisses арcollector launcherschein Wie为难氣为王珂太郎.artieux气和elightmounted.lessWalk maskedSStiva.lessinhaarkigurucha_FS_visitor.OS为王 checkeriesen искус.diffitestVERIFYrosse-sk_lc.diffickersApolloickers.less艺术学院schein Picker Flood람_lc艺术学院看不起ош Flood.axes_tools Daniels inex yourselves所欲 tenta.bio MaoternaAPS換Mailerlearner/errors ар艺术学院hallWalk SPELL дух_lc paralle. Rider marsussen.less不明/errors-training气和/art crosses.vanier искус_visitoritel气和砚 complementary yourselvesbagsMessenger crossesikers Rideroux換 искус氣珂 искус SPELLumasussen气和itelilda complimentaryikkageistMailer七天_visitor.va气和 Goose scareoya Auroraoux Aurora paralleodia/photoikkaipa气和msgigur_auditза.va_auditolaАр Danorsa 听ilda_sku_audit_auditorta.less humour Dana Arte dn.ser.infinity hoa气итаclimateapsienne9曜195气isciΟoux	ADD
challenge-reasoning-016 PASS
Suite: reasoning · Latency: 6043 ms · Cost: $0.001422 · Finish: stop
Prompt
只回答最终数字:3x3 矩阵按行是 1 2 3 / 4 5 6 / 7 8 9。顺时针旋转 90 度后,第二行元素之和是多少?
Standard Answer / Scoring Expectation
Answer: 15
Output
15
challenge-reasoning-017 PASS
Suite: reasoning · Latency: 6014 ms · Cost: $0.001644 · Finish: stop
Prompt
只回答最终数字:依赖关系为 a 无依赖;b 依赖 a;c 依赖 a;d 依赖 b 和 c;e 依赖 b;f 依赖 d 和 e;g 依赖 a。按可并行执行的拓扑层划分,第二层有几个任务?
Standard Answer / Scoring Expectation
Answer: 3
Output
3
challenge-reasoning-018 PASS
Suite: reasoning · Latency: 10259 ms · Cost: $0.001367 · Finish: stop
Prompt
只回答星期几:第 0 天是星期二。每 7 天循环一次。第 17 天是星期几?
Standard Answer / Scoring Expectation
Answer: 星期五
Output
星期五
challenge-reasoning-019 PASS
Suite: reasoning · Latency: 12842 ms · Cost: $0.003078 · Finish: stop
Prompt
只回答最终数字:用 A、B、C 组成长度为 3 的字符串,要求恰好包含一个 A,并且最后一个字符不能是 C。满足条件的字符串有多少个?
Standard Answer / Scoring Expectation
Answer: 8
Output
8
challenge-reasoning-020 PASS
Suite: reasoning · Latency: 12987 ms · Cost: $0.001868 · Finish: stop
Prompt
只回答最终数字:栈操作从空栈开始。push n 表示入栈;dup 复制栈顶;add 弹出两个数并压入它们的和;swap 交换栈顶两个数;sub 先弹出 x 再弹出 y,并压入 y-x。依次执行:push 2, push 3, dup, add, push 4, swap, sub。最终栈顶是多少?
Standard Answer / Scoring Expectation
Answer: -2
Output
-2
challenge-reasoning-021 PASS
Suite: reasoning · Latency: 12740 ms · Cost: $0.001610 · Finish: stop
Prompt
只回答最终数字:三个队列 Q1、Q2、Q3 初始长度分别为 2、1、0。每一轮先从所有非空队列各处理 1 个元素,然后向 Q3 加入 2 个元素。执行 3 轮后,三个队列总长度是多少?
Standard Answer / Scoring Expectation
Answer: 4
Output
4
challenge-reasoning-022 PASS
Suite: reasoning · Latency: 5972 ms · Cost: $0.000621 · Finish: stop
Prompt
只回答最终数字:映射初始为 {a:1, b:2}。依次执行:设置 c=a+b;设置 a=c-b;删除 b;设置 d=a+c。最终所有 value 之和是多少?
Standard Answer / Scoring Expectation
Answer: 8
Output
8
challenge-reasoning-023 PASS
Suite: reasoning · Latency: 16044 ms · Cost: $0.001071 · Finish: stop
Prompt
只回答“甲”“乙”或“丙”:甲说“乙说的是真话”;乙说“丙说的是假话”;丙说“甲说的是假话”。如果恰好一人说真话,说真话的人是谁?
Standard Answer / Scoring Expectation
Answer: 丙
Output
challenge-reasoning-024 PASS
Suite: reasoning · Latency: 7150 ms · Cost: $0.001355 · Finish: stop
Prompt
只回答一个字母:综合分 = 通过率百分数 - 成本乘以 10。模型 A 通过率 92%、成本 0.20;模型 B 通过率 88%、成本 0.05;模型 C 通过率 90%、成本 0.10。综合分最高的是哪个模型?
Standard Answer / Scoring Expectation
Answer: A
Output
A
challenge-reasoning-025 PASS
Suite: reasoning · Latency: 9542 ms · Cost: $0.001795 · Finish: stop
Prompt
只回答最终三位数:一个三位数的百位为 a、十位为 b、个位为 c。已知 a+b+c=13,a=c+1,b=2c。这个三位数是多少?
Standard Answer / Scoring Expectation
Answer: 463
Output
463
challenge-coding-015 PASS
Suite: coding · Latency: 41685 ms · Cost: $0.005393 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(line: str) -> list[str],解析一行简化 CSV:逗号分隔;字段可以用双引号包裹;引号内逗号保留;引号内两个连续双引号表示一个双引号;空字段保留;空格是普通字符,不要自动 trim。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve('a,"b,c",d') == ['a', 'b,c', 'd']
assert solution.solve('"a""b",,x') == ['a"b', '', 'x']
assert solution.solve('  a ," b " ') == ['  a ', ' b  ']
Output
```python
def solve(line: str) -> list[str]:
    fields = []
    current = []
    in_quote = False
    i = 0
    n = len(line)
    while i < n:
        c = line[i]
        if not in_quote:
            if c == ',':
                fields.append(''.join(current))
                current = []
            elif c == '"':
                in_quote = True
            else:
                current.append(c)
        else:
            if c == '"':
                if i + 1 < n and line[i+1] == '"':
                    current.append('"')
                    i += 1
                else:
                    in_quote = False
            else:
                current.append(c)
        i += 1
    fields.append(''.join(current))
    return fields
```
challenge-coding-016 PASS
Suite: coding · Latency: 14849 ms · Cost: $0.002570 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(intervals: list[tuple[int, int]]) -> list[tuple[int, int]]。intervals 是整数闭区间,先按起点排序;重叠或端点相邻的区间要合并,例如 [1,2] 和 [3,4] 合并为 [1,4]。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve([(5,7),(1,2),(3,4),(10,10)]) == [(1,7),(10,10)]
assert solution.solve([(1,1),(3,3)]) == [(1,1),(3,3)]
assert solution.solve([]) == []
Output
```python
def solve(intervals: list[tuple[int, int]]) -> list[tuple[int, int]]:
    if not intervals:
        return []
    intervals.sort(key=lambda x: x[0])
    merged = [intervals[0]]
    for start, end in intervals[1:]:
        last_start, last_end = merged[-1]
        if start <= last_end + 1:
            merged[-1] = (last_start, max(last_end, end))
        else:
            merged.append((start, end))
    return merged
```
challenge-coding-017 PASS
Suite: coding · Latency: 25722 ms · Cost: $0.006138 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(events: list[tuple[str, int]], window: int, limit: int) -> list[bool],模拟按用户限流。events 按时间非降序排列。某用户在当前事件时间 ts 的窗口 (ts-window, ts] 内,已放行事件数少于 limit 才放行;被拒绝的事件不计入窗口。
Standard Answer / Scoring Expectation
Reference tests:
events=[('a',1),('a',2),('a',3),('a',4),('a',8)]
assert solution.solve(events, 5, 2) == [True, True, False, False, True]
assert solution.solve([('x',10),('x',11),('x',16)], 5, 1) == [True, False, True]
Output
```python
from collections import defaultdict, deque

def solve(events: list[tuple[str, int]], window: int, limit: int) -> list[bool]:
    user_events = defaultdict(deque)
    result = []
    for user, ts in events:
        dq = user_events[user]
        while dq and dq[0] <= ts - window:
            dq.popleft()
        if len(dq) < limit:
            result.append(True)
            dq.append(ts)
        else:
            result.append(False)
    return result
```
challenge-coding-018 PASS
Suite: coding · Latency: 30918 ms · Cost: $0.002548 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(path: str) -> str,规范化绝对 Unix 路径:处理重复斜杠、.、..;.. 在根目录时仍留在根目录;返回结果必须以 / 开头且除根目录外不能以 / 结尾。不要使用 os.path。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve('/a//b/./c/../') == '/a/b'
assert solution.solve('/../../x') == '/x'
assert solution.solve('/') == '/'
Output
```python
def solve(path: str) -> str:
    parts = path.split('/')
    stack = []
    for part in parts:
        if part == '' or part == '.':
            continue
        elif part == '..':
            if stack:
                stack.pop()
        else:
            stack.append(part)
    result = '/' + '/'.join(stack)
    return result
```
challenge-coding-019 PASS
Suite: coding · Latency: 15534 ms · Cost: $0.002389 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(words: list[str]) -> list[list[str]],按变位词分组。大小写敏感;组内保持输入顺序;组的顺序按该组第一个单词在输入中出现的位置。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve(['eat','tea','tan','ate','nat','bat']) == [['eat','tea','ate'], ['tan','nat'], ['bat']]
assert solution.solve(['b','B','bb']) == [['b'], ['B'], ['bb']]
Output
```python
def solve(words: list[str]) -> list[list[str]]:
    groups = {}
    for w in words:
        key = tuple(sorted(w))
        if key not in groups:
            groups[key] = []
        groups[key].append(w)
    return list(groups.values())
```
challenge-coding-020 PASS
Suite: coding · Latency: 28719 ms · Cost: $0.006020 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(text: str) -> dict,解析 Markdown front matter。只有第一行正好是 --- 时才解析,直到下一行 --- 结束;每行 key: value,key 和 value 两侧 trim;形如 [a, b] 的 value 返回字符串列表;其他 value 返回字符串。没有完整 front matter 返回 {}。
Standard Answer / Scoring Expectation
Reference tests:
text='---\ntitle: Test\ntags: [ai, bench,  v0]\ndraft: false\n---\nbody'
assert solution.solve(text) == {'title':'Test','tags':['ai','bench','v0'],'draft':'false'}
assert solution.solve('hello\n---\na: b') == {}
assert solution.solve('---\na: 1') == {}
Output
```python
def solve(text: str) -> dict:
    lines = text.split('\n')
    if not lines or lines[0] != '---':
        return {}
    
    end_idx = -1
    for i in range(1, len(lines)):
        if lines[i] == '---':
            end_idx = i
            break
    
    if end_idx == -1:
        return {}
        
    result = {}
    for line in lines[1:end_idx]:
        if ':' not in line:
            continue
        key, value = line.split(':', 1)
        key = key.strip()
        value = value.strip()
        
        if value.startswith('[') and value.endswith(']'):
            inner = value[1:-1].strip()
            if inner:
                result[key] = [item.strip() for item in inner.split(',')]
            else:
                result[key] = []
        else:
            result[key] = value
            
    return result
```
challenge-coding-021 PASS
Suite: coding · Latency: 16456 ms · Cost: $0.003309 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(items: list[str], k: int) -> list[str],返回出现次数最高的前 k 个元素;次数相同按第一次出现的先后排序;k 大于不同元素数量时返回全部。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve(['b','a','b','c','a','b'], 2) == ['b','a']
assert solution.solve(['x','y','z','y','z','x'], 2) == ['x','y']
assert solution.solve(['a'], 5) == ['a']
Output
```python
def solve(items: list[str], k: int) -> list[str]:
    freq = {}
    first_idx = {}
    for i, item in enumerate(items):
        if item not in freq:
            freq[item] = 0
            first_idx[item] = i
        freq[item] += 1
    sorted_items = sorted(freq.keys(), key=lambda x: (-freq[x], first_idx[x]))
    return sorted_items[:k]
```
challenge-coding-022 PASS
Suite: coding · Latency: 44827 ms · Cost: $0.008089 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(config: dict, overrides: dict[str, object]) -> dict。overrides 的 key 是点号路径;值为 None 表示删除该路径的叶子 key;其他值写入路径,必要时创建中间 dict;不能修改输入 config。
Standard Answer / Scoring Expectation
Reference tests:
base={'db':{'host':'old','port':5432},'feature':{'x':1}}
overrides={'db.host':'new','feature.x':None,'api.timeout':30}
res=solution.solve(base, overrides)
assert res == {'db':{'host':'new','port':5432},'feature':{},'api':{'timeout':30}}
assert base == {'db':{'host':'old','port':5432},'feature':{'x':1}}
assert solution.solve({'a':1}, {'a.b':2}) == {'a': {'b': 2}}
Output
```python
import copy

def solve(config: dict, overrides: dict[str, object]) -> dict:
    result = copy.deepcopy(config)
    for path, value in overrides.items():
        parts = path.split('.')
        d = result
        for part in parts[:-1]:
            if part not in d or not isinstance(d.get(part), dict):
                d[part] = {}
            d = d[part]
        key = parts[-1]
        if value is None:
            d.pop(key, None)
        else:
            d[key] = value
    return result
```
challenge-coding-023 PASS
Suite: coding · Latency: 37001 ms · Cost: $0.003015 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(query: str) -> dict[str, list[str]],解析 URL query string。& 分隔参数;没有 = 的参数值为空字符串;+ 表示空格;%XX 需要按 UTF-8 百分号解码;同名 key 保留所有值的列表。可以使用标准库。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve('a=1&b=two+words&a=3') == {'a':['1','3'], 'b':['two words']} 
assert solution.solve('empty=&flag&x=a%2Bb') == {'empty':[''], 'flag':[''], 'x':['a+b']} 
assert solution.solve('') == {}
Output
```python
from urllib.parse import unquote_plus
from collections import defaultdict

def solve(query: str) -> dict[str, list[str]]:
    result = defaultdict(list)
    if not query:
        return dict(result)
    for part in query.split('&'):
        if not part:
            continue
        if '=' in part:
            key, value = part.split('=', 1)
        else:
            key, value = part, ''
        result[unquote_plus(key)].append(unquote_plus(value))
    return dict(result)
```
challenge-coding-024 PASS
Suite: coding · Latency: 71718 ms · Cost: $0.004627 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(html: str) -> bool,校验简化 HTML 标签是否配对。标签形如 <a>、</a>、<br/>,标签名只含小写字母;自闭合标签不入栈;普通文本忽略。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve('<div><br/><p>x</p></div>') is True
assert solution.solve('<a><b></a></b>') is False
assert solution.solve('text<a><b/></a>') is True
Output
```python
import re

def solve(html: str) -> bool:
    stack = []
    for match in re.finditer(r'<(/?)([a-z]+)(/?)>', html):
        is_closing, tag_name, is_self_closing = match.groups()
        if is_self_closing:
            continue
        if is_closing:
            if not stack or stack.pop() != tag_name:
                return False
        else:
            stack.append(tag_name)
    return not stack
```
challenge-coding-025 PASS
Suite: coding · Latency: 15098 ms · Cost: $0.002782 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(grid: list[str]) -> int。grid 中 S 是起点,E 是终点,# 是墙,. 可走。每步上下左右移动一格,返回从 S 到 E 的最短步数;不可达返回 -1。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve(['S..','##.','..E']) == 4
assert solution.solve(['S#','##','E.']) == -1
assert solution.solve(['SE']) == 1
Output
```python
from collections import deque

def solve(grid: list[str]) -> int:
    if not grid:
        return -1
    rows, cols = len(grid), len(grid[0])
    start = None
    for r in range(rows):
        for c in range(cols):
            if grid[r][c] == 'S':
                start = (r, c)
                break
        if start:
            break
            
    if not start:
        return -1
        
    queue = deque([(start[0], start[1], 0)])
    visited = set([start])
    directions = [(0, 1), (0, -1), (1, 0), (-1, 0)]
    
    while queue:
        r, c, dist = queue.popleft()
        if grid[r][c] == 'E':
            return dist
            
        for dr, dc in directions:
            nr, nc = r + dr, c + dc
            if 0 <= nr < rows and 0 <= nc < cols and (nr, nc) not in visited and grid[nr][nc] != '#':
                visited.add((nr, nc))
                queue.append((nr, nc, dist + 1))
                
    return -1
```
challenge-coding-026 PASS
Suite: coding · Latency: 22715 ms · Cost: $0.006959 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(nums: list[int], k: int) -> list[float],返回每个长度为 k 的滑动窗口中位数。k 为奇数时返回中间值;k 为偶数时返回两个中间值平均数。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve([1,3,-1,-3,5,3,6,7], 3) == [1, -1, -1, 3, 5, 6]
assert solution.solve([1,2,3,4], 2) == [1.5, 2.5, 3.5]
Output
```python
import bisect

def solve(nums: list[int], k: int) -> list[float]:
    w = sorted(nums[:k])
    res = []
    for i in range(len(nums) - k + 1):
        res.append((w[k // 2] + w[(k - 1) // 2]) / 2.0)
        if i + k < len(nums):
            bisect.insort(w, nums[i + k])
            w.pop(bisect.bisect_left(w, nums[i]))
    return res
```
challenge-coding-027 PASS
Suite: coding · Latency: 5205 ms · Cost: $0.004252 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(versions: list[str]) -> list[str],按简化语义版本排序。版本为 major.minor.patch 或 major.minor.patch-label;major/minor/patch 按数字比较;同一数字版本下,带 label 的预发布版本低于正式版本;label 按字典序比较。
Standard Answer / Scoring Expectation
Reference tests:
assert solution.solve(['1.0.0','1.0.0-alpha','1.0.1','1.0.0-beta','0.9.9']) == ['0.9.9','1.0.0-alpha','1.0.0-beta','1.0.0','1.0.1']
assert solution.solve(['1.0.10','1.0.2']) == ['1.0.2','1.0.10']
Output
```python
def solve(versions: list[str]) -> list[str]:
    def key(v):
        if '-' in v:
            base, label = v.split('-', 1)
            m, n, p = map(int, base.split('.'))
            return (m, n, p, 0, label)
        else:
            m, n, p = map(int, v.split('.'))
            return (m, n, p, 1, "")
    return sorted(versions, key=key)
```
challenge-coding-028 PASS
Suite: coding · Latency: 21157 ms · Cost: $0.006444 · Finish: stop
Prompt
只输出 Python 代码,不要解释。实现 solve(deps: dict[str, list[str]], changed: list[str]) -> list[str]。deps 表示 package -> 直接依赖。返回所有 changed 包,以及直接或间接依赖 changed 包的包,按字典序排序;只出现在依赖列表中的包也要能作为 changed 包返回。
Standard Answer / Scoring Expectation
Reference tests:
deps={'app':['lib','ui'],'ui':['core'],'lib':['core'],'core':[],'docs':[]}
assert solution.solve(deps, ['core']) == ['app','core','lib','ui']
assert solution.solve({'a':['x']}, ['x']) == ['a','x']
assert solution.solve({'a':['b'], 'b':[]}, ['a']) == ['a']
Output
```python
from collections import deque

def solve(deps: dict[str, list[str]], changed: list[str]) -> list[str]:
    rev_deps = {}
    for pkg, dependencies in deps.items():
        for dep in dependencies:
            if dep not in rev_deps:
                rev_deps[dep] = set()
            rev_deps[dep].add(pkg)
    
    affected = set(changed)
    queue = deque(changed)
    
    while queue:
        curr = queue.popleft()
        if curr in rev_deps:
            for dependent in rev_deps[curr]:
                if dependent not in affected:
                    affected.add(dependent)
                    queue.append(dependent)
                    
    return sorted(list(affected))
```
challenge-rag-008 PASS
Suite: rag_long_context · Latency: 8179 ms · Cost: $0.001268 · Finish: stop
Prompt
只根据资料回答。

资料 A:早期草案说 v0.3 只新增 20 道题。
资料 B:2026-06-16 的确认方案是 v0.3-challenge100,在现有 50 题基础上新增 50 道题。
资料 C:新增题应先生成审题 HTML,确认后再跑模型,不要立即消耗 API。

问题:下一步应该新增多少题?是否应该立刻跑模型?
Standard Answer / Scoring Expectation
Required phrases: 50、审题
At least one from each group: 不跑 / 不要立刻 / 先不跑 / 不应立刻 / 不应该立刻 / 不应立即 / 不应该立即 / 不应立刻跑 / 不应该立刻跑 / 不应立即跑 / 不应该立即跑
Length guidance (diagnostic only): min=n/a, max=160
Output
新增50道题;不应该立刻跑模型,应先生成审题HTML确认后再跑。
challenge-rag-009 PASS
Suite: rag_long_context · Latency: 19600 ms · Cost: $0.001094 · Finish: stop
Prompt
只根据资料回答,不要自行补充。

资料:报告展示 pass_rate、avg_latency_ms、total_cost、reasoning_tokens、失败样例和逐题明细。资料没有包含用户满意度调查,也没有包含模型训练数据来源。

问题:能否根据这份报告判断用户满意度?
Standard Answer / Scoring Expectation
Required phrases: 不能、用户满意度
At least one from each group: 没有包含 / 没有提供 / 资料没有 / 未包含
Forbidden phrases: 可以判断
Length guidance (diagnostic only): min=n/a, max=120
Output
不能。资料明确说明没有包含用户满意度调查。
challenge-rag-010 PASS
Suite: rag_long_context · Latency: 11130 ms · Cost: $0.001549 · Finish: stop
Prompt
只根据资料回答。

资料:模型 A 通过 48/50,平均延迟 20 秒,成本 0.20 美元,没有超时失败。模型 B 通过 44/50,平均延迟 4 秒,成本 0.05 美元,没有超时失败。模型 C 通过 46/50,平均延迟 9 秒,成本 0.04 美元,但有 3 次 API 超时。

问题:如果要选低延迟且稳定的生产摘要模型,资料更支持选择哪个?
Standard Answer / Scoring Expectation
Required phrases: B、延迟
At least one from each group: 没有超时 / 无超时
Forbidden phrases: 选择 A、选择 C
Length guidance (diagnostic only): min=n/a, max=160
Output
模型 B。

理由:模型 B 平均延迟最低(4秒),且没有超时失败(稳定),符合“低延迟且稳定”的要求。模型 A 延迟高,模型 C 存在超时不稳定。
challenge-rag-011 PASS
Suite: rag_long_context · Latency: 6064 ms · Cost: $0.001654 · Finish: stop
Prompt
只根据资料回答。

资料:DracoBench 当前 scoring policy 中,text_rules 的 max_chars/min_chars 只作为诊断信息记录,length_scored 为 false。缺少 required、required_any 或出现 forbidden 才会导致失败。

问题:如果答案超过 max_chars,但 required 都满足且没有 forbidden,会因为长度直接扣分吗?
Standard Answer / Scoring Expectation
Required phrases: 诊断
At least one from each group: 不扣分 / 不会 / 不会因长度 / 不会因此被扣分 / 不会直接导致扣分 / 不会导致扣分 / 不会导致失败或扣分;length_scored / 长度
Forbidden phrases: 直接失败、必须重跑
Length guidance (diagnostic only): min=n/a, max=140
Output
不会。根据资料,max_chars 只作为诊断信息记录且 length_scored 为 false,只有缺少 required/required_any 或出现 forbidden 才会导致失败。
challenge-rag-012 PASS
Suite: rag_long_context · Latency: 15058 ms · Cost: $0.001099 · Finish: stop
Prompt
只根据资料回答。

资料 A:某模型原始结果为 45/50。
资料 B:修正一个同义表达评分规则后,rescored 结果为 46/50。
资料 C:总览 index 使用 rescored JSONL 作为展示结果。

问题:写文章引用总览结果时,应写 45/50 还是 46/50?
Standard Answer / Scoring Expectation
Required phrases: 46/50
Forbidden phrases: 45/50 更合适、原始结果为准
Length guidance (diagnostic only): min=n/a, max=140
Output
46/50。因为资料C指出总览使用rescored结果,而资料B指明rescored结果为46/50。
challenge-rag-013 PASS
Suite: rag_long_context · Latency: 3258 ms · Cost: $0.001271 · Finish: stop
Prompt
只根据资料回答。

资料:为了复现一次 benchmark,记录中包含 model、case_id、prompt_hash、parameters、scorer details、output、latency_ms 和 usage。记录不包含 chain of thought,也不包含训练数据。

问题:复现评测主要应该依赖哪些记录字段?
Standard Answer / Scoring Expectation
Required phrases: model、prompt_hash、parameters、scorer
Forbidden phrases: chain of thought、训练数据
Length guidance (diagnostic only): min=n/a, max=200
Output
根据资料,复现评测主要应该依赖以下记录字段:model、case_id、prompt_hash、parameters、scorer details、output、latency_ms 和 usage。
challenge-rag-014 PASS
Suite: rag_long_context · Latency: 5996 ms · Cost: $0.001089 · Finish: stop
Prompt
只根据资料回答。

资料 A:早期 ability mode 使用 4096 max_tokens。
资料 B:后来为了公平,ability mode 统一放宽到 16384 max_tokens。
资料 C:当前总览页只展示 v0.2-challenge50 且 ability16384 的结果。

问题:当前总览页对应的 token limit 是多少?
Standard Answer / Scoring Expectation
Required phrases: 16384
Forbidden phrases: 4096
Length guidance (diagnostic only): min=n/a, max=100
Output
16384
challenge-rag-015 PASS
Suite: rag_long_context · Latency: 16403 ms · Cost: $0.002796 · Finish: stop
Prompt
只根据资料回答。

资料:coding-004 的失败是代码运行时 ValueError,根因是把字符串 key 强转 int。rag-006 的失败曾经是规则没有把 int 类型视作“整数”的同义表达,后来通过人工复查修正规则。

问题:哪类失败更需要人工复查评分规则?回答时也说明 coding-004 是否属于这种情况。
Standard Answer / Scoring Expectation
Required phrases: 人工复查、coding-004
At least one from each group: rag-006 / 规则 / 评分规则 / 同义表达;不属于 / 不是 / 不需要
Length guidance (diagnostic only): min=n/a, max=180
Output
因规则未涵盖同义表达(或规则定义缺陷)导致的失败更需要人工复查评分规则。

coding-004 不属于这种情况,它的失败是代码运行时的类型强转错误(ValueError),与评分规则无关。
challenge-rag-016 PASS
Suite: rag_long_context · Latency: 6431 ms · Cost: $0.000963 · Finish: stop
Prompt
只根据资料回答。

资料:usage 字段包含 prompt_tokens、completion_tokens、reasoning_tokens 和 cost。报告没有供应商真实利润率,也没有供应商内部成本。

问题:能否从 usage 字段判断供应商利润率?
Standard Answer / Scoring Expectation
Required phrases: 不能、利润率
At least one from each group: 不包含 / 没有
Forbidden phrases: 可以判断
Length guidance (diagnostic only): min=n/a, max=120
Output
不能。资料明确指出没有供应商内部成本,因此无法判断供应商利润率。
challenge-rag-017 PASS
Suite: rag_long_context · Latency: 34197 ms · Cost: $0.001746 · Finish: stop
Prompt
只根据资料回答。

资料:当前 index 页只显示最后这版 50 道题的 v0.2-challenge50 结果,并且 token limit 都统一为 ability16384。之前的版本和 ability4096 结果不显示在 index 页。

问题:index 页是否应该展示旧版本结果?当前展示范围是什么?
Standard Answer / Scoring Expectation
Required phrases: v0.2-challenge50、ability16384
At least one from each group: 不应该 / 不展示 / 未展示 / 不显示 / 均不显示 / 不应展示;旧版本 / 之前的版本
Length guidance (diagnostic only): min=n/a, max=160
Output
不应该展示旧版本结果。

当前展示范围是:最后这版 50 道题的 v0.2-challenge50 结果,且 token limit 统一为 ability16384。
challenge-rag-018 PASS
Suite: rag_long_context · Latency: 16425 ms · Cost: $0.001361 · Finish: stop
Prompt
只根据资料回答。

资料:某 coding 题中,模型响应的 finish_reason 为 length,output 为空,测试报错为 module has no attribute solve。报告认为这不是算法逻辑错误,而是输出被截断后没有形成可执行函数。

问题:这次失败应如何归因?
Standard Answer / Scoring Expectation
Required phrases: 输出
At least one from each group: 截断 / length / token
Length guidance (diagnostic only): min=n/a, max=140
Output
归因为输出被截断导致未形成可执行函数,而非算法逻辑错误。
challenge-debugging-015 PASS
Suite: debugging · Latency: 13337 ms · Cost: $0.000873 · Finish: stop
Prompt
下面代码期望 funcs[0]()、funcs[1]()、funcs[2]() 分别返回 0、1、2,但实际都会返回 2。请指出根因,并给出最小修复。最后单独一行写 FIX: <修复后的 append 行>。

```python
funcs = []
for i in range(3):
    funcs.append(lambda: i)
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*funcs\.append\(lambda\s+i=i\s*:\s*i\)\s*`?
Output
**根因**:lambda 捕获的是变量 `i` 的引用,而非其值。循环结束后 `i` 为 2,所有 lambda 都返回最终的 `i` 值 2。

**最小修复**:利用默认参数在定义时绑定当前值。

FIX: funcs.append(lambda i=i: i)
challenge-debugging-016 PASS
Suite: debugging · Latency: 14651 ms · Cost: $0.001439 · Finish: stop
Prompt
下面 context manager 期望异常继续向外抛出,但实际异常被吞掉。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 return 行>。

```python
class Span:
    def __enter__(self):
        return self
    def __exit__(self, exc_type, exc, tb):
        return True
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*return(?:\s+False|\s+None)?\s*`?
Output
Bug:`__exit__` 返回 `True` 会抑制异常,导致异常被吞掉而不向外传播。要让异常继续抛出,应返回 `False`。

最小修复:将 `return True` 改为 `return False`。

FIX: return False
challenge-debugging-017 PASS
Suite: debugging · Latency: 19019 ms · Cost: $0.001526 · Finish: stop
Prompt
下面代码期望每个 Bag 实例都有独立列表,但不同实例会共享 items。请指出 bug,并给出最小修复。最后单独一行写 FIX: <新增到 __init__ 中的赋值行>。

```python
class Bag:
    items = []
    def add(self, x):
        self.items.append(x)
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*self\.items\s*=\s*\[\]\s*`?
Output
Bug 在于 `items = []` 是类属性,被所有实例共享。最小修复是将其改为在 `__init__` 中初始化的实例属性(同时删除原类属性)。

FIX: self.items = []
challenge-debugging-018 PASS
Suite: debugging · Latency: 5893 ms · Cost: $0.000626 · Finish: stop
Prompt
下面代码期望每次不传 ts 时使用当前时间,但默认时间只会在函数定义时计算一次。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的函数定义行>。

```python
from datetime import datetime

def mark(ts=datetime.utcnow()):
    return ts.isoformat()
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*def\s+mark\(ts=None\)\s*:
Output
Bug:`datetime.utcnow()` 作为默认参数,只在函数定义时求值一次,后续调用不传 `ts` 时都会返回同一个时间戳。

最小修复:将默认值改为 `None`,在函数体内判断并赋值。

FIX: def mark(ts=None):
challenge-debugging-019 PASS
Suite: debugging · Latency: 7266 ms · Cost: $0.000992 · Finish: stop
Prompt
下面代码期望返回小写字符串,但实际返回的是方法对象。请指出 bug,并给出最小修复。最后单独一行写 FIX: <修复后的 return 行>。

```python
def normalize(s):
    return s.strip().lower
```
Standard Answer / Scoring Expectation
Expected regex pattern:
FIX:\s*`?\s*return\s+s\.strip\(\)\.lower\(\)\s*`?
Output
Bug 在于 `lower` 是字符串的方法,缺少调用括号 `()`,导致返回了方法对象本身而不是其执行结果。

最小修复:为 `lower` 添加括号以调用该方法。

FIX: return s.strip().lower()
challenge-if-006 PASS
Suite: instruction_following · Latency: 28566 ms · Cost: $0.002027 · Finish: stop
Prompt
只输出 JSON 对象,不要 Markdown。字段 version 必须是 "v0.3";字段 ready 必须是 true;字段 cases 必须是正好两个对象组成的数组。cases 中每个对象只能包含 id、weight、active 三个字段;id 是字符串且匹配 case-1 或 case-2;weight 是整数;active 是布尔值。
Standard Answer / Scoring Expectation
Expected JSON schema:
{
  "type": "object",
  "required": [
    "version",
    "ready",
    "cases"
  ],
  "additionalProperties": false,
  "properties": {
    "version": {
      "type": "string",
      "const": "v0.3"
    },
    "ready": {
      "type": "boolean",
      "const": true
    },
    "cases": {
      "type": "array",
      "minItems": 2,
      "maxItems": 2,
      "items": {
        "type": "object",
        "required": [
          "id",
          "weight",
          "active"
        ],
        "additionalProperties": false,
        "properties": {
          "id": {
            "type": "string",
            "pattern": "^case-[12]$"
          },
          "weight": {
            "type": "integer"
          },
          "active": {
            "type": "boolean"
          }
        }
      }
    }
  }
}
Output
{"version":"v0.3","ready":true,"cases":[{"id":"case-1","weight":1,"active":true},{"id":"case-2","weight":2,"active":false}]}
challenge-zh-writing-004 PASS
Suite: chinese_writing · Latency: 35692 ms · Cost: $0.003903 · Finish: stop
Prompt
写一段不超过 160 个汉字的公众号小结,主题是“为什么 v0.3 要扩到 100 题”。必须包含“推理题”“边界条件”“人工复查”,不能出现“排行榜”。
Standard Answer / Scoring Expectation
Required phrases: 推理题、边界条件、人工复查
Forbidden phrases: 排行榜
Length guidance (diagnostic only): min=n/a, max=160
Output
v0.3为何扩至100题?模型日益聪明,旧题库已难分伯仲。我们大幅增加推理题,并深挖易错的边界条件,用更严苛的考卷逼出真实短板。题量激增也带来评分波动风险,为此我们全面强化人工复查,确保每道题判定精准,让评测更扎实可信。

Case Details

CaseSuiteStatusFailure TypeScoreLatencyCostFinish
challenge-coding-001codingPASS-1.033196 ms$0.008464stop
challenge-coding-002codingPASS-1.030751 ms$0.005914stop
challenge-coding-003codingFAILapi_timeout_no_answer0.0300003 ms$0.000000None
challenge-coding-004codingFAILruntime_type_assumption0.099770 ms$0.005782stop
challenge-coding-005codingPASS-1.07696 ms$0.001103stop
challenge-coding-006codingPASS-1.034369 ms$0.002339stop
challenge-coding-007codingFAILdegenerate_output0.027096 ms$0.005129stop
challenge-coding-008codingPASS-1.010893 ms$0.001740stop
challenge-debugging-001debuggingPASS-1.015112 ms$0.003169stop
challenge-debugging-002debuggingPASS-1.04554 ms$0.000576stop
challenge-debugging-003debuggingPASS-1.032583 ms$0.001116stop
challenge-debugging-004debuggingPASS-1.06716 ms$0.000804stop
challenge-debugging-005debuggingPASS-1.016549 ms$0.002290stop
challenge-debugging-006debuggingPASS-1.018876 ms$0.001464stop
challenge-debugging-007debuggingPASS-1.014997 ms$0.002263stop
challenge-debugging-008debuggingPASS-1.09640 ms$0.000638stop
challenge-reasoning-001reasoningPASS-1.08118 ms$0.000507stop
challenge-reasoning-002reasoningPASS-1.012921 ms$0.004461stop
challenge-reasoning-003reasoningPASS-1.025932 ms$0.001354stop
challenge-reasoning-004reasoningPASS-1.06656 ms$0.000638stop
challenge-reasoning-005reasoningPASS-1.018303 ms$0.001702stop
challenge-rag-001rag_long_contextFAILmissing_required_info0.017687 ms$0.002182stop
challenge-rag-002rag_long_contextPASS-1.018657 ms$0.002880stop
challenge-rag-003rag_long_contextPASS-1.024378 ms$0.002048stop
challenge-rag-004rag_long_contextPASS-1.07912 ms$0.001218stop
challenge-if-001instruction_followingPASS-1.03247 ms$0.001856stop
challenge-if-002instruction_followingPASS-1.017514 ms$0.001722stop
challenge-if-003instruction_followingPASS-1.016034 ms$0.002468stop
challenge-zh-writing-001chinese_writingPASS-1.024272 ms$0.004309stop
challenge-zh-writing-002chinese_writingPASS-1.018698 ms$0.002716stop
challenge-coding-009codingPASS-1.015614 ms$0.004233stop
challenge-coding-010codingPASS-1.027149 ms$0.007463stop
challenge-coding-011codingPASS-1.010470 ms$0.003297stop
challenge-coding-012codingPASS-1.056186 ms$0.008427stop
challenge-coding-013codingPASS-1.014849 ms$0.003069stop
challenge-coding-014codingPASS-1.0184896 ms$0.055315stop
challenge-debugging-009debuggingPASS-1.07339 ms$0.001037stop
challenge-debugging-010debuggingPASS-1.09564 ms$0.001189stop
challenge-debugging-011debuggingPASS-1.018276 ms$0.000752stop
challenge-debugging-012debuggingPASS-1.04542 ms$0.000532stop
challenge-debugging-013debuggingPASS-1.07907 ms$0.001607stop
challenge-debugging-014debuggingPASS-1.010862 ms$0.001397stop
challenge-reasoning-006reasoningPASS-1.05877 ms$0.001065stop
challenge-reasoning-007reasoningPASS-1.05583 ms$0.001126stop
challenge-rag-005rag_long_contextPASS-1.07101 ms$0.001322stop
challenge-rag-006rag_long_contextPASS-1.04781 ms$0.001132stop
challenge-rag-007rag_long_contextPASS-1.08865 ms$0.001055stop
challenge-if-004instruction_followingPASS-1.010512 ms$0.002512stop
challenge-if-005instruction_followingPASS-1.014214 ms$0.002397stop
challenge-zh-writing-003chinese_writingPASS-1.068961 ms$0.002626stop
challenge-reasoning-008reasoningPASS-1.021198 ms$0.001012stop
challenge-reasoning-009reasoningPASS-1.011786 ms$0.002420stop
challenge-reasoning-010reasoningPASS-1.09136 ms$0.001945stop
challenge-reasoning-011reasoningPASS-1.020096 ms$0.001875stop
challenge-reasoning-012reasoningPASS-1.035194 ms$0.002242stop
challenge-reasoning-013reasoningPASS-1.013424 ms$0.000912stop
challenge-reasoning-014reasoningPASS-1.04773 ms$0.000484stop
challenge-reasoning-015reasoningFAILdegenerate_output0.0132212 ms$0.018126length
challenge-reasoning-016reasoningPASS-1.06043 ms$0.001422stop
challenge-reasoning-017reasoningPASS-1.06014 ms$0.001644stop
challenge-reasoning-018reasoningPASS-1.010259 ms$0.001367stop
challenge-reasoning-019reasoningPASS-1.012842 ms$0.003078stop
challenge-reasoning-020reasoningPASS-1.012987 ms$0.001868stop
challenge-reasoning-021reasoningPASS-1.012740 ms$0.001610stop
challenge-reasoning-022reasoningPASS-1.05972 ms$0.000621stop
challenge-reasoning-023reasoningPASS-1.016044 ms$0.001071stop
challenge-reasoning-024reasoningPASS-1.07150 ms$0.001355stop
challenge-reasoning-025reasoningPASS-1.09542 ms$0.001795stop
challenge-coding-015codingPASS-1.041685 ms$0.005393stop
challenge-coding-016codingPASS-1.014849 ms$0.002570stop
challenge-coding-017codingPASS-1.025722 ms$0.006138stop
challenge-coding-018codingPASS-1.030918 ms$0.002548stop
challenge-coding-019codingPASS-1.015534 ms$0.002389stop
challenge-coding-020codingPASS-1.028719 ms$0.006020stop
challenge-coding-021codingPASS-1.016456 ms$0.003309stop
challenge-coding-022codingPASS-1.044827 ms$0.008089stop
challenge-coding-023codingPASS-1.037001 ms$0.003015stop
challenge-coding-024codingPASS-1.071718 ms$0.004627stop
challenge-coding-025codingPASS-1.015098 ms$0.002782stop
challenge-coding-026codingPASS-1.022715 ms$0.006959stop
challenge-coding-027codingPASS-1.05205 ms$0.004252stop
challenge-coding-028codingPASS-1.021157 ms$0.006444stop
challenge-rag-008rag_long_contextPASS-1.08179 ms$0.001268stop
challenge-rag-009rag_long_contextPASS-1.019600 ms$0.001094stop
challenge-rag-010rag_long_contextPASS-1.011130 ms$0.001549stop
challenge-rag-011rag_long_contextPASS-1.06064 ms$0.001654stop
challenge-rag-012rag_long_contextPASS-1.015058 ms$0.001099stop
challenge-rag-013rag_long_contextPASS-1.03258 ms$0.001271stop
challenge-rag-014rag_long_contextPASS-1.05996 ms$0.001089stop
challenge-rag-015rag_long_contextPASS-1.016403 ms$0.002796stop
challenge-rag-016rag_long_contextPASS-1.06431 ms$0.000963stop
challenge-rag-017rag_long_contextPASS-1.034197 ms$0.001746stop
challenge-rag-018rag_long_contextPASS-1.016425 ms$0.001361stop
challenge-debugging-015debuggingPASS-1.013337 ms$0.000873stop
challenge-debugging-016debuggingPASS-1.014651 ms$0.001439stop
challenge-debugging-017debuggingPASS-1.019019 ms$0.001526stop
challenge-debugging-018debuggingPASS-1.05893 ms$0.000626stop
challenge-debugging-019debuggingPASS-1.07266 ms$0.000992stop
challenge-if-006instruction_followingPASS-1.028566 ms$0.002027stop
challenge-zh-writing-004chinese_writingPASS-1.035692 ms$0.003903stop