SWE-bench Verified, an auna a kan DGX Spark guda
A 2026-10-08, wakilin Ragon 1.5.1 tare da Qwen3.8-27B ya warware ayyuka 68 cikin 100 na SWE-bench Verified a kan DGX Spark guda, ba tare da intanet ba: tsayayyen samfurin ayyuka 100, ƙoƙari ɗaya a kowane aiki, yanayin da aka saba. Idan aka mayar da shi ga dukan ayyuka 500, kimantawa ce ta ≈69% (kewayon 95% na 63.9–73.4%). Wannan aunawarmu ce, a na'urarmu kuma a kan ɓangare na jerin, ba shigarwa a leaderboard ba.
| Tsari | Ayyuka 100 iri ɗaya | Dukan ayyuka 500 |
| Ragon 1.5.1 + Qwen3.8-27B (NVFP4), DGX Spark guda, ba tare da intanet ba1 | 68/100 | ≈69% kimantawa |
| Ragon 1.5.1 + Qwen3.8-27B (NVFP4), yanayin refine, DGX Spark guda, ba tare da intanet ba1 | 69/100 | ≈70% kimantawa |
| Qwen3-Coder-480B-A35B (OpenHands)2 | 68/100 | 69.6% |
| GLM-4.6 (wakilin Z.ai)3 | 68/100 | 68.2% |
| Kimi K2.5, high (mini-SWE-agent)4 | 70/100 | 70.8% |
| DeepSeek V3.2, high (mini-SWE-agent)5 | 70/100 | 70.0% |
| Claude Opus 4.5, wakilai huɗu6 | 77–80/100 | 76.8–79.2% |
| Gemini 3 Pro (live-SWE-agent)7 | 79/100 | 77.4% |
| Ƙananan samfuran buɗaɗɗen tushe (parameters biliyan 20–40) |
| Qwen3-Coder-30B-A3B (EntroPO + R2E), test-time scaling8 | 60/100 | 60.4% |
| Qwen3-Coder-30B-A3B (EntroPO + R2E), ƙoƙari ɗaya8 | 56/100 | 52.2% |
| Devstral Small 2512 (mini-SWE-agent)9 | 57/100 | 56.4% |
| Kayayyakin wakilan rubuta lamba, kamar yadda masu samar da su ke wallafa su |
| GitHub Copilot agent mode (VS Code), tare da Claude 3.7 Sonnet10 | ba a wallafa ba | 56.0% |
| Claude Code11 | ba a wallafa ba | ba a wallafa ba |
| Cursor12 | ba a wallafa ba | ba a wallafa ba |
Ka karanta shi a matsayin kwatancen abokan tafiya, ba jerin matsayi ba. A kan ayyuka 100 iri ɗaya Ragon ya warware adadin da Qwen3-Coder-480B da GLM-4.6 suka warware, samfuran da suka fi shi girma sau da yawa, amma kowane layi yana amfani da wakili daban, kayan aiki daban kuma, ga layukan leaderboard, hotunan gwaji na x86 na hukuma, yayin da namu ya yi aiki a kan hotunan arm64 na al'umma. A yanayin refine zaman farko yana nazarin aikin ba tare da canza komai ba, sabon zama kuma ya yi shi: ya warware aiki ɗaya fiye da yanayin da aka saba (69 da 68, kowane ɓangare yana cin ayyukan da ɗayan ya rasa), da kusan sau 2.2 na lokaci a kowane aiki, don haka ya kasance na zaɓi (ragon --refine).
Yadda ake kimanta ≈69%
Ayyuka 100 kashi ɗaya cikin biyar ne na Verified, don haka adadin dukan jerin kimantawa ne. An ɗora samfurin item-response (Rasch) mai parameter ɗaya a kan sakamakon kowane aiki na dukan gabatarwar Verified 175 na jama'a: kowane aiki yana da wahalarsa kuma kowane tsari yana da iyawarsa. Iyawar Ragon ita ce wadda ke bayyana ayyuka 68 da aka warware a kan waɗannan 100; ≈69% shi ne makin da ake tsammani na wannan iyawa a kan dukan 500 (regression na yau da kullum a kan makin da aka wallafa ya yarda cikin maki ɗaya). Don gwada shi, an ba kowace gabatarwar jama'a maki a kan waɗannan ayyuka 100 kaɗai, aka hasashen makinta na gaba ɗaya haka nan: kuskuren ya kasance maki 2.4 (RMSE), kuma kewayon da ke sama ±1.96 ne na wannan. Kowane lamba a teburin scripts/swe_bench_compare.py yana sake samar da ita daga bayanan kowane aiki na jama'a na SWE-bench/experiments.
An rarraba ayyuka 100 a repositories na Verified daidai gwargwado da dukan jerin, a cikin ayyukan da muhallin gwajinsu ke aiki kuma ya tabbata a arm64 (gold patch yana wucewa, canji maras komai yana faɗuwa). Tsarukan da aka wallafa suna samun maki 1.0 zuwa 1.5 ƙasa a kan waɗannan 100 fiye da a kan dukan 500, don haka samfurin ba mai sauƙi ba ne. Ba shigarwa a leaderboard ba ne: tawagar SWE-bench ce ke ba su maki a kan nasu kayan aiki.
Dukan lambobin leaderboard: swebench.com da repository na SWE-bench/experiments; "ayyuka 100 iri ɗaya" sakamakon kowane aiki ne da kowace gabatarwa ta wallafa, da aka taƙaita ga ayyukan da Ragon ya gudanar.