Model Benchmark Report · v2

Platform models vs international flagships (accuracy anchors)Test date: 2026-07-137 models × 25 questions

1. What this report answers

You may not know which model a customer used before — but by putting our three platform models, alongside today's strongest international flagships (Claude, GPT), through one identical question set and one identical grading standard, a customer gets an objective coordinate for the models running on our compute — how far the gap is, where the strengths are, and the value for money。

In this report the international flagships serve as “accuracy anchors” (a reference frame), not speed competitors — see the fairness note in §2.

Platform top score

24/25

kimi-k2.6

Platform fastest

8.3s/q

deepseek-v4-flash

Flagship full-mark line

25/25

gap to flagships below

2. Test environment & fairness design

Item	Details
Hardware	Standard test host (only issues API requests; inference runs remotely, so local specs do not affect results)
Software	macOS 12.7.6 · Python 3.14.5 · Codex CLI 0.140 · Claude CLI 2.1.195
Network	Home WiFi (5GHz); avg latency 33.5ms to the platform endpoint, 0% packet loss (see the v1.1 report)
Platform models	deepseek-v4-flash · kimi-k2.6 (w4a8 quantized) · qwen3.5-flash (forced reasoning), called via the platform API endpoint
Anchor · Claude	Claude Fable 5 + Claude Opus 4.8, via Claude CLI, reasoning effort: medium
Anchor · GPT	GPT-5.6-sol + GPT-5.5, via Codex CLI, reasoning effort: medium

Fairness controls (all reproducible)

Same questions, same prompts: All 7 models ran the exact same 25 questions, prompts word-for-word identical, each including a “no tools, no internet” instruction.
Tools disabled, single-turn: on the Claude/Codex CLI side all tools were disabled and MCP cleared, so every contestant sat a closed-book exam — otherwise it would be open-book vs closed-book.
Uniform medium reasoning: both CLI flagships were set to medium reasoning (not the highest tier, to avoid over-strong anchors); temperature was set to 0 wherever configurable to remove randomness.
Zero subjective grading: math checks the answer, coding runs the model's code locally against hidden test cases, instruction tasks are validated by a checker program, and function-calling compares JSON. The grader passed a self-check on 25 positive + 5 negative cases.
All questions original: all 25 are original (to avoid memorized answers); every reference answer was verified by a local program.
Fair truncation compensation: if a reasoning model burned its token budget on thinking without producing an answer, it was re-run with double the budget (32768); only if it still could not answer was it marked failed (see §5).

Speed caveat (important): Platform models are called over raw API, while Claude/GPT run as CLI subprocesses (a fixed 2–4s startup overhead). Therefore speed is only comparable among the three platform models; flagship timings are not compared directly against platform models — they serve only as accuracy anchors.

3. Total scores (7 models)

Platform modelsIntl. flagship (accuracy anchor)

4. Scores by section

Scores across the six sections. The gap between platform models and flagships shows up mainly at the edge cases of competition math and instruction following.

PlatformIntl. flagship

5. Key findings

Platform models are already close to international flagships: the top scorer, kimi-k2.6 scored 24/25(reaching 96%),of the full mark, just 1 question(s) behind a flagship's perfect score. On this competition-grade set, the accuracy gap between an affordable domestic model and the most expensive international flagship is already small.
deepseek-v4-flash is the value champion: 8.3s/q per question — the fastest of the three platform models, with first-tier accuracy. For customer support, translation, high-volume Q&A and agent workloads that need to be fast and cheap, this is the first choice.
kimi-k2.6 (quantized) can “fail to converge” on a few hard problems: on competition-math M2 (sum of n with φ(n)=24), even with the token budget doubled to 32768 it stayed stuck in a reasoning loop and produced no answer. This is a possible stability cost of w4a8 quantized deployment — worth noting for reasoning-heavy use.
qwen3.5-flash hits a “thinks-too-long-and-times-out” ceiling: forced deep reasoning makes it the slowest, and more importantly, on the hardest reasoning items (e.g. competition-math M2/M5) even a doubled 32768 budget could not stop its thinking from exceeding the ~600s server timeout and returning empty — these are not wrong answers but no answer at all. It suits unhurried deep analysis but not high-concurrency real-time use; stress-test timeouts before any reasoning-heavy workload.

About this report: this is a 2026-07-13 measurement snapshot. The issues surfaced here (a model not converging on very hard reasoning, timeout risk in forced-reasoning mode) have been reported straight back to the compute service provider, and the relevant model parameters are being tuned. We continuously test and track the models we run — closing the loop the moment an issue appears rather than trusting a one-off — which is why we publish these measured reports to customers.

6. Speed of the three platform models

Bars show average time on correctly-answered questions (excluding timeout failures, reflecting real answer latency). All three platform models use the same raw-API path, so the comparison is apples-to-apples.

Platform models	Avg time (correct qs)	Avg all qs (incl. timeouts)	Note
kimi-k2.6	37.8s	60.4s	incl. 1 not passed
deepseek-v4-flash	8.5s	8.3s	incl. 2 not passed
qwen3.5-flash	116.4s	174.6s	incl. 3 not passed

“Avg all qs” includes questions that returned empty after thinking past the ~600s server timeout; the large gap between the two columns for qwen3.5-flash is exactly its “thinks-too-long-and-times-out” risk on hard problems.

7. Per-question pass/fail matrix (7×25)

✅ pass ❌ wrong ⛔ error/no-output. Hover a cell for grading detail.

ID	Section	claude-fable-5	claude-opus-4-8	gpt-5.6-sol	kimi-k2.6	gpt-5.5	deepseek-v4-flash	qwen3.5-flash
M1	Competition math	✅	✅	✅	✅	✅	✅	✅
M2	Competition math	✅	✅	✅	❌	❌	✅	❌
M3	Competition math	✅	✅	✅	✅	✅	✅	✅
M4	Competition math	✅	✅	✅	✅	✅	✅	✅
M5	Competition math	✅	✅	✅	✅	✅	✅	❌
P1	Competition coding	✅	✅	✅	✅	✅	✅	✅
P2	Competition coding	✅	✅	✅	✅	✅	✅	✅
P3	Competition coding	✅	✅	✅	✅	✅	❌	❌
P4	Competition coding	✅	✅	✅	✅	✅	✅	✅
P5	Competition coding	✅	✅	✅	✅	✅	✅	✅
P6	Competition coding	✅	✅	✅	✅	✅	✅	✅
I1	Instruction following	✅	✅	✅	✅	✅	❌	✅
I2	Instruction following	✅	✅	✅	✅	✅	✅	✅
I3	Instruction following	✅	✅	✅	✅	✅	✅	✅
I4	Instruction following	✅	✅	✅	✅	✅	✅	✅
I5	Instruction following	✅	✅	✅	✅	✅	✅	✅
F1	Function calling	✅	✅	✅	✅	✅	✅	✅
F2	Function calling	✅	✅	✅	✅	✅	✅	✅
F3	Function calling	✅	✅	✅	✅	✅	✅	✅
F4	Function calling	✅	✅	✅	✅	✅	✅	✅
L1	Long-context	✅	✅	✅	✅	✅	✅	✅
L2	Long-context	✅	✅	✅	✅	✅	✅	✅
L3	Long-context	✅	✅	✅	✅	✅	✅	✅
C1	Chinese comprehension	✅	✅	✅	✅	✅	✅	✅
C2	Chinese comprehension	✅	✅	✅	✅	✅	✅	✅

8. Selection guide

Use case	Recommended	Why
Customer support / translation / high-volume Q&A / agent tool-calling	deepseek-v4-flash	fastest, cheapest, first-tier accuracy
Content creation / general-purpose tasks	kimi-k2.6	strong all-round, perfect on coding; note it is quantized and occasionally fails to converge on very hard reasoning
Deep reasoning / complex math analysis (not time-critical)	qwen3.5-flash	most solid reasoning, but slowest and highest token cost; reserve a large budget

Appendix A · The 25 questions, section rationale & reference answers

Section	Why we test this way
Competition math	AIME/competition level, single numeric answer. Used to break the ceiling — even flagships can get these wrong, which creates separation. Every answer verified by local brute-force computation.
Competition coding	LiveCodeBench-style algorithm problems. Graded by actually running the model's code locally against hidden test cases — the most discriminating and hardest-to-game section.
Instruction following	IFEval-style multi-constraint tasks (character count / keywords / format / sum all at once). Tests whether the model follows instructions exactly — directly relevant to agent / structured-output use.
Function calling	Given a tool schema, the model must output the correct call JSON. What agent customers care about most; graded by deep JSON comparison (city names accepted in Chinese or English).
Long-context	A ~15k-character fictional annual report with three probe types embedded: needle, multi-hop, and aggregation. Tests long-context retrieval and reasoning; single-answer match.
Chinese comprehension	Classical Chinese / idioms — the deep end of Chinese; single answer. Tests native-level Chinese understanding.

ID	Question (summary)	Reference answer
M1	How many trailing zeros does 2026! (factorial) have in decimal?	505
M2	Sum of all positive integers n with φ(n)=24, where φ is Euler's totient.	621
M3	Let S = 1·2¹ + 2·2² + … + 100·2¹⁰⁰. Find S mod 1000.	450
M4	How many integer triples (a,b,c) satisfy a+b+c=2026 with 1≤a≤b≤c?	342056
M5	Sequence: a₁=1, and for n≥1, aₙ₊₁ = aₙ + gcd(n, aₙ). Find a₁₀₀.	101
P1	Implement solve(intervals): given closed intervals (list of lists, possibly unordered/empty), merge all overlapping or touching intervals ([1,4] and [4,5] count as touching), return the merged list sorted by start.	see checker
P2	Implement solve(s): return the length of the longest palindromic substring of s. Empty string returns 0.	see checker
P3	Implement solve(expr): evaluate an arithmetic expression string and return an integer. Supports + - * /, parentheses, unary minus and arbitrary spaces; division is integer division truncated toward zero (note: (8-10)/3 = 0, not -1).	see checker
P4	Implement solve(coins, amount): number of combinations (order-independent) to make amount using coins (each usable unlimited times). Returns 1 when amount is 0.	see checker
P5	Implement solve(n, prerequisites): n courses 0..n-1; each [a,b] means b must precede a. Return the lexicographically smallest valid order (list); return [] if a cycle makes it impossible.	see checker
P6	Implement solve(heights): given histogram bar heights (non-negative ints, each width 1), return the largest rectangle area. Empty list returns 0.	see checker
I1	Write a cloud-product blurb of exactly 50 Chinese characters, containing both 「云端」 and 「安全」, with no punctuation, digits or Latin letters (tests exact Chinese output).	see checker
I2		see checker
I3	Write a 4-line Chinese acrostic poem: each line exactly 7 characters, no punctuation, first characters spelling 「算力出海」.	see checker
I4	Sort these 5 words in reverse alphabetical order: apple, banana, cherry, date, elderberry. Output 5 lines formatted “N) WORD”, N counting down 5→1, WORD uppercase.	see checker
I5	Translate a Chinese sentence into English with constraints: exactly two sentences; each ≤15 words; must contain the word “infrastructure”.	see checker
F1	Tool: get_weather(city: string, unit: "celsius"\|"fahrenheit"). User: “Check the temperature in Singapore right now, in Celsius.”	{"tool": "get_weather", "arguments": {"city": "Singapore", "unit": "celsius"}}
F2	Tool: book_meeting_room(room_id: string, start: ISO8601 e.g. 2026-01-02T09:00:00, duration_minutes: integer). User: “Book room A-301 on 2026-07-15 starting 2pm for one and a half hours.”	{"tool": "book_meeting_room", "arguments": {"room_id": "A-301", "start": "2026-07-15T14:00:00", "duration_minutes": 90}}
F3	Tools (pick exactly one): 1. send_email(to: string, subject: string, body: string) 2. create_task(title: string, due_date: string format YYYY-MM-DD, priority: "low"\|"medium"\|"high") 3. search_docs(keyword: strin…(long text omitted)	{"tool": "create_task", "arguments": {"title": "Prepare Q3 vendor review materials", "due_date": "2026-07-20", "priority": "high"}}
F4	Tool: create_order(customer_id: string, items: array where each element is an object {sku: string, qty: integer}). User: “Customer C88 orders: 2× K100, 1× M200, in that order.”	{"tool": "create_order", "arguments": {"customer_id": "C88", "items": [{"sku": "K100", "qty": 2}, {"sku": "M200", "qty": 1}]}}
L1	Read the annual report below and answer: what was the data center's average PUE in Q3 2025?	1.18
L2	Read the annual report and answer: what is the overseas division's annual budget (in 10k CNY)? The figure is not stated directly and must be inferred.	13600 / 1.36 / 13,600
L3	Read the annual report and answer: total 2025 revenue (in 100M CNY)? Only quarterly figures are given; sum them, two decimals.	11.98
C1	In Zhuge Liang's 《出师表》, the opening refers to 「先帝」 (the late emperor) — who is it? (Answer: Liu Bei)	Liu Bei
C2	The idiom 「汗牛充栋」 describes a great abundance of what? (Answer: books)	books

Appendix B · Reproducibility

The original question bank, machine graders, the three benchmark runners and all raw responses (each model's verbatim output, tokens and timing) are retained and available on request for independent third-party verification.
Grading is fully automated: math/long-context/Chinese match a single answer, coding runs the model's code in an isolated environment against hidden tests, and instruction/function-calling use checker programs.