The number everyone quoted is the least interesting one

Semgrep ran an open-weight model against the same IDOR benchmark it uses on frontier coding agents, and GLM 5.2 scored 39% F1, beating Claude Code at 32%, for roughly $0.17 per vulnerability found. That's the line getting passed around. It's also the wrong thing to take away.

Two sentences later in the same post: Semgrep's own multimodal pipeline scored 53–61% F1 on the identical task. The purpose-built harness beat every raw model by a wide margin, and that is the finding that matters.

IDOR detection F1 score (%)
IDOR detection F1 score (%)

What actually happened

GLM 5.2 is Zhipu AI's latest open-weight model: MIT-licensed, a Mixture-of-Experts with about 750 billion total parameters but only roughly 40 billion active per token, with usable context pushed from 200K to 1M. It shipped to GLM Coding Plan members on June 13, 2026, with weights following on June 16. In Semgrep's test it ran inside a plain Pydantic AI harness with the same IDOR prompt every other model got, no endpoint discovery and no guided navigation. With none of that scaffolding it still edged out Claude Opus 4.8 among bare-prompt models.

So two results sit in one post. A bare-prompt open-weight model got within striking distance of frontier agents. And the harness still won. The second is the one that answers the question Semgrep said it set out to ask: how much of vulnerability detection comes from the model versus the scaffolding around it.

Model versus harness

The strongest critique landed fast. WithinReason put it plainly: "Claude Code is an agent harness, not an LLM. Claude is a brand, not an LLM." [1] That isn't pedantry. The comparison is GLM-5.2-in-a-Pydantic-harness against Claude-Code-the-harness, two different scaffolds wrapping two different models, reported as if it were model against model. admax88qqq added the related complaint that a benchmark headline carrying no precise model number invites skepticism by default. [2] The body does name Opus 4.8 and Claude Code. The headline doesn't, and headlines are what travel.

danslo went further, calling it an ad and noting that IDORs are arguably the easiest class of vulnerability to find. [3]

“It reads like an ad. Secondly these are "just" IDORs, arguably the easiest class of vulnerabilities. Thirdly it compares to GPT 5.5 and Opus 4.8. No, we don't have Mythos at home.”
— danslo on Hacker News

He's right that IDOR is a narrow, structured task: access-control checks across endpoints, not memory-corruption chains. A 39% F1 on the easy class is not evidence about the hard ones.

The obsolescence question

veselin raised the sharpest builder question: if a single "find IDOR" prompt gets you 39%, why not spin up subagents in Claude Code, Codex, or OpenCode and reproduce what Semgrep's harness does, making the product obsolete? [4] The benchmark answers him, against itself. The roughly 20-point gap between the bare prompt and the purpose-built pipeline is exactly the value of endpoint enumeration, context selection, and guided navigation. Reproducing that isn't a prompt, it's the harness Semgrep sells. Read that way, the post is a product demo wearing a model-comparison costume.

Where I'd push Semgrep the other way: the open-weight angle is underplayed. An MIT-licensed model you can run entirely inside your own environment is a real advantage for security teams handling sensitive code, and that, not seven F1 points, is the genuine reason to care about GLM 5.2 over a closed frontier model.

The China-cyber subtext

himata4113 found the numbers low against his own Windows kernel work, and said it wouldn't surprise him anymore if Chinese labs surpassed US public models in narrow categories like cyber, at lower cost and already capable enough to assist their own training. [5] That anxiety is the whole frame. The title riffs on AriaWestcott's viral claim that Anthropic built a "Claude Mythos Preview" so good at finding vulnerabilities and designing exploits that it declined to release it. [6] "We have Mythos at home" is the joke: an open model, on your hardware, doing security work. solenoid0937 took it to the next step and predicted US export controls on open models like GLM within months. [7]

My advice

If you build vulnerability detection, treat the model as a commodity input and spend your effort on the harness. The 20 points between a bare prompt and Semgrep's pipeline is where the performance actually lives, and that's reproducible engineering, not a better foundation model.

If you're evaluating GLM 5.2 specifically: it's free to try and open weight, so run it on your own infrastructure for sensitive code. [8] That isolation is its edge over Claude here, far more than the benchmark margin.

And as a reader: discount any "X beats Claude" headline until you know which model and which harness sat on each side. In this case the honest version is narrower and more useful than the viral one. The harness is the moat, and Semgrep just published the receipt.

Sources

Primary source: We have Mythos at Home: GLM 5.2 beats Claude in our Cyber Benchmarks

  1. HN comment by WithinReason — WithinReason · community
  2. HN comment by admax88qqq — admax88qqq · community
  3. HN comment by danslo — danslo · community
  4. HN comment by veselin — veselin · community
  5. HN comment by himata4113 — himata4113 · community
  6. Tweet by @AriaWestcott — @AriaWestcott · community
  7. HN comment by solenoid0937 — solenoid0937 · community
  8. Tweet by @israfill — @israfill · community
  9. HN comment by kordlessagain — kordlessagain · community
  10. HN comment by rode1974 — rode1974 · community
  11. Tweet by @hey_madni — @hey_madni · community