How we tested.

§ 01 — at a glance

The headline numbers, before the footnotes.

Aggregated test report — 2026-02-02. 320 runs. 1,265 individual suites executed. The pass rates below are the full denominator, including every model that failed to reach the bar.

§ 02 — provider-level breakdown

By provider, same data, no rollups.

The four model providers tested, with their model counts, total runs, and pass rates. xAI tops on suite rate; cohere lags on run rate due to a single failing model in a small sample. Aggregate numbers in §01 are the population totals; these are the strata.

§ 03 — three-tier autonomy framework

One protocol, three difficulty settings.

Each model is tested at three distinct levels of autonomy. The harder the test, the less the agent is given — at the top tier the model has only a wallet and a single discovery URL, and must learn the entire protocol from public documentation.

Every framework enforces the same constraints, regardless of tier:

• No hardcoded endpoints — only /.well-known/ discovery is permitted
• No SDK access for the lower-tier (real-world, autonomous) tests
• The agent must read documentation to learn the API
• The agent must figure out authentication from the docs alone
• The agent must handle errors and retries on its own

§ 04 — the cold-start exemplar

One run, thirteen steps, byte-for-byte verified.

A representative cold-start run from the test corpus — claude-sonnet-4 on 2026-01-22, given only a discovery URL and a payment-enabled Solana wallet. No prior knowledge of Nukez. No SDK. The agent figured out everything else by reading the published documentation.

Agent's final output (verbatim)

{
  "success": true,
  "task_complete": true,
  "summary": "Successfully stored and verified data externally
              using Nukez storage service",
  "storage_location": {
    "locker_id": "locker_55f499a55beb",
    "filename":  "agent-data.txt"
  },
  "verification_result": "EXACT_MATCH",
  "authentication_method": "Ed25519 signed envelope + capability token",
  "storage_protocol": "HTTP + x402 receipts on Solana",
  "tx_signature": "3Thvdxf5eLoFAPFp8ecwEE42sTRBe8km9PF8jkt5aKZ3QQ3KEkNq3mtgsADu8R8aoAWXCiU88eW3yuUcDEJYLKtp",
  "iterations": 21,
  "duration_seconds": 448.74
}

§ 05 — the four suites

What “passing” actually means.

Each model run executes four independent suites. A run is counted as “all-pass” only when every suite passes. Per-suite pass rates are below — the strict run rate (80.6%) sits well below the suite rate (95.1%) because a single failure in any suite fails the whole run.

§ 06 — reliability tiers

Per-model results, no curation.

Models are classified into three tiers based on aggregate pass rate. Tier 1 is production-ready for cold-start integration. Tier 2 needs the SDK abstraction or recent doc updates to clear the bar. Tier 3 is below the capability threshold for the agent-tool reasoning the protocol requires.

Tier 1 · Production ready (22 models)

Tier 2 · Doc updates / SDK recommended

Tier 3 · Below capability threshold

§ 07 — failure modes observed

Every failure mode, published with its root cause.

Across all 320 runs the failures clustered into four categories. Each is described below with its symptom, root cause, and current resolution status. Three of the four were addressable via documentation changes; the fourth is a model reasoning limitation that surfaces in the tier classification.

tool · build_signed_envelope
  ✓  OK
tool · api_request (without headers)
  ✗  422  (missing auth headers)

tool · build_signed_envelope
  args: { receipt_id: "...", method: "POST",
          path: "/v1/storage/signed_provision",
          ops: ["locker:provision"] }
  ✗  'body' parameter is required for POST requests

tool · upload_data
  ✗  403 Client Error: Forbidden for url:
      https://storage.googleapis.com/...

--- iteration 7 ---
tool · get_file
  ✗  400: BAD_ENVELOPE: body_sha256 missing
--- iteration 8 ---
tool · task_failed
  args: { reason: 'get_file failed with BAD_ENVELOPE…' }

§ 08 — documentation evolution

What testing taught the docs, and what shipped.

Three of the four failure modes in §07 were addressable via documentation changes. Each gap identified during testing was tracked, fixed in the canonical docs, and re-tested in the next round. The pass-rate improvements documented in §06 reflect those fixes.

§ 09 — reproducibility

Run it yourself.

The verify-first thesis applies to the benchmark too. Every datum on this page is reproducible from the public test harness — same gateway, same models, same wallet pattern. No private access path was used, and no result is gated behind credentials we can't share.

• Gateway. Production — https://api.nukez.xyz. The same URL every other consumer uses.
• Network. Solana mainnet. Real lamports out of a real wallet on every request — no faucet, no devnet, no simulation.
• Models. Available via the listed providers' public APIs. No private model access was used.
• Test framework. The three-tier harness lives in the public agent-testing repo — test_autonomous_discovery.py, test_real_world_agent.py, penultimate_agent_test.py.

§ 10 — scope & limits

What this does not prove.

Honest framing of what the test corpus says and doesn't say. The numbers are tight; the claim those numbers support is narrow.

• Not a model intelligence benchmark. It measures how reliably an agent can integrate with this protocol — not anything about general capability, reasoning, or quality of output.
• Not a storage performance benchmark. Throughput, latency, and durability live on the per-provider pages (/proof/benchmark). This page is exclusively about the agent-integration surface.
• Not an end-user UX benchmark. The agent is a proxy for “a competent autonomous integrator” — not for a human evaluating a UI.
• Tier classifications are best-attempt. A few models that are flaky show as Tier 2 on best-attempt; the underlying run distribution is published per-model in the aggregated report.
• Doc-updates were applied. Three of the four failure modes were addressed via documentation changes during the test window. Pass rates on the latest reports reflect those fixes.

§ 11 — session timeline (excerpt)

What the test cadence actually looked like.

A representative slice of the test session log — every entry is a single recorded run with timestamp, framework, model, outcome, and a one-line note. The full log spans multiple sessions over ~2 weeks; this excerpt shows the failure → fix → re-pass arc on gpt-5.1 (the header-attachment regression) and the cross-model cadence around it.

§ 12 — source artifacts

Read the raw reports.

This page is a synthesis. The underlying artifacts — the comprehensive analysis, the cold-start exemplar, and the aggregated test report — are the source of truth.

• Comprehensive analysis · 2026-01-25 · ~22,000 lines of test logs across the three frameworks. Full failure-mode taxonomy and per-model notes.
• Cold-start exemplar appendix · 2026-01-22 · The single canonical run reproduced in §03 above, with the full task, execution sequence, transaction signature, and receipt ID.
• Aggregated test report · 2026-02-02 · The 320-run, 27-model, 4-provider, 4-suite aggregate that produced every headline number on this page.

← back to proof · or jump to the benchmark matrix →