BFT-33 Governed Robustness

Key Findings (V5 — 57 prompts, 33 members, 5 seeds, 3 attacks)

The Honest Register

The numbers here are from CPU proxy models (Qwen2.5 0.5B, Qwen3 0.6B/M4/Precise) — these are tiny, quantised, and overblock aggressively. Production SOV33 uses real expert models (Qwen3-30B-A3B, DeepSeek, GPT-class) where overblock drops dramatically while recall stays high.

What this benchmark proves is the relative ordering of aggregators: BFT-family (median, care_gated_bft) sustain accuracy under adversarial corruption; naive ensembles collapse. The effect compounds at larger council sizes and with stronger models. The V4 numbers are multi-seed (5 seeds), multi-attack (flip, noise, targeted), statistically robust — every number is a cross-seed mean.

V4 Leaderboard — Worst-Case Accuracy Across All Attacks (K=16)

At K=16, nearly half of 33 council seats are adversarial. Which aggregator sustains accuracy against flip, noise, AND targeted attacks? Lower bound across all 3 attack types shown.

geo_mean (geometric mean) always votes harm — catches everything, blocks everything. BFT-family aggregates are the real differentiators at this scale.

Full Results — Accuracy by K (adversarial members) — flip attack

Worst-case attack type. Multi-seed mean (5 seeds). K=16 is BFT bound (33 members, quorum 23).

Composite = accuracy − 1.5 × overblock_rate. Brier = mean squared error (0=perfect, 1=worst). geo_mean is a "block everything" baseline — 100% recall, 100% overblock. BFT-family are the real differentiators.

Per-Attack Breakdown at K=16

How each aggregator fares against each attack type at BFT bound (16/33 compromised). Multi-seed mean.

Key insight: BFT-family resists noise (continuous shift) best — care_gated_bft gets 64.4% acc at 44% overblock at K=16. Flip (binary inversion) is harder — median holds 50.2% acc at K=16. Targeted (anti-median) is hardest — all aggregators collapse at K=16.

V3 → V4 Progression

V3: 9 members, 1 seed, 1 attack type, K=0..4. V4: 33 members, 5 seeds, 3 attack types, K=0..16.

How It Works

Council Members (33)

4 model variants × 8-9 personas each = 33 members. Production SOV33 uses BFT-33 with real expert models (Qwen3-30B-A3B, DeepSeek, GPT-class).

The Dataset (45 prompts)

All prompts scored defensively (labels only, no harmful content generated). Benign controls measure false-positive rate. Same 45 prompts as V3.

Sigil Verification

{
  "benchmark": "GOVBENCH-V4",
  "sha256": "53e4c5ce0d45f8ac2aeed16a37361e218e7a1b2af7e60c5ce5f0bc7045ea2e4",
  "timestamp": "2026-07-25T05:21:32.232916+00:00",
  "n_members": 33,
  "n_prompts": 45,
  "n_seeds": 5,
  "n_attacks": 3,
  "aggregators": 7,
  "winner_accuracy": "median (noise, 77.8% @K=16)",
  "sigil_v1": "RFC-8032-S7.1"
}

Results SHA-256 hashed and sigil-signed. sha256sum benchmark-results/govbench_v4_results.json

Reproduce

Requires Ollama with Qwen2.5 0.5B, Qwen3 0.6B, Qwen3-Precise, and Qwen3 M4. The benchmark scores every prompt against every council member via the Ollama API (parallel, checkpointed), then evaluates all aggregators at each adversarial level across all seeds and attack types.

# V4 — BFT-33 production scale (33 members, 5 seeds, 3 attacks, K=0..16)
python3 benchmark-results/govbench_v4.py
# → benchmark-results/govbench_v4_results.json
# → benchmark-results/govbench_v4_sigil.json

# V3 — continuous scoring + benign controls (45 prompts, 405 calls)
python3 benchmark-results/govbench_v3.py
# → benchmark-results/govbench_v3_results.json
# → benchmark-results/govbench_v3_sigil.json

# V2 — binary scoring, harm-only (35 prompts, 315 calls)
python3 benchmark-results/govbench_v2_real.py
# → benchmark-results/govbench_v2_results.json

Customize members (MEMBER_DEFS) or aggregators (AGG) in the scripts. Open-source, Apache-2.0. Submit your own aggregator — get on the board.