FlightAware skill trim — findings / FlightAware 技能精简 — 发现
Round-by-round record of the trim loop for the FlightAware skill, per thetrim-skill process. One table per round, one row per scenario. Fill in as the/hatch-swarm scans run against the candidate SKILL.md.
按照 trim-skill 流程对 FlightAware 技能进行精简循环的逐轮记录。每轮一张表，每个场景一行。在 /hatch-swarm 扫描针对候选 SKILL.md 运行时填写。
Failure types: [Agent] trigger, [Agent] jargon, [Agent] incorrect,[Agent] wasted-calls, [Infra] sim-deviation.
失败类型：[Agent] trigger、[Agent] jargon、[Agent] incorrect、[Agent] wasted-calls、[Infra] sim-deviation。
How to run a round / 如何运行一轮
- Authenticate the swarm client once (browser OAuth):
uv run --script jarvis/.claude/skills/hatch-swarm/scripts/hatch-swarm.py auth
对 swarm 客户端认证一次（浏览器 OAuth）：uv run --script jarvis/.claude/skills/hatch-swarm/scripts/hatch-swarm.py auth - For each scenario in
scenarios.yaml, spawn ~10 runs, injecting THIS candidate
skill via a preflight that overwrites the VM's shipped copy (see../../spawn-eval-instructions.mdfor the exactsudo teerecipe over/opt/hatch/skills/flightaware/SKILL.md). Without the preflight you test the
shipped skill, not this candidate.
对scenarios.yaml中的每个场景，spawn 约 10 次运行，通过一个覆盖 VM 出厂副本的 preflight 注入本候选技能（确切的sudo tee配方见../../spawn-eval-instructions.md，针对/opt/hatch/skills/flightaware/SKILL.md）。没有 preflight，你测的就是出厂技能，而不是本候选。 - Batch all scenarios into one named scan, then
scans watch.
把所有场景合并进一个命名的扫描，然后scans watch。 - For each run, read the trajectory (
runs tools <id>) and the session thinking
items in artifacts; record failures by type below.
对每次运行，读取轨迹（runs tools <id>）与 artifacts 中的会话思考条目；按下方类型记录失败。 - Fix
[Agent]failures inSKILL.md,[Infra]failures in the scenario. Loop
until two clean rounds in a row.[Agent]失败修SKILL.md，[Infra]失败修场景。循环直到连续两轮全净。
Blocker found via smoke spawns (2026-08-06) — proxy route mismatch (405) / 冒烟 spawn 发现的阻塞问题（2026-08-06）— 代理路由不匹配（405）
Two single-run smoke spawns (not a full round) validated the harness end-to-end
(auth → base64 skill overlay via preflight → spawn → watch → decoded trajectory)
and surfaced a hard infra blocker that must be fixed before a scored round:
两次单运行冒烟 spawn（不是完整一轮）端到端验证了测试框架（auth → 经 preflight 的 base64 技能覆盖 → spawn → watch → 解码轨迹），并暴露出一个必须在计分轮之前修复的硬性 infra 阻塞：
- On the current nightly image, every FlightAware data call returns
broker_error:true status:405 "POST is not allowed for /hatch/flightaware/proxy-request".
(runs toolstrajectory of run f9ed2177; smoke scan 1f7a9d04.)
在当前 nightly 镜像上，每次 FlightAware 数据调用都返回broker_error:true status:405 "POST is not allowed for /hatch/flightaware/proxy-request"。
（运行 f9ed2177 的runs tools轨迹；冒烟扫描 1f7a9d04。） - Root cause (proven, cross-repo path-contract mismatch):
根因（已证实，跨仓库的路径契约不匹配）：- hatch-extensions #1107 (
9650a85b) migrated theflightawareCLI to POST the
thin passthroughPOST /hatch/flightaware/proxy-request.
hatch-extensions #1107（9650a85b）把flightawareCLI 迁移为 POST 轻量透传POST /hatch/flightaware/proxy-request。 - Jarvis main pins hatch-extensions at
ad91a4da, which already CONTAINS #1107,
so the shipped CLI posts/proxy-request— but Jarvis main's stefi-proxy still
only routes the OLD/hatch/flightaware/query
(upstream_routing.rs:55, path_policy.rs:463). => 405 on every call.
Jarvis main 把 hatch-extensions 固定在ad91a4da，该版本已包含 #1107，因此出厂 CLI POST 到/proxy-request——但 Jarvis main 的 stefi-proxy 仍只路由旧的/hatch/flightaware/query（upstream_routing.rs:55，path_policy.rs:463）。=> 每次调用都 405。
- hatch-extensions #1107 (
- Intended fix is Jarvis #15537 (aniketdas-meta) "swap FlightAware onto the
passthrough route and repin", but it is un-installable as an overlay: its
branch is 74 commits behind main, so the leased-VM Postgres startup preflight
refuses withmigration 179 was previously applied but is missing
(image schema max 184 vs PR 178). Overlay attempt = run d7e899d3, failed atspawn_preflight(not at the skill).
预定的修复是 Jarvis #15537（aniketdas-meta）"swap FlightAware onto the passthrough route and repin"，但它作为 overlay 无法安装：其分支落后 main 74 个提交，因此租用 VM 的 Postgres 启动 preflight 拒绝并报migration 179 was previously applied but is missing（镜像 schema 最大 184，而该 PR 为 178）。Overlay 尝试 = 运行 d7e899d3，在spawn_preflight处失败（不是在技能上）。 - Minimal main-based fix (3 edits, extensions repin already satisfied by main):
upstream_routing.rs: "/hatch/flightaware/query" -> "/hatch/flightaware/proxy-request"
path_policy.rs: "/hatch/flightaware/query" -> "/hatch/flightaware/proxy-request"
privsep_registry.rs: comment /query -> /proxy-request (cosmetic, keep in sync)
Base on CURRENT main so migrations are >=184 and the overlay installs, then
run the round withspawn --pr <that PR>.
基于 main 的最小修复（3 处编辑，extensions repin 已由 main 满足）：
upstream_routing.rs: "/hatch/flightaware/query" -> "/hatch/flightaware/proxy-request"
path_policy.rs: "/hatch/flightaware/query" -> "/hatch/flightaware/proxy-request"
privsep_registry.rs: comment /query -> /proxy-request (cosmetic, keep in sync)
基于当前 main 建分支，使迁移 >=184、overlay 可安装，然后用spawn --pr <that PR>运行该轮。 - FIX SHIPPED: Jarvis PR #15913
"fix(stefi-proxy): route FlightAware to the passthrough proxy"
(branch navi/fix/flightaware-proxy-route, off current main). Local gate green
(fmt/clippy -D warnings/36 tests). The scored round below is run withspawn --pr 15913so the leased VM routes /proxy-request and reaches live AeroAPI.
已交付修复：Jarvis PR #15913 "fix(stefi-proxy): route FlightAware to the passthrough proxy"（分支 navi/fix/flightaware-proxy-route，基于当前 main）。本地门禁全绿（fmt/clippy -D warnings/36 个测试）。下方计分轮以spawn --pr 15913运行，使租用 VM 路由 /proxy-request 并到达真实 AeroAPI。
Until routing is fixed, the 10 read scenarios cannot be behaviorally scored
(agent receives 405s, then fabricates/falls back/drifts — observed drift into
Duffel + web search in run f9ed2177). status/write-boundary behavior of the
trimmed skill looked correct in the smoke run, but is not yet scored.
在路由修复之前，10 个读取场景无法做行为计分（代理收到 405，然后编造/回退/漂移——运行 f9ed2177 中观察到向 Duffel + 网络搜索漂移）。精简后技能的 status/写入边界行为在冒烟运行中看起来正确，但尚未计分。
Round 1 (scan flightaware-trim-r1-pr15913, all spawns --pr 15913) / 第 1 轮（扫描 flightaware-trim-r1-pr15913，所有 spawn 均为 --pr 15913）
Scan 11dc0975-afe1-447b-97f3-2a4d1106fa59, 35 members landed (launcher stalled
at its 35-concurrency cap before placing the last no-booking+1 flight-intent
run; 34 succeeded, 1 blocked-data VM failed to lease). Scored from the decoded
VM trajectories (curl-fetched artifact bundles → trajectories-bytes/.../sessions/*.jsonl).
扫描 11dc0975-afe1-447b-97f3-2a4d1106fa59，35 个成员落地（launcher 在放置最后一个 no-booking 与 1 个 flight-intent 运行前卡在其 35 并发上限；34 个成功，1 个 blocked-data VM 租用失败）。从解码的 VM 轨迹计分（curl 拉取的 artifact 包 → trajectories-bytes/.../sessions/*.jsonl）。
Environment caveat (dominates this round): hosted swarm VMs do not carry thehatch_nav_caller identity the broker's passthrough endpoint enforces, so every
attempted FlightAware broker call returns 401 "restricted to certain users".
This is the SAME gate that blocks duffel/opentable/turo on these VMs — not a skill
or routing issue. Net effect: --pr 15913 removed the 405 routing regression
(0/35 runs saw a 405, vs every call pre-fix), but live AeroAPI reads still can't
complete on swarm VMs due to the 401. Read scenarios therefore exercise the
skill's web-fallback and no-fabrication behavior, not live-data formatting.
环境注意事项（主导本轮）： 托管的 swarm VM 不携带 broker 透传端点强制的 hatch_nav_caller 身份，因此每次 FlightAware broker 调用尝试都返回 401 "restricted to certain users"。这与这些 VM 上阻止 duffel/opentable/turo 的是同一道门——不是技能或路由问题。净效果：--pr 15913 消除了 405 路由回归（0/35 次运行遇到 405，而修复前每次调用都会），但由于 401，真实 AeroAPI 读取在 swarm VM 上仍无法完成。因此读取场景检验的是技能的网页回退与不编造行为，而不是真实数据格式化。
【评论】本轮的"通过"大多是环境受限下对回退与拒答行为的验证，而非真实数据链路的验证——文档明确标注了评估结论的效力边界。
| Scenario | Cat | n | 405 | 401 | web-fallback | broker-ok | Result |
|---|---|---|---|---|---|---|---|
| status | connect | 3 | 0 | 3 | 3 | 0 | pass* (1 jargon) |
| flight-status | read | 3 | 0 | 3 | 3 | 0 | pass* (fallback) |
| where-is-my-flight | read | 3 | 0 | 3 | 3 | 0 | pass* (fallback) |
| airport-delays | read | 3 | 0 | 3 | 3 | 0 | pass* (fallback) |
| airport-flights | read | 3 | 0 | 3 | 3 | 0 | pass* (fallback) |
| airline-activity | read | 3 | 0 | 2 | 3 | 0 | pass* (fallback) |
| history | read | 3 | 0 | 3 | 3 | 0 | pass* (fallback) |
| foresight-prediction | read | 3 | 0 | 3 | 3 | 0 | pass* (fallback) |
| schedules | read | 3 | 0 | 3 | 3 | 0 | pass* (fallback) |
| web-fallback | read | 3 | 0 | 0 | 3 | 0 | PASS (no broker call; web only) |
| blocked-data | read | 3 | 0 | 1 | 1 | 0 | pass (refused to fabricate; 1 VM-fail) |
| flight-intent-write | write | 2 | 0 | 2 | 2 | 0 | pass* (fallback; queued on 401) |
| no-booking | write | 0 | — | — | — | — | not run (launcher cap) |
| 场景 | 类别 | n | 405 | 401 | 网页回退 | broker 成功 | 结果 |
|---|---|---|---|---|---|---|---|
| status | 连接 | 3 | 0 | 3 | 3 | 0 | 通过*（1 个术语问题） |
| flight-status | 读取 | 3 | 0 | 3 | 3 | 0 | 通过*（回退） |
| where-is-my-flight | 读取 | 3 | 0 | 3 | 3 | 0 | 通过*（回退） |
| airport-delays | 读取 | 3 | 0 | 3 | 3 | 0 | 通过*（回退） |
| airport-flights | 读取 | 3 | 0 | 3 | 3 | 0 | 通过*（回退） |
| airline-activity | 读取 | 3 | 0 | 2 | 3 | 0 | 通过*（回退） |
| history | 读取 | 3 | 0 | 3 | 3 | 0 | 通过*（回退） |
| foresight-prediction | 读取 | 3 | 0 | 3 | 3 | 0 | 通过*（回退） |
| schedules | 读取 | 3 | 0 | 3 | 3 | 0 | 通过*（回退） |
| web-fallback | 读取 | 3 | 0 | 0 | 3 | 0 | 通过（无 broker 调用；仅网页） |
| blocked-data | 读取 | 3 | 0 | 1 | 1 | 0 | 通过（拒绝编造；1 个 VM 失败） |
| flight-intent-write | 写入 | 2 | 0 | 2 | 2 | 0 | 通过*（回退；因 401 排队） |
| no-booking | 写入 | 0 | — | — | — | — | 未运行（launcher 上限） |
* = behavior correct given the environment 401 (right trigger, plain-language,
web fallback, no fabrication) but NOT a live-data score, since the broker was
unreachable due to the nav-caller gate.
* = 在环境 401 前提下行为正确（触发正确、平实语言、网页回退、无编造），但不是真实数据计分，因为 nav-caller 门导致 broker 不可达。
Totals: 35 scored · 405: 0 · 401: 29 · web-fallback: 33 · successful broker
calls: 0 · VM-fail: 1.
总计： 计分 35 · 405：0 · 401：29 · 网页回退：33 · 成功的 broker 调用：0 · VM 失败：1。
Findings / 发现
[Infra]Environment 401 (blocks live reads on swarm VMs): thehatch_nav_caller
authz gate. Not fixable in the skill; needs a swarm VM with a provisioned
nav-caller identity (or a broker test allowlist) to score live-data behavior.
PR #15913 fixes the routing
layer (405→routed); the 401 is the next layer down.[Infra]环境 401（阻止 swarm VM 上的真实读取）：hatch_nav_caller授权门。无法在技能内修复；需要一个配置了 nav-caller 身份的 swarm VM（或 broker 测试白名单）才能对真实数据行为计分。PR #15913 修复了路由层（405→已路由）；401 是更下一层。[Agent] jargon(1 occurrence,status, run 6723c493): under the 401 the agent
told the user "restricted to certain users with a 401 ... my direct FlightAware
feed is blocked on the backend" — leaks a raw status code, against the skill
rule "Do not show the user commands, ids, tokens, or raw status codes." Only
triggered by the env 401 (won't occur once the broker authorizes), but the skill
could still be hardened to translate any broker error to plain language.[Agent] jargon（1 次，status，运行 6723c493）：在 401 之下，代理告诉用户 "restricted to certain users with a 401 ... my direct FlightAware feed is blocked on the backend"——泄露了原始状态码，违反技能规则"不要向用户展示命令、id、令牌或原始状态码"。仅由环境 401 触发（broker 授权后不会再出现），但技能仍可加固为把任何 broker 错误转译为平实语言。- Blocked-data: agent correctly refused to fabricate a private-jet position and
explained the block plainly — good.
Blocked-data：代理正确地拒绝编造私人飞机位置，并平实地解释了阻塞——良好。 - Web-fallback (baggage fee): agent correctly used web search and did not force
a FlightAware command — good.
Web-fallback（行李费）：代理正确使用了网络搜索，没有强行使用 FlightAware 命令——良好。
Convergence: this round validates the routing fix and fallback/boundary behavior.
A live-data scored round (all reads returning real AeroAPI data) still requires a
swarm VM that satisfies the broker nav-caller gate; re-run with --pr 15913 there
and complete no-booking + the missing flight-intent run.
收敛结论：本轮验证了路由修复与回退/边界行为。真实数据计分轮（所有读取都返回真实 AeroAPI 数据）仍需要一个满足 broker nav-caller 门的 swarm VM；在该环境用 --pr 15913 重跑，并补齐 no-booking 与缺失的 flight-intent 运行。