I. VITAL SIGNS

各項目健康度概覽

項目二: 情境調用與完成 (10 cases)

各維度品質分 · 覆蓋率

走對流程召回

scenario

70.0%

[40.0% – 90.0%] · n=10

F1 70.0% · 召回 70.0% · 精確 70.0%

10 / 10 適用

用對工具召回

tool_usage

30.0%

[10.0% – 60.0%] · n=10

F1 30.0% · 召回 30.0% · 精確 100.0%

10 / 10 適用

答得到位平均

answer_quality

40.3%

[25.7% – 55.0%] · n=10

10 / 10 適用

II. RUN ECONOMICS

本次 Run 的 LLM 成本概覽

Judge 總成本（USD） $0.0064 18 次 judge 呼叫

Judge 總 Token 數 53,646 輸入 50,258 · 輸出 3,388

LLM 成本來源追蹤狀態
來源	追蹤狀態	數值	備註
Eris Case Generator	已追蹤	$0.0261 187,117 tokens · 114 calls	案例生成階段 LLM 呼叫（sampler / info_held / validator / kb_matcher / response_rules judge）— Round 8 起 CostTracker 自動寫入 EvalCaseGeneration 對應欄位
Eris LLM Judge	已追蹤	$0.0064 53,646 tokens	EvalJudgement 表記錄真實 OpenAI usage（faithfulness / answer_quality / retrieval_relevance 三類 judge）
Neptune Bot Inference	受阻	—	Neptune `unified_llm.tool_calling_stream` streaming 模式下 final chunk 未帶 usage（OpenAI `stream_options.include_usage` 未啟用），導致 `metadata.usage` 全 0。Eris 端無法從 chat response 重建。Fix 在 Neptune team 的 tracking ticket 內。
UserSimulator	尚未實作	—	UserSimulator multi-turn LLM 呼叫目前 Eris 沒寫追蹤；這部分要新增 service + 在 simulator client 裡寫入 token 紀錄

Judge	次數	Input	Output	Cost
忠實度（faithfulness）	7	10,604	808	$0.0015
回答品質（answer_quality）	11	39,654	2,580	$0.0049
檢索相關性（retrieval_relevance）	0	0	0	$0.0000

III. ACTIONABLE FINDINGS

優先處理項目

依「最低分 sub-metric」排序的 worst-3 case，標示出此 run 中最該優先檢視的 scenario × 維度組合。每筆同時列出實際觸發低分的 sub-metric 名稱，方便定位是 KB 漏洞、工具排序問題、還是規則違反。

01

Knowledge and Product Search 忠實度

0.0% n=6 最低 sub_metric： rule_compliance
02

Transfer to Human Agent 忠實度

0.0% n=1 最低 sub_metric： rule_compliance
03

Knowledge and Product Search 工具使用

30.0% n=10 最低 sub_metric： tools_recall

IV. DEEP DIVE

全段詳細數據

# 兩項目評估報告

- **EvalRun**: #137（suite #77 `auto-85ebd775-r1-053958840`）
- **Target**: #1 production-baseline
- **Cases**: 11（N=1, attempts=11/11）
- **Generated**: 2026-05-16T13:28:03+08:00

本報告含兩個獨立評估項目。**項目一** 評估 bot 進入「知識與產品查詢」fallback
後的撈取 + 回答能力；**項目二** 評估 bot 跨全 enabled scenarios 的 routing →
tool → answer 三段 funnel 健康度。


---

## 項目一: 知識庫精準度

## 評估範圍 (prerequisite: 進對 scenario)

- **進對 scenario 的 attempts**: 7 / 11 (63.6%)
- 只對這些 attempts 算 retrieval + answer
- 沒進對 scenario 的 attempts → 排除（routing 失敗不該污染 KB 訊號）


## Per-scenario 評估

| Scenario | qualifying / total attempts | retrieval_relevance | answer_correctness |
|---|---|---|---|
| Knowledge and Product Search | 7 / 10 | ❌ 0.0% | — |
| Transfer to Human Agent | 0 / 1 | — | — |


## 待修方向（worst-3）

1. **Knowledge and Product Search × retrieval_relevance** = 0.0% — bot 進對 scenario 但沒呼叫 search tool — 流程 / prompt 問題（不是 KB content gap）

---

## 項目二: 情境調用與完成

## 整體 funnel（全 scenarios 加總）

| Stage | Pass count | % of total |
|---|---|---|
| Total attempts | 11 | 100.0% |
| Step 1: scenario_routing | 7 | 63.6% |
| Step 2: + tool_calling   | 7 | 63.6% |
| Step 3: + answer         | 0 | 0.0% |


## Per-scenario funnel

| Scenario (n_attempts) | Step 1 (routing) | Step 2 (tool) | Step 3 (answer) | end-to-end |
|---|---|---|---|---|
| Knowledge and Product Search (10) | ⚪ 7/10 (70.0%) | ⚪ 7/10 (70.0%) | ❌ 0/10 (0.0%) | ❌ 0/10 (0.0%) |
| Transfer to Human Agent (1) | ❌ 0/1 (0.0%) | ❌ 0/1 (0.0%) | ❌ 0/1 (0.0%) | ❌ 0/1 (0.0%) |


## Drop-off 最大的 5 個 scenario

1. **Transfer to Human Agent** drop at step1 (-100.0pp)
2. **Knowledge and Product Search** drop at step3 (-70.0pp)

---

## Audit

Reproduce combined report: `bin/rails runner "puts Eval::EvaluationReport.call(run: EvalRun.find(137))"`
Or fetch each item separately:
- `bin/rails runner "puts Eval::KbAccuracyReport.call(run: EvalRun.find(137))"`
- `bin/rails runner "puts Eval::ScenarioFunnelReport.call(run: EvalRun.find(137))"`


---

# Per-Scenario × Per-Dim — Run #137

Suite: **IM Motor (bulk R1)** · scenarios: 2 · dims: 5 · populated cells: 7/10

| Scenario | Scenario | Tool | Retrieval | Faith | AnsQ |
| --- | --- | --- | --- | --- | --- |
| Knowledge and Product Search | 70.0% [40.0–100.0] (n=10) | 30.0% [0.0–60.0] (n=10) | — | 0.0% [0.0–0.0] (n=6) | 40.3% [25.3–56.3] (n=10) |
| Transfer to Human Agent | 100.0% (n=1) | — | — | 0.0% (n=1) | 40.0% (n=1) |

## Worst-3 cells (lowest primary score)

1. **Knowledge and Product Search** × **Faith** · 0.0% (n=6) · lowest sub_metric: `rule_compliance`
2. **Transfer to Human Agent** × **Faith** · 0.0% (n=1) · lowest sub_metric: `rule_compliance`
3. **Knowledge and Product Search** × **Tool** · 30.0% (n=10) · lowest sub_metric: `tools_recall`

兩項目評估報告

EvalRun: #137（suite #77 auto-85ebd775-r1-053958840）
Target: #1 production-baseline
Cases: 11（N=1, attempts=11/11）
Generated: 2026-05-16T13:28:03+08:00

本報告含兩個獨立評估項目。項目一 評估 bot 進入「知識與產品查詢」fallback 後的撈取 + 回答能力；項目二 評估 bot 跨全 enabled scenarios 的 routing → tool → answer 三段 funnel 健康度。

項目一: 知識庫精準度

評估範圍 (prerequisite: 進對 scenario)

進對 scenario 的 attempts: 7 / 11 (63.6%)
只對這些 attempts 算 retrieval + answer
沒進對 scenario 的 attempts → 排除（routing 失敗不該污染 KB 訊號）

Per-scenario 評估

Scenario	qualifying / total attempts	retrieval_relevance	answer_correctness
Knowledge and Product Search	7 / 10	❌ 0.0%	—
Transfer to Human Agent	0 / 1	—	—

待修方向（worst-3）

Knowledge and Product Search × retrieval_relevance = 0.0% — bot 進對 scenario 但沒呼叫 search tool — 流程 / prompt 問題（不是 KB content gap）

項目二: 情境調用與完成

整體 funnel（全 scenarios 加總）

Stage	Pass count	% of total
Total attempts	11	100.0%
Step 1: scenario_routing	7	63.6%
Step 2: + tool_calling	7	63.6%
Step 3: + answer	0	0.0%

Per-scenario funnel

Scenario (n_attempts)	Step 1 (routing)	Step 2 (tool)	Step 3 (answer)	end-to-end
Knowledge and Product Search (10)	⚪ 7/10 (70.0%)	⚪ 7/10 (70.0%)	❌ 0/10 (0.0%)	❌ 0/10 (0.0%)
Transfer to Human Agent (1)	❌ 0/1 (0.0%)	❌ 0/1 (0.0%)	❌ 0/1 (0.0%)	❌ 0/1 (0.0%)

Drop-off 最大的 5 個 scenario

Transfer to Human Agent drop at step1 (-100.0pp)
Knowledge and Product Search drop at step3 (-70.0pp)

Audit

Reproduce combined report: bin/rails runner "puts Eval::EvaluationReport.call(run: EvalRun.find(137))" Or fetch each item separately:

bin/rails runner "puts Eval::KbAccuracyReport.call(run: EvalRun.find(137))"
bin/rails runner "puts Eval::ScenarioFunnelReport.call(run: EvalRun.find(137))"

Per-Scenario × Per-Dim — Run #137

Suite: IM Motor (bulk R1) · scenarios: 2 · dims: 5 · populated cells: 7/10

Scenario	Scenario	Tool	Retrieval	Faith	AnsQ
Knowledge and Product Search	70.0% [40.0–100.0] (n=10)	30.0% [0.0–60.0] (n=10)	—	0.0% [0.0–0.0] (n=6)	40.3% [25.3–56.3] (n=10)
Transfer to Human Agent	100.0% (n=1)	—	—	0.0% (n=1)	40.0% (n=1)

Worst-3 cells (lowest primary score)

Knowledge and Product Search × Faith · 0.0% (n=6) · lowest sub_metric: rule_compliance
Transfer to Human Agent × Faith · 0.0% (n=1) · lowest sub_metric: rule_compliance
Knowledge and Product Search × Tool · 30.0% (n=10) · lowest sub_metric: tools_recall