FeedLens 评测报告
本文件由 python -m feedlens.evaluate --report 根据 eval/results/ 下的实际运行结果自动生成，请勿手改数字。
计分口径见 feedlens/metrics.py。所有系统使用同一批样本、同一个解析器、同一套指标；格式不合规的输出按错误计。
参评系统
| 系统 |
说明 |
rules |
关键词规则基线，不使用模型（feedlens/baseline_rules.py） |
base-zeroshot |
训练前：MiniMind 官方 full_sft 权重，使用与微调时相同的简短指令 |
base-detailed |
训练前：同一份官方权重，提示词里写明标签集合并给一个示例 |
scratch-v1 |
对照：同结构随机初始化，只用任务数据训练，不加载任何预训练权重 |
ft-lora-v1 |
在官方 full_sft 权重上训练 LoRA 适配器（主干冻结），取验证损失最低的检查点（第 350 步） |
ft-lora-v1-last |
同一次 LoRA 训练在结束时（第 966 步）的权重，用来对比“按验证损失选检查点”是否合适 |
ft-full-v1 |
在官方 full_sft 权重上全参数微调（验证损失最低即训练结束时，第 483 步） |
ft-full-v2 |
v0.2：全参数微调，训练数据换成 v2；取验证损失最低的检查点（第 150 步，不到一轮） |
ft-full-v2-last |
v0.2：同一次训练结束时（第 789 步）的权重 |
同分布测试集 test（合成语法，核心表达未在训练集出现）
样本数：977
| 系统 |
格式合规 |
类别准确率 |
类别 macro-F1 |
情感准确率 |
紧急度准确率 |
紧急度 macro-F1 |
三标签全对 |
摘要 ROUGE-L |
摘要完全一致 |
rules |
100.0 |
75.2 |
75.4 |
79.6 |
78.9 |
80.5 |
71.9 |
29.3 |
0.0 |
base-zeroshot |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
base-detailed |
18.8 |
2.9 |
1.9 |
16.6 |
5.9 |
9.0 |
1.3 |
0.9 |
0.1 |
scratch-v1 |
100.0 |
68.3 |
70.3 |
93.3 |
81.4 |
81.5 |
58.8 |
48.6 |
33.4 |
ft-lora-v1 |
99.6 |
78.3 |
79.7 |
94.7 |
84.6 |
84.5 |
70.3 |
65.4 |
46.6 |
ft-lora-v1-last |
99.6 |
78.9 |
80.0 |
94.9 |
85.7 |
85.6 |
71.4 |
63.8 |
44.2 |
ft-full-v1 |
100.0 |
75.5 |
76.2 |
91.1 |
83.8 |
83.8 |
66.8 |
62.2 |
44.9 |
ft-full-v2 |
99.7 |
75.7 |
76.9 |
93.2 |
86.4 |
85.8 |
69.2 |
54.6 |
31.1 |
ft-full-v2-last |
100.0 |
82.6 |
82.9 |
93.9 |
84.9 |
84.9 |
73.7 |
63.8 |
42.2 |
95% 置信区间（自助法 1000 次重采样）：
| 系统 |
类别准确率 |
三标签全对 |
摘要 ROUGE-L |
rules |
72.3 – 78.1 |
69.1 – 74.6 |
28.0 – 30.7 |
base-zeroshot |
0.0 – 0.0 |
0.0 – 0.0 |
0.0 – 0.0 |
base-detailed |
1.8 – 4.0 |
0.7 – 2.1 |
0.6 – 1.3 |
scratch-v1 |
65.4 – 71.2 |
55.9 – 62.0 |
46.1 – 51.3 |
ft-lora-v1 |
75.6 – 81.0 |
67.5 – 73.4 |
63.0 – 67.8 |
ft-lora-v1-last |
76.4 – 81.4 |
68.7 – 74.2 |
61.4 – 66.3 |
ft-full-v1 |
72.9 – 78.3 |
64.0 – 70.0 |
59.8 – 64.7 |
ft-full-v2 |
73.4 – 78.3 |
66.4 – 72.1 |
52.1 – 57.0 |
ft-full-v2-last |
80.2 – 85.0 |
70.9 – 76.4 |
61.3 – 65.9 |
分布外测试集 ood_test（手写，不经过语法）
样本数：117
| 系统 |
格式合规 |
类别准确率 |
类别 macro-F1 |
情感准确率 |
紧急度准确率 |
紧急度 macro-F1 |
三标签全对 |
摘要 ROUGE-L |
摘要完全一致 |
rules |
100.0 |
65.0 |
66.6 |
69.2 |
72.6 |
66.0 |
58.1 |
32.4 |
0.0 |
base-zeroshot |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
base-detailed |
17.1 |
2.6 |
2.0 |
12.0 |
5.1 |
11.8 |
1.7 |
1.5 |
0.0 |
scratch-v1 |
100.0 |
58.1 |
56.7 |
82.1 |
74.4 |
71.7 |
51.3 |
22.2 |
3.4 |
ft-lora-v1 |
99.1 |
69.2 |
67.4 |
87.2 |
80.3 |
77.0 |
60.7 |
38.0 |
10.3 |
ft-lora-v1-last |
99.1 |
70.9 |
70.6 |
88.9 |
82.1 |
79.3 |
64.1 |
34.6 |
6.8 |
ft-full-v1 |
99.1 |
72.6 |
72.3 |
85.5 |
72.6 |
69.7 |
58.1 |
35.9 |
11.1 |
ft-full-v2 |
99.1 |
65.8 |
65.3 |
79.5 |
71.8 |
66.3 |
54.7 |
30.7 |
7.7 |
ft-full-v2-last |
100.0 |
63.2 |
64.3 |
83.8 |
75.2 |
72.1 |
53.0 |
33.5 |
7.7 |
95% 置信区间（自助法 1000 次重采样）：
| 系统 |
类别准确率 |
三标签全对 |
摘要 ROUGE-L |
rules |
56.4 – 73.5 |
48.7 – 67.5 |
28.1 – 36.4 |
base-zeroshot |
0.0 – 0.0 |
0.0 – 0.0 |
0.0 – 0.0 |
base-detailed |
0.0 – 5.1 |
0.0 – 4.3 |
0.5 – 2.8 |
scratch-v1 |
49.6 – 66.7 |
42.7 – 60.7 |
18.1 – 27.0 |
ft-lora-v1 |
60.7 – 77.8 |
52.1 – 70.1 |
32.6 – 44.0 |
ft-lora-v1-last |
62.4 – 78.6 |
54.7 – 72.7 |
29.6 – 39.5 |
ft-full-v1 |
65.0 – 80.3 |
49.6 – 66.7 |
30.4 – 42.0 |
ft-full-v2 |
57.3 – 73.5 |
45.3 – 64.1 |
25.5 – 36.4 |
ft-full-v2-last |
54.7 – 71.8 |
43.6 – 62.4 |
28.4 – 38.6 |
验证集 val（用于选模型，不作为最终结论）
样本数：969
| 系统 |
格式合规 |
类别准确率 |
类别 macro-F1 |
情感准确率 |
紧急度准确率 |
紧急度 macro-F1 |
三标签全对 |
摘要 ROUGE-L |
摘要完全一致 |
rules |
100.0 |
76.8 |
77.3 |
80.9 |
80.6 |
80.6 |
74.3 |
29.8 |
0.0 |
base-zeroshot |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
scratch-v1 |
100.0 |
68.3 |
70.4 |
92.7 |
77.1 |
76.4 |
58.7 |
48.4 |
34.0 |
ft-lora-v1 |
100.0 |
76.4 |
77.3 |
94.2 |
81.8 |
79.9 |
64.0 |
60.9 |
42.7 |
ft-lora-v1-last |
100.0 |
77.2 |
78.3 |
94.0 |
85.4 |
84.9 |
69.0 |
59.4 |
41.2 |
ft-full-v1 |
100.0 |
83.8 |
85.5 |
96.8 |
90.1 |
89.8 |
78.9 |
63.6 |
43.8 |
ft-full-v2 |
100.0 |
81.0 |
81.4 |
92.1 |
79.5 |
76.9 |
68.3 |
60.4 |
42.1 |
ft-full-v2-last |
99.3 |
86.0 |
85.9 |
98.1 |
86.2 |
85.7 |
75.0 |
68.3 |
47.1 |
95% 置信区间（自助法 1000 次重采样）：
| 系统 |
类别准确率 |
三标签全对 |
摘要 ROUGE-L |
rules |
74.0 – 79.5 |
71.5 – 76.9 |
28.4 – 31.1 |
base-zeroshot |
0.0 – 0.0 |
0.0 – 0.0 |
0.0 – 0.0 |
scratch-v1 |
65.3 – 71.5 |
55.4 – 61.9 |
45.9 – 51.0 |
ft-lora-v1 |
73.7 – 79.0 |
61.0 – 66.8 |
58.4 – 63.4 |
ft-lora-v1-last |
74.4 – 79.8 |
66.0 – 72.0 |
56.9 – 61.9 |
ft-full-v1 |
81.4 – 86.1 |
76.4 – 81.6 |
61.3 – 65.9 |
ft-full-v2 |
78.3 – 83.4 |
65.4 – 71.2 |
57.9 – 62.9 |
ft-full-v2-last |
83.8 – 88.0 |
72.3 – 77.6 |
65.9 – 70.7 |
分布外开发集 ood_dev（用于错误分析；v2 的语法依据它的失败案例设计，对 v2 系统不再是独立数据）
样本数：63
| 系统 |
格式合规 |
类别准确率 |
类别 macro-F1 |
情感准确率 |
紧急度准确率 |
紧急度 macro-F1 |
三标签全对 |
摘要 ROUGE-L |
摘要完全一致 |
rules |
100.0 |
54.0 |
52.8 |
57.1 |
60.3 |
58.5 |
41.3 |
28.9 |
0.0 |
base-zeroshot |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
base-detailed |
17.5 |
0.0 |
0.0 |
14.3 |
1.6 |
2.8 |
0.0 |
0.2 |
0.0 |
scratch-v1 |
100.0 |
63.5 |
62.7 |
81.0 |
73.0 |
73.0 |
54.0 |
31.6 |
7.9 |
ft-lora-v1 |
100.0 |
76.2 |
75.6 |
85.7 |
74.6 |
72.4 |
63.5 |
41.8 |
11.1 |
ft-lora-v1-last |
100.0 |
74.6 |
73.8 |
81.0 |
73.0 |
71.2 |
61.9 |
42.9 |
11.1 |
ft-full-v1 |
100.0 |
63.5 |
63.1 |
84.1 |
71.4 |
70.8 |
54.0 |
36.9 |
9.5 |
ft-full-v2 |
100.0 |
82.5 |
83.2 |
88.9 |
85.7 |
85.9 |
76.2 |
52.1 |
27.0 |
ft-full-v2-last |
100.0 |
93.7 |
93.6 |
92.1 |
93.7 |
94.0 |
90.5 |
68.7 |
44.4 |
95% 置信区间（自助法 1000 次重采样）：
| 系统 |
类别准确率 |
三标签全对 |
摘要 ROUGE-L |
rules |
41.3 – 66.7 |
28.6 – 54.0 |
24.2 – 33.7 |
base-zeroshot |
0.0 – 0.0 |
0.0 – 0.0 |
0.0 – 0.0 |
base-detailed |
0.0 – 0.0 |
0.0 – 0.0 |
0.0 – 0.6 |
scratch-v1 |
52.4 – 74.6 |
41.3 – 66.7 |
24.2 – 39.0 |
ft-lora-v1 |
65.1 – 85.7 |
52.4 – 74.6 |
34.3 – 49.2 |
ft-lora-v1-last |
63.5 – 84.1 |
50.8 – 73.0 |
35.6 – 50.4 |
ft-full-v1 |
52.4 – 74.6 |
42.9 – 65.1 |
30.0 – 44.5 |
ft-full-v2 |
73.0 – 92.1 |
66.7 – 87.3 |
42.4 – 60.4 |
ft-full-v2-last |
87.3 – 98.4 |
82.5 – 96.8 |
60.4 – 77.2 |
v2 新增问题类型的留出集 test_v2new（合成语法 v2，核心表达未在任何训练集出现）
样本数：381
| 系统 |
格式合规 |
类别准确率 |
类别 macro-F1 |
情感准确率 |
紧急度准确率 |
紧急度 macro-F1 |
三标签全对 |
摘要 ROUGE-L |
摘要完全一致 |
rules |
100.0 |
50.7 |
55.4 |
50.4 |
55.1 |
47.8 |
34.9 |
23.5 |
0.0 |
ft-full-v1 |
100.0 |
65.4 |
62.6 |
90.6 |
78.5 |
77.5 |
58.3 |
31.3 |
0.3 |
ft-full-v2 |
99.5 |
75.3 |
78.1 |
93.7 |
85.8 |
85.2 |
64.8 |
58.4 |
32.3 |
ft-full-v2-last |
100.0 |
90.8 |
91.7 |
93.2 |
94.5 |
94.7 |
89.0 |
68.9 |
42.8 |
95% 置信区间（自助法 1000 次重采样）：
| 系统 |
类别准确率 |
三标签全对 |
摘要 ROUGE-L |
rules |
45.7 – 55.9 |
30.2 – 40.2 |
21.3 – 25.8 |
ft-full-v1 |
60.9 – 69.8 |
53.5 – 63.2 |
28.5 – 33.8 |
ft-full-v2 |
70.9 – 79.5 |
59.8 – 69.3 |
54.7 – 62.0 |
ft-full-v2-last |
87.7 – 93.7 |
85.8 – 92.1 |
65.4 – 72.1 |
各类别的类别准确率 · test
| 系统 |
功能故障 |
性能体验 |
功能建议 |
界面交互 |
账号登录 |
支付订单 |
广告骚扰 |
表扬认可 |
其他 |
rules |
63.1 |
62.5 |
95.7 |
48.2 |
86.1 |
100.0 |
90.5 |
55.9 |
82.0 |
base-zeroshot |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
base-detailed |
19.9 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
scratch-v1 |
44.0 |
62.5 |
93.6 |
68.4 |
75.7 |
83.2 |
62.9 |
49.0 |
93.4 |
ft-lora-v1 |
71.6 |
72.5 |
100.0 |
77.2 |
73.0 |
87.2 |
67.6 |
78.4 |
83.6 |
ft-lora-v1-last |
74.5 |
71.7 |
100.0 |
78.9 |
77.4 |
86.4 |
69.5 |
78.4 |
75.4 |
ft-full-v1 |
82.3 |
61.7 |
98.9 |
54.4 |
87.8 |
77.6 |
62.9 |
80.4 |
77.0 |
ft-full-v2 |
47.5 |
77.5 |
92.6 |
75.4 |
87.0 |
83.2 |
61.9 |
85.3 |
83.6 |
ft-full-v2-last |
88.7 |
79.2 |
100.0 |
55.3 |
75.7 |
91.2 |
86.7 |
84.3 |
85.2 |
各紧急度档位的紧急度准确率 · test
| 系统 |
高 |
中 |
低 |
rules |
90.0 |
70.6 |
82.5 |
base-zeroshot |
0.0 |
0.0 |
0.0 |
base-detailed |
24.2 |
0.0 |
0.0 |
scratch-v1 |
81.7 |
82.7 |
79.1 |
ft-lora-v1 |
75.4 |
86.7 |
89.0 |
ft-lora-v1-last |
81.2 |
86.1 |
88.7 |
ft-full-v1 |
84.2 |
80.9 |
88.0 |
ft-full-v2 |
72.1 |
91.0 |
91.1 |
ft-full-v2-last |
89.2 |
80.9 |
87.3 |
各类别的类别准确率 · ood_test
| 系统 |
功能故障 |
性能体验 |
功能建议 |
界面交互 |
账号登录 |
支付订单 |
广告骚扰 |
表扬认可 |
其他 |
rules |
38.5 |
76.9 |
61.5 |
46.2 |
84.6 |
69.2 |
76.9 |
30.8 |
100.0 |
base-zeroshot |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
base-detailed |
23.1 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
0.0 |
scratch-v1 |
61.5 |
69.2 |
69.2 |
46.2 |
69.2 |
84.6 |
76.9 |
15.4 |
30.8 |
ft-lora-v1 |
84.6 |
69.2 |
84.6 |
53.8 |
76.9 |
84.6 |
69.2 |
7.7 |
92.3 |
ft-lora-v1-last |
84.6 |
76.9 |
84.6 |
69.2 |
69.2 |
76.9 |
69.2 |
23.1 |
84.6 |
ft-full-v1 |
92.3 |
53.8 |
92.3 |
84.6 |
92.3 |
61.5 |
69.2 |
30.8 |
76.9 |
ft-full-v2 |
23.1 |
76.9 |
84.6 |
92.3 |
69.2 |
84.6 |
30.8 |
69.2 |
61.5 |
ft-full-v2-last |
76.9 |
61.5 |
84.6 |
61.5 |
61.5 |
76.9 |
61.5 |
38.5 |
46.2 |
各紧急度档位的紧急度准确率 · ood_test
| 系统 |
高 |
中 |
低 |
rules |
35.7 |
67.2 |
91.1 |
base-zeroshot |
0.0 |
0.0 |
0.0 |
base-detailed |
42.9 |
0.0 |
0.0 |
scratch-v1 |
78.6 |
79.3 |
66.7 |
ft-lora-v1 |
64.3 |
89.7 |
73.3 |
ft-lora-v1-last |
71.4 |
93.1 |
71.1 |
ft-full-v1 |
78.6 |
70.7 |
73.3 |
ft-full-v2 |
42.9 |
70.7 |
82.2 |
ft-full-v2-last |
78.6 |
72.4 |
77.8 |
未见领域词的影响 · test
训练集完全不含“学习”领域的词（课程、题库、错题本等），测试集含。
| 系统 |
见过的领域：样本数 |
三标签全对 |
摘要 ROUGE-L |
未见领域：样本数 |
三标签全对 |
摘要 ROUGE-L |
rules |
877 |
72.5 |
29.7 |
100 |
66.0 |
26.0 |
base-zeroshot |
877 |
0.0 |
0.0 |
100 |
0.0 |
0.0 |
base-detailed |
877 |
1.4 |
0.9 |
100 |
1.0 |
1.6 |
scratch-v1 |
877 |
59.9 |
49.4 |
100 |
49.0 |
42.0 |
ft-lora-v1 |
877 |
71.8 |
66.0 |
100 |
57.0 |
59.8 |
ft-lora-v1-last |
877 |
73.0 |
64.8 |
100 |
58.0 |
55.0 |
ft-full-v1 |
877 |
67.2 |
62.6 |
100 |
64.0 |
58.8 |
ft-full-v2 |
877 |
69.3 |
55.3 |
100 |
68.0 |
48.2 |
ft-full-v2-last |
877 |
74.3 |
63.8 |
100 |
68.0 |
63.5 |
速度与资源
批量：按提示词长度分组成批的整体吞吐，在“批量评测设备”上测得。单请求：batch=1 逐条调用，固定在 CPU 上测（应用默认用 CPU 推理），对应网页上点一次按钮。进程峰值内存包含评测时同时加载的批量模型与 CPU 模型，高于应用单独运行时的占用。
| 系统 |
批量评测设备 |
参数量(M) |
权重文件(MB) |
批量 样本/秒 |
平均生成 token |
CPU 单请求 平均/P50/P95 (ms) |
进程峰值内存(MB) |
rules |
cpu |
0 |
0 |
87100.3 |
0.0 |
— |
1419.8 |
base-zeroshot |
mps |
63.91 |
131.3 |
26.6 |
46.0 |
287.0 / 285.8 / 349.4 |
1211.2 |
base-detailed |
mps |
63.91 |
131.3 |
14.3 |
47.9 |
391.3 / 399.4 / 408.3 |
1294.0 |
scratch-v1 |
mps |
63.91 |
131.3 |
43.5 |
23.7 |
156.1 / 163.0 / 196.1 |
1419.8 |
ft-lora-v1 |
mps |
64.31 |
132.1 |
45.3 |
24.0 |
130.1 / 127.6 / 147.1 |
1257.3 |
ft-lora-v1-last |
mps |
64.31 |
132.1 |
47.7 |
24.1 |
129.6 / 126.7 / 144.9 |
1353.8 |
ft-full-v1 |
mps |
63.91 |
131.3 |
44.2 |
24.0 |
134.2 / 127.3 / 175.8 |
1419.8 |
ft-full-v2 |
mps |
63.91 |
131.3 |
31.5 |
24.5 |
193.7 / 191.9 / 219.4 |
1256.3 |
ft-full-v2-last |
mps |
63.91 |
131.3 |
44.5 |
24.6 |
125.5 / 124.0 / 144.3 |
1254.6 |
环境：macOS-27.0.1-arm64-arm-64bit；上游 MiniMind 提交 f659b55。
本页由 eval/REPORT.md 生成。