infrastructure/infra
2
0
Fork 1
Code Issues 1 Pull requests 8 Projects Releases Packages Wiki Activity Actions

fix(gen10plus): retire stale unregistered Forgejo runner #277

Open
multica-gitops wants to merge 1 commit from agent/gitops/fe66ed081a06 into main
pull from: agent/gitops/fe66ed081a06
merge into: infrastructure:main
Conversation 0 Commits 1 Files changed 2 +2 -54
multica-gitops commented 2026-10-18 13:25:32 +00:00
Collaborator
Copy link

问题

microserver-gen10plus 的 systemd 周期性 degraded。失败单元为 gitea-runner-gen10plus.service：

Active: failed (Result: start-limit-hit)
gitea-runner-gen10plus[...]: Error: unauthenticated: unregistered runner

本次 boot 内该错误出现 543,278 次，随后触发 systemd start-limit。主机大部分时间因 runner 轮询进程存活而显示 running，一旦进程退出进入重启风暴即变为 degraded，这解释了 BEACO-198 中 running/degraded 反复横跳。

根因

Phase 1b（2026-09-29，docs/infra-ops/nix-collector-build-split.md）已把 nix-builder 标签的 runner 全部注销，Forgejo 仅保留 nix-collector、nix-check、Mac 三个活动 runner；当前 workflow 的 runs-on 也只有这三个标签，没有任何作业使用 nix-builder。

但 hosts/personal/fixed/microserver-gen10plus/configuration.nix 仍声明 services.gitea-actions-runner，于是主机继续运行一个服务端已注销的 runner：本地 .runner（8 月 30 日）的注册凭据失效，daemon 报 unregistered runner。这是配置未跟上下游清理的残留，不是新的 Forgejo/Tailscale 故障。

改动

  • 删除 microserver-gen10plus 的 services.gitea-actions-runner 实例、forgejo/runner/token secret 与 forgejo-runner.env 模板、TimeoutStopSec 覆盖。
  • 主机 inventory 记录 runner 已随 Phase 1b 退役。

验证

  • nix-instantiate --parse hosts/personal/fixed/microserver-gen10plus/configuration.nix → OK
  • nix eval .#nixosConfigurations.microserver-gen10plus.config.networking.hostName → OK
  • 求值后 systemd.services 中已无任何 *runner* 服务
  • utils/check-secrets.py --staged → OK
  • 未在本地构建完整 NixOS 闭包（遵循仓库 AGENTS.md；交由 Forgejo CI）

风险 / 回滚

  • 风险低：移除的是一台已无法注册、长期失败的 runner；无 workflow 依赖该标签。
  • 回滚：revert 本 PR 即可恢复声明式配置。若日后仍需该 runner 作为人工/备用构建机，应重新在 Forgejo 注册并更新 token 后再启用，而不是恢复旧的失效状态。

部署

该主机由 comin 跟踪发布分支；合并/发布后 activation 会停止并移除 gitea-runner-gen10plus.service。在发布前可临时 systemctl reset-failed gitea-runner-gen10plus.service 让 is-system-running 立即回到 running（不启动服务、不影响其他单元）。

相关

  • BEACO-198（本 issue）
  • 遗留：docs/infra-ops/runner-maintenance.md 仍按「三台 Harvester runner + gen10plus」描述，与 Phase 1b 后的实际 runner 拓扑不一致，建议另开 docs issue 统一。
## 问题 `microserver-gen10plus` 的 `systemd` 周期性 `degraded`。失败单元为 `gitea-runner-gen10plus.service`： ``` Active: failed (Result: start-limit-hit) gitea-runner-gen10plus[...]: Error: unauthenticated: unregistered runner ``` 本次 boot 内该错误出现 543,278 次，随后触发 systemd start-limit。主机大部分时间因 runner 轮询进程存活而显示 `running`，一旦进程退出进入重启风暴即变为 `degraded`，这解释了 BEACO-198 中 running/degraded 反复横跳。 ## 根因 Phase 1b（2026-09-29，`docs/infra-ops/nix-collector-build-split.md`）已把 `nix-builder` 标签的 runner 全部注销，Forgejo 仅保留 `nix-collector`、`nix-check`、Mac 三个活动 runner；当前 workflow 的 `runs-on` 也只有这三个标签，没有任何作业使用 `nix-builder`。 但 `hosts/personal/fixed/microserver-gen10plus/configuration.nix` 仍声明 `services.gitea-actions-runner`，于是主机继续运行一个服务端已注销的 runner：本地 `.runner`（8 月 30 日）的注册凭据失效，daemon 报 `unregistered runner`。这是配置未跟上下游清理的残留，不是新的 Forgejo/Tailscale 故障。 ## 改动 - 删除 `microserver-gen10plus` 的 `services.gitea-actions-runner` 实例、`forgejo/runner/token` secret 与 `forgejo-runner.env` 模板、`TimeoutStopSec` 覆盖。 - 主机 inventory 记录 runner 已随 Phase 1b 退役。 ## 验证 - `nix-instantiate --parse hosts/personal/fixed/microserver-gen10plus/configuration.nix` → OK - `nix eval .#nixosConfigurations.microserver-gen10plus.config.networking.hostName` → OK - 求值后 `systemd.services` 中已无任何 `*runner*` 服务 - `utils/check-secrets.py --staged` → OK - 未在本地构建完整 NixOS 闭包（遵循仓库 AGENTS.md；交由 Forgejo CI） ## 风险 / 回滚 - 风险低：移除的是一台已无法注册、长期失败的 runner；无 workflow 依赖该标签。 - 回滚：revert 本 PR 即可恢复声明式配置。若日后仍需该 runner 作为人工/备用构建机，应重新在 Forgejo 注册并更新 token 后再启用，而不是恢复旧的失效状态。 ## 部署 该主机由 comin 跟踪发布分支；合并/发布后 activation 会停止并移除 `gitea-runner-gen10plus.service`。在发布前可临时 `systemctl reset-failed gitea-runner-gen10plus.service` 让 `is-system-running` 立即回到 `running`（不启动服务、不影响其他单元）。 ## 相关 - BEACO-198（本 issue） - 遗留：`docs/infra-ops/runner-maintenance.md` 仍按「三台 Harvester runner + gen10plus」描述，与 Phase 1b 后的实际 runner 拓扑不一致，建议另开 docs issue 统一。
fix(gen10plus): retire stale unregistered Forgejo runner
Some checks are pending
Check Docs i18n / check-docs-i18n (push) Successful in 8s
Details
Check Secrets / check-secrets (push) Successful in 8s
Details
policy/merge-gate Waiting for /approve on this exact commit targeting main
Details
Check Docs i18n / check-docs-i18n (pull_request) Successful in 6s
Details
Check Secrets / check-secrets (pull_request) Successful in 9s
Details
Nix Validation / nix-evaluation (push) Successful in 2m53s
Details
Nix Validation / nix-evaluation (pull_request) Successful in 3m4s
Details
788cce8a93
 
Phase 1b (2026-09-29) deregistered the duplicate nix-builder/gen10plus
runner record; no workflow uses the nix-builder label anymore. The host
config still enabled gitea-runner-gen10plus, so the daemon failed with
'unauthenticated: unregistered runner' and eventually hit start-limit,
leaving systemd degraded and spamming the journal.

Remove the runner instance, its token secret/template and the
TimeoutStopSec override, and record the retirement in the host inventory.
Some checks are pending
Check Docs i18n / check-docs-i18n (push) Successful in 8s
Details
Check Secrets / check-secrets (push) Successful in 8s
Details
policy/merge-gate Waiting for /approve on this exact commit targeting main
Required
Details
Check Docs i18n / check-docs-i18n (pull_request) Successful in 6s
Details
Check Secrets / check-secrets (pull_request) Successful in 9s
Required
Details
Nix Validation / nix-evaluation (push) Successful in 2m53s
Details
Nix Validation / nix-evaluation (pull_request) Successful in 3m4s
Required
Details
This pull request is blocked because it's outdated.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin agent/gitops/fe66ed081a06:agent/gitops/fe66ed081a06
git switch agent/gitops/fe66ed081a06
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
infrastructure/infra!277
No description provided.