The monitor: the Foundry checks itself

The monitor is a watcher that looks at the Foundry itself and writes down what is wrong, so you do not have to find it. It is off until you add it: on the Watchers page choose Add watcher and the source The Foundry itself. It needs only an id and an interval. It has no repository, no flow and no label, only one is allowed, and only an admin can add it. It starts nothing and changes no run, label or file of a run; it writes its findings and, if you set report_to, bug stories (see Bug stories). The whole loop, in one place, is in chapter 13.

Each check runs these detectors. Each one reads the runs, the queue, the watchers and the server log; none calls an AI or GitHub.

Critical means work has stopped, major means work is slowed or wrong, minor means wrong but harmless. A detector that crashes shows up as a finding "Detector X failed"; the others still run.

Change the thresholds in config.yaml:

watchers:
  - { id: monitor, source: monitor, every: 5m }
monitor:
  restart_loop: { resumes: 5, within_minutes: 10 }   # resumes: at most 49
  watcher_error: { checks: 3 }
  github_limit: { percent: 80 }
  watcher_silent: { intervals: 5 }
  unexplained_failure: { runs: 1, within_hours: 24 }
  stuck_run: { extra_minutes: 10, no_timeout_minutes: 120 }
  same_step_failing: { issues: 3, within_hours: 24 }
  label_mismatch: { checks: 3 }
  orphan_lock: { minutes: 10 }
  queue_stalled: { minutes: 15 }
  restart_overdue: { hours: 2 }
  develop_red: { failures: 2, within_hours: 24 }
  slow_step: { factor: 3, times: 3, within_hours: 24 }

What is never a finding: a run that waits for a person (approval or questions), a usage limit that resets by itself, a sign-out (that is your turn, already shown), a story that waits for a dependency or a release, and a factory:done label. While the server waits to restart, the label and watcher-silent detectors are quiet.

Notes:

• Label data comes from the watchers' last checks. A done label is never a mismatch.
• A run inside a sub-flow step is not checked for being stuck.
• Same step keeps failing counts runs that ended failed, not steps that failed and were fixed in the same run. A step that only handles another step's failure is not named.
• Develop is red counts every test run after a merge, also several of one story, and looks back within_hours.
• A lock without owner is counted from the check that first saw it. A lock whose owner's run.json cannot be read is counted the same way.
• Slow step compares with the newest 500 succeeded runs that finished before the last within_hours, rebuilt about once an hour.
• The monitor keeps running while a new version waits for the server to restart.

Findings are kept in monitor-findings.json in the data folder and survive a restart. Each has a detector, a fingerprint (the same problem gives the same one), a severity, one sentence, the evidence, when it was first and last seen and how often. A finding not seen for 24 hours becomes gone (kept 30 days; at most 500 findings are kept, not counting those with a bug story or an owed one). A file that cannot be read is kept as monitor-findings.json.broken. If the monitor's own check fails, the Health line says so. The monitor's card on the Watchers page shows what waits or is wrong with its bug stories, and lists the findings and the mutes (see Mute a detector or a finding). The Problems page shows the same with the state of each finding.

Bug stories from the monitor#

Set report_to and the monitor writes a lasting problem up as one GitHub issue, so the normal flow can fix it. Try it on a private repository first, and read one real story before you point it at a public one.

monitor:
  report_to: your-name/your-foundry-repo
  report_limits: { per_day: 3, per_check: 1 }   # per_check: at most 3
  cooldown_minutes: 10                          # quiet time after a server start; 0: none
  fix_wait_days: 7                              # watch a closed story anyway after this many days
  breaker: { new_findings: 5, within_minutes: 60, failed_fixes: 3 }   # the circuit breaker

• When. Critical and major findings: after 2 checks in a row. Minor: after 3 different days. A story that is owed is made later even if the problem has gone away.

• What. Labels bug and the build label of the repository's issue watcher (with several watchers: the issue-gitflow one). With no watcher the story only gets bug, and the card says so. The text comes from a fixed template (no AI): what happened, since when, effect, evidence, what should happen, how to see it again, where to look, acceptance criteria.

• Cleaning. Other repositories, people, e-mail addresses, folders, links and keys are removed. In log lines only words of a fixed list stay. If a line cannot be cleaned with certainty, the story says the evidence is left out.

• Only once. A hidden marker in the story tells the monitor it exists. While it is open, a comment "Seen again: N times since …" is added at most every 6 hours.

• Came back. If the story was closed as completed and the problem returns after the fix runs (new proof from after the 24-hour clock started, see "Did the fix work?"), and the usual rule is met (2 checks in a row; minor: 3 different days after the clock started), a new story links the old one. Closed as not planned means muted: reopen the story to unmute it. (That is not an admin's mute; see below.) A story that is reopened is open again: its clock and verdict are forgotten.

• Did the fix work? A story closed as completed is waiting for the update until the fix is running. Sightings then are not "came back" and make no story. The 24-hour clock starts: when the running Foundry contains the fix commit (the full commit is read from the run that built the story); else at the first server start after the close (no git checkout, fix commit not known, or report_to is not the Foundry's own repository); and in any case fix_wait_days (default 7) after the close (the log says "waited"). If the problem is not seen for 24 hours of normal work (time with no check, such as a sleep, does not count), the finding is fixed and the story gets one comment "Not seen since the fix." It is written once, also after a lost answer or when GitHub could not be reached; while bug stories are off, quiet or stopped by the breaker, or the finding is muted, it waits. A story closed long before the clock (fix_wait_days or more) gets no comment. If the problem is seen again after the clock started, with new proof, "fixed" is not said. The findings list shows: waiting for the update, being watched, fixed. A story whose problem is not seen is still checked on GitHub every 6 hours, so a close or a reopen is noticed.

• Two tries, then a person. The monitor counts the bug stories it made for a finding. After two that did not fix it, it makes no third: the finding needs you, the log says so once (story-skipped, reason two_tries), and Needs you on Home shows one item for it. A story that existed before the upgrade counts as one. Nothing becomes "needs you" while bug stories are off, quiet after a restart or stopped by the breaker, or while the finding is muted.

  • Try again or mute. On the Watchers page the findings list shows "needs you" and a Try again button (admin only). It starts the count anew; the next check may make a story, which links the newest earlier story. A mute (see above) also makes the item go away; when the mute ends and the finding still needs a person, the item is back.
  • When the item is hidden. While the finding is gone (not seen for 24 hours), while its story is open again, and once it is fixed. The count is kept, so no third story is made when it returns. A story closed as not planned does not hide the item: Try again then starts the count anew, but no story is made until that story is reopened.
  • A new count. A finding that became fixed no longer needs you, and starts a new count when it is seen again. A finding that needs you is kept (never pruned) until someone acts.

• Limits. 3 new stories a day, 1 per check (most severe first). The rest waits and the card says how many. At most 6 GitHub calls per check. If GitHub cannot be reached or its request limit is used up, nothing is lost; the stories are made at a later check.

• Never a story about a story. A run that builds a bug story of the monitor is never a finding, whether it fails, hangs, is slow, loops, holds a lock or its label is wrong. This also holds for a story that a newer one replaced. A failed run of a bug story is written on the finding as "the fix failed" (how often and when) and in the log.

Unexplained is strict: no rule in the Foundry's failure rules matched (an AI's summary does not count). An ordinary failing command (exit code 1) is explained. The request-limit numbers come from gh api rate_limit, read at most once a minute for the server's own login and the bot token.

Stop bug stories: the off switch and the quiet time#

One switch stops the monitor from making bug stories. It needs no GitHub.

• Switch. On the monitor's card on the Watchers page, the line says "Bug stories: on", "Bug stories: off since 14:05" or "Bug stories: quiet until 14:15 after the restart", and a button switches them off or on. Only an admin can do this. On the command line: scf monitor off, scf monitor on and scf monitor status. The commands only read and write the state file, so they work when the server is not running. The server reads the state at every check and again right before each GitHub call that makes or touches a story. The state (since when, by whom: the account id or cli) is kept in monitor-guard.json in the data folder and survives a restart.
• While it is off. The monitor still records findings. It makes no story, writes no comment and makes no GitHub call for stories. Nothing new becomes owed; stories owed before stay owed and are made after you switch on, at the next check.
• Off does not stop building. Stories that already exist still carry the build label, so the issue watcher keeps building them (with hotfix_to_main on, it still pushes fixes to main). To stop that, disable the issue watcher or remove the label from the issue.
• Quiet time. For the first cooldown_minutes (default 10, 0 = none) after the server started, findings are recorded but no story is made and none becomes owed. A server that restarts more often than that never makes a story.
• A state file that cannot be read. Stories stop and the card says so. scf monitor on or the button keeps the old file as monitor-guard.json.broken (older ones as .broken.1, .broken.2, …), starts a fresh file and switches on. Later parts add other state to this file; it is reset too (the mutes as well), and the card and the log say so. "Off" does nothing on an unreadable file.
• The log. monitor-log.jsonl in the data folder has one JSON line per event: off, on, story-made, story-skipped, breaker-open, breaker-closed, fix-failed, clock-started (reason update, restart or waited), fixed, came-back, try-again (with by, the account id), mute-made and mute-ended, with the reason for a skipped story (off, cooldown, unreadable, breaker, muted, two_tries, day_limit, check_limit, request_limit, github). A story skipped because of a mute also has the mute's id and its reason text. A skipped story is written once per finding and reason, not at every check (and again each time the circuit breaker opens). The file moves to monitor-log.1.jsonl at 512 KiB, so it keeps between 512 KiB and 1 MiB of history. Lines from the server also show in the card's recent activity; a switch made on the command line does not.
• Lock. Changes take monitor.lock in the data folder for a moment. If scf monitor on says the lock is held, try again; if no process uses it, remove the folder monitor.lock. "Off" never waits for it: it takes the lock over.

The circuit breaker#

The monitor stops making stories by itself when many things go wrong at once. It only works when report_to is set and bug stories are not switched off.

• When it opens. More than new_findings (default 5) different findings first appear within within_minutes (60), or the newest failed_fixes (3) finished runs of bug stories all failed. A succeeded run between failed ones resets the count; cancelled, stopped and interrupted runs neither count nor reset. Findings first seen in the quiet time after a restart do not count.
• What stops. No story, as with the off switch. Findings are still recorded. The log has a breaker-open line and one story-skipped line (reason breaker) for each story that is owed. The reason and the time are kept in monitor-guard.json, so a restart does not close it.
• Your turn. An admin sees one item, "The monitor stopped making bug stories", with the reason and the way back: switch bug stories on again on the Watchers page. It cannot be dismissed and goes away when the breaker is closed. The card says "Bug stories: stopped by the circuit breaker since …"; scf monitor status says so too.
• Switch on again. Only an admin: the button on the monitor's card or scf monitor on. The log gets on and breaker-closed. Every switch-on starts the counts anew: findings and failed runs from before do not open the breaker again. Look at what went wrong first.

Mute a detector or a finding#

Some problems are known noise. An admin can mute one detector (by name) or one finding, so that it does not become a bug story.

• Make a mute. On the monitor's card (Watchers page) open Findings (N) · Mutes (M). Press Mute on a finding, or Mute a detector. Give a reason (required, at most 200 characters) and choose how long: for good, 1 hour, 1 day, 1 week or 30 days. The API takes any number of hours up to 8760 (one year).
• See and end mutes. The list shows each mute with its reason, since when and until when. End mute ends it at once. An admin can end any mute. A mute for a time ends by itself; the log gets mute-ended with the reason expired. scf monitor status lists the mutes too.
• What a mute stops. No story is owed or made, and no "seen again" comment is written, for a muted finding. A story that was owed before the mute is not made while the mute lasts; it is made at the next check after the mute ends (if the problem still lasts).
• What it does not stop. Findings are still recorded (count, last seen) and shown. A story that already exists keeps its build label, so it is still built. Muted findings do not count for the circuit breaker, and neither do the failed runs of their stories.
• The log. mute-made and mute-ended (with who, and the reason text), and one story-skipped line with reason muted for a story that a mute held back.
• Limits. One mute per detector and one per finding at a time (a finding can be muted next to its detector; the mute of the finding wins). At most 200 mutes. Mutes are kept in monitor-guard.json as account ids, never names or e-mail addresses.

The Problems page#

Problems (#/problems, admins only; users do not see it) shows what the monitor found and what happens to each problem.

• The sentence at the top. One sentence says whether the monitor is running, whether bug stories are on, off, quiet or stopped, when it last checked, how many bug stories it made today and the daily limit, and whether the circuit breaker is open, with its reason. The count comes from the monitor's log, so it can be above the limit: "Make a story now" does not count against the limit. "Last checked" is kept in memory: after a server restart it says "has not checked since the server started" until the monitor runs again.
• The findings. Newest and most severe first (gone ones last), at most 100 at first and then Show N more. Each row has the summary, the severity, since when, how often ("seen in N checks"), the state and the bug story. The summary links to the finding's own page (#/problems/<id>, crumbs Administration › Problems › <id>). It shows the evidence, the bug stories (links; "(earlier)" for older ones) and the runs that build them. Its buttons are in two groups: Bug story (Make a story now, Try again) and Mute (mute, This is not a problem). The rows have no Mute button. An unknown id shows the server's sentence and a link back. If the findings file cannot be read, the page says so instead of "No findings."

• Actions. Switch the monitor off / on (the same as on the Watchers page; while it is off the monitor still records problems, but makes no bug story and writes no comment). Mute and End mute. Try again for needs you. This is not a problem mutes the finding for good with a reason you give; you can end it here later.
• Make a story now. For a finding that is seen, no story yet. It skips the waiting rules: the severity, "longer than one check", 3 days for minor, the daily and per-check limits, the quiet time and an open circuit breaker. It never skips the off switch, the mutes, "only once" (a story already on GitHub is taken up, not made again) or the two tries; the cleaning of private text applies too. It makes only that one story. The button says why it cannot be used: the monitor is off, is not running, or no repository is set. The API answers 409 when the monitor is off or not running, or the finding has a story, is muted, used its two tries, is gone or has no repository; 502 when GitHub fails.
• Detectors. Each detector is listed with what it looks for, its threshold numbers, when it last found something and whether it is muted. Change a number and press Save; this changes the setting monitor.<detector> like the Settings page.
• Thresholds and settings. The Detectors table has one number field per threshold of each detector. Below the Mutes, the Monitor settings section holds two groups. Bug stories: the repository (report_to), the limits per day and per check, the quiet time after a start and the days to wait for a fix. Circuit breaker: the new-findings count, the time window and the failed fixes. Each field shows what it means, its default, its allowed range and "Takes effect at once". A value outside the range, a fraction where a whole number is needed, an empty number or a repository that is not owner/repo is not saved; the error line of that form names the field and the range. Bug stories with their evidence are written as issues to the repository in report_to, and everyone who can read that repository sees them. Check it before you set it. Back to default removes the setting from config.yaml, so the default applies. It is enabled only when the setting is in the file. A comment on the same line goes with it; comment lines above it stay. A section left empty goes too.