test(e2e): time the guest console capture and baseline it on a green run - #3841

What this PR does

A tenant worker's guest serial console is stamped in seconds since that guest's own kernel started. So a log that ends mid-boot tells you how far the guest got and nothing at all about when that was. Whether it ends there because the guest started late or because the guest stopped printing is the difference between two unrelated failures, and neither instant was written down anywhere: the read's own instant was nowhere in the file, surviving only as the artifact's mtime, which is metadata rather than content and so says nothing to anyone who opens the log, and the guest's start only inside a describe dump that the capture triggers exclusively when some console came back silent. On a run where every console returned bytes, that dump does not exist.

I went looking for the answer in three collected reports before changing anything. Every one of the twelve captures across them exited zero, so no read was truncated by its 30s bound or by --limit-bytes, and the logs are 23 to 79 KB against a 1 MiB cap. The captures stop at 305, 370, 432, 456, 474, 497, 564, 570, 583, 599 and 639 guest seconds. A fixed cutting mechanism does not produce a 334 second spread. The logs end where the guests stopped printing. On the single worker where both ends were recoverable at all, by reading the file's mtime and the describe together, the guest had been alive for 955 seconds and had printed up to second 564.

Each console log now carries the instant its own read started and returned, and CAPTURE-TIMELINE.txt beside it carries each Pod's phase, its start time and the start of its compute and console containers, from one bounded read taken once for the whole selector rather than per Pod. Those two subtract to how long a guest had been alive when it was read, which is what the log's last kernel stamp gets compared against. Close together means the guest was still printing and the log ends because it had not got further. Far apart means it stopped. A Pod with no compute start never ran a guest at all, whatever its console file says. The legend in the file names the units on both ends, because an epoch integer and an RFC 3339 string subtract to nonsense if you take them as printed. A read that failed and a read that answered with nothing are written down as themselves instead of leaving an absent file.

The same capture now also runs on the passing path, after the last assertion and before the tenant is deleted, because the container holding the stream goes with the Pod and no read recovers it afterwards. Every console this tree has collected came from a failure, so nobody has ever measured what a healthy boot looks like, while "slower than usual" is the claim the node-join failure keeps resting on. That caller asks for the pool's minimum rather than its maximum, since two workers booting is the same measurement as ten, and it warns in the job log instead of failing a suite that has already proved everything it exists to prove. Both paths write the same directory, so the capture records which of them produced it.

The cost moves by one bounded read. The serial console family goes from roughly 5m50s to 6m25s at worst and from 3m30s to 4m05s at the pool minimum, both against the 7m diagnostics phase budget, and the figure in both suite comments moves with it under the guard that holds the two in step. The passing path adds six bounded reads at 30s and 5s kill grace, so 210s.

That 210s is real wall clock at the tail of an operation with a fixed 50m ceiling, which is a way for a run that already passed to be killed while collecting a debugging aid. So the passing-path capture declines itself out loud when the ceiling no longer has room for it, against a reserve derived from its own cost plus the deletes after it. The ceiling it measures against is read from both sides by a guard, so a suite that changes its timeout: fails there rather than leaving the reserve protecting a number that moved.

What this deliberately does not do is widen a capture window. I could not derive a number for one that would be honest. The reads are already complete, so there is no read side bound to relax, and the only remaining lever is waiting longer before reading. How much longer depends on whether the tail silence is the guest freezing or the stream stopping, and that is precisely what was not measurable. The evidence leans toward the guest: the logs end on complete lines, and one of them ends on an RCU stall backtrace, which is the kernel reporting it was not being scheduled. If that holds, the silence has no ceiling and no fixed window can promise to contain a milestone. The green baseline this adds is what would set the number, and it does not exist yet.

Context for #3513, which this does not fix. It restores the instrument that investigation needs.

Screenshots

Not a UI change.

Downstream repositories

I walked the trigger map in docs/agents/contributing.md against the actual diff. It touches docs/agents/e2e-testing.md, the shared e2e helper hack/e2e-chainsaw/_lib/run-kubernetes.sh, the two kubernetes-* suite files and three unit test files under hack/. Nothing under packages/, no ApplicationDefinition, no values schema or enum or default, no namespace, no emitted metric, no label or annotation another component reads, no node prerequisite in hack/e2e-prepare-cluster.bats, no make target and no file moved or renamed under hack/. No trigger in the map matches.

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:

Release note

test(e2e): stamp each tenant worker guest console capture with the instant it was read and the instant its guest started, and take the same capture on passing runs so a failing boot has a healthy one to be compared against

Summary by CodeRabbit

• Diagnostics

  • Improved Kubernetes serial-console diagnostics with capture context, bounded collection, per-read timestamps, and cluster-wide Pod start-time timelines.
  • Added clearer reporting for incomplete, unavailable, or failed diagnostic data.
  • Added absolute timestamps and explicit time units to support reliable troubleshooting.

• Reliability

  • Passing test runs now collect baseline diagnostics only when sufficient time remains, helping prevent suite timeouts.

• Tests & Documentation

  • Expanded diagnostic coverage and updated documented timing expectations for supported Kubernetes versions.