tests: document diagnostic oracle expectations

Why: Test assertions should prove the observable behavior operators rely
on, not just that rsyslogd wrote a message to its process streams.

Impact: Clarifies test-authoring and babysitting guidance for rsyslog
diagnostics.

Before/After: Tests could use stdout/stderr oracles without a clear
exception; guidance now prefers configured rsyslog output destinations
after synchronized shutdown.

Technical Overview: Add the authoritative rule to tests/AGENTS.md.
Point rsyslog_test at that rule when explaining test oracle
documentation. Add a review-time reminder to rsyslog_pr_babysitting so
PR feedback catches weak stdout/stderr diagnostic oracles. Exceptions
remain allowed when the tested behavior is process output or
configuration is not usable yet, but they must be documented in the test
header.

With the help of AI-Agents: OpenAI Codex
This commit is contained in:
Rainer Gerhards 2026-05-21 16:07:37 +02:00
parent c4d8f4e701
commit 2b2fcd5ef5
No known key found for this signature in database
GPG Key ID: 0CB6B2A8BE80B499
3 changed files with 20 additions and 0 deletions

View File

@ -153,6 +153,12 @@ work. Handle small, localized fixes directly, validate them, and push the
updated branch. Examples include typo fixes, documentation wording, metadata
formatting, simple example corrections, and narrow test expectation updates.
When reviewing or adjusting test assertions, apply the oracle rule from
`tests/AGENTS.md`: diagnostics emitted by rsyslog itself should normally be
asserted through the configured rsyslog output destination after synchronized
shutdown, not through rsyslogd stdout/stderr. Treat stdout/stderr diagnostic
oracles as actionable unless the test documents a specific exception.
AI review comments need an explicit GitHub reply so later readers know the
comment was considered. If the requested change was implemented, a short
`Done.` reply is sufficient. If the comment is invalid, not applicable, or not

View File

@ -63,6 +63,12 @@ When changing a test, verify that the head comment still matches the actual
setup, stimulus, oracle, and pass/fail conditions after the edit; update it in
the same commit if it does not.
For diagnostics emitted by rsyslog itself, follow `tests/AGENTS.md`: assert the
configured rsyslog output destination, usually testbench omfile output such as
`RSYSLOG_OUT_LOG`, after synchronized shutdown. Avoid rsyslogd stdout/stderr as
the oracle unless the test is specifically about process-level output or the
exception is documented in the test header.
### 4. Using diag.sh Helpers
All tests include `tests/diag.sh` using the POSIX `.` command. You should use its standardized helpers:
- `cmp_exact`: Verify file content matches.

View File

@ -31,6 +31,14 @@ agents.
When changing a test, verify that the head comment still matches the actual
setup, stimulus, oracle, and pass/fail conditions after the edit; update it in
the same commit if it does not.
- For diagnostics emitted by rsyslog itself, prefer asserting the configured
rsyslog output destination, usually testbench omfile output such as
`RSYSLOG_OUT_LOG`, after synchronized shutdown. Do not use rsyslogd
stdout/stderr as the oracle
unless the behavior being tested is specifically process-level stdout/stderr
emission, startup before configuration is usable, or another documented case
where the message cannot pass through rsyslog's normal output path. Explain
such exceptions in the test header.
- Prefer harness helpers such as `cmp_exact`, `command_deny`, and
`require_plugin` over ad-hoc shell to keep diagnostics uniform.
- **Config format coverage**: When a module parameter or config object is tested