mirror of
https://github.com/rsyslog/rsyslog.git
synced 2026-06-05 00:32:57 +02:00
tests: document diagnostic oracle expectations
Why: Test assertions should prove the observable behavior operators rely on, not just that rsyslogd wrote a message to its process streams. Impact: Clarifies test-authoring and babysitting guidance for rsyslog diagnostics. Before/After: Tests could use stdout/stderr oracles without a clear exception; guidance now prefers configured rsyslog output destinations after synchronized shutdown. Technical Overview: Add the authoritative rule to tests/AGENTS.md. Point rsyslog_test at that rule when explaining test oracle documentation. Add a review-time reminder to rsyslog_pr_babysitting so PR feedback catches weak stdout/stderr diagnostic oracles. Exceptions remain allowed when the tested behavior is process output or configuration is not usable yet, but they must be documented in the test header. With the help of AI-Agents: OpenAI Codex
This commit is contained in:
parent
c4d8f4e701
commit
2b2fcd5ef5
@ -153,6 +153,12 @@ work. Handle small, localized fixes directly, validate them, and push the
|
||||
updated branch. Examples include typo fixes, documentation wording, metadata
|
||||
formatting, simple example corrections, and narrow test expectation updates.
|
||||
|
||||
When reviewing or adjusting test assertions, apply the oracle rule from
|
||||
`tests/AGENTS.md`: diagnostics emitted by rsyslog itself should normally be
|
||||
asserted through the configured rsyslog output destination after synchronized
|
||||
shutdown, not through rsyslogd stdout/stderr. Treat stdout/stderr diagnostic
|
||||
oracles as actionable unless the test documents a specific exception.
|
||||
|
||||
AI review comments need an explicit GitHub reply so later readers know the
|
||||
comment was considered. If the requested change was implemented, a short
|
||||
`Done.` reply is sufficient. If the comment is invalid, not applicable, or not
|
||||
|
||||
@ -63,6 +63,12 @@ When changing a test, verify that the head comment still matches the actual
|
||||
setup, stimulus, oracle, and pass/fail conditions after the edit; update it in
|
||||
the same commit if it does not.
|
||||
|
||||
For diagnostics emitted by rsyslog itself, follow `tests/AGENTS.md`: assert the
|
||||
configured rsyslog output destination, usually testbench omfile output such as
|
||||
`RSYSLOG_OUT_LOG`, after synchronized shutdown. Avoid rsyslogd stdout/stderr as
|
||||
the oracle unless the test is specifically about process-level output or the
|
||||
exception is documented in the test header.
|
||||
|
||||
### 4. Using diag.sh Helpers
|
||||
All tests include `tests/diag.sh` using the POSIX `.` command. You should use its standardized helpers:
|
||||
- `cmp_exact`: Verify file content matches.
|
||||
|
||||
@ -31,6 +31,14 @@ agents.
|
||||
When changing a test, verify that the head comment still matches the actual
|
||||
setup, stimulus, oracle, and pass/fail conditions after the edit; update it in
|
||||
the same commit if it does not.
|
||||
- For diagnostics emitted by rsyslog itself, prefer asserting the configured
|
||||
rsyslog output destination, usually testbench omfile output such as
|
||||
`RSYSLOG_OUT_LOG`, after synchronized shutdown. Do not use rsyslogd
|
||||
stdout/stderr as the oracle
|
||||
unless the behavior being tested is specifically process-level stdout/stderr
|
||||
emission, startup before configuration is usable, or another documented case
|
||||
where the message cannot pass through rsyslog's normal output path. Explain
|
||||
such exceptions in the test header.
|
||||
- Prefer harness helpers such as `cmp_exact`, `command_deny`, and
|
||||
`require_plugin` over ad-hoc shell to keep diagnostics uniform.
|
||||
- **Config format coverage**: When a module parameter or config object is tested
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user