diff --git a/doc/IMPLEMENTATION_PLAN.md b/doc/IMPLEMENTATION_PLAN.md new file mode 100644 index 000000000..6d45a07b1 --- /dev/null +++ b/doc/IMPLEMENTATION_PLAN.md @@ -0,0 +1,299 @@ +# Rsyslog Documentation: Implementation & Quality Improvement Plan + +**Version: 1.0** + +**Related Document:** `STRATEGY.md` + +## 1. Introduction + +This document is the practical playbook for executing the official Rsyslog Documentation Strategy. It breaks down the high-level strategy into concrete phases and actionable tasks. Its purpose is to guide the development team through the setup, migration, automation, and long-term improvement of the documentation. + +--- + +## 2. Phase 1: Foundational Setup + +This phase focuses on creating the new structure and environment. + +### Task 2.1: Establish the New Directory Structure + +The first step is to create the new, hierarchical directory structure within `doc/source/`. This provides a clean foundation for all subsequent work. + +**Action:** Create the following directories and move the existing `.rst` files into their most logical new home. For files that are difficult to categorize, place them in a temporary `_needs_triage` directory. + +``` + +doc/ +├── examples/ \# For validated, standalone rsyslog.conf examples +└── source/ +├── \_static/ \# For CSS, images, etc. +├── \_templates/ \# For custom HTML templates +├── \_needs\_triage/ \# Temporary home for uncategorized old files +├── 01\_getting\_started/ +├── 02\_concepts/ +├── 03\_configuration/ +├── 04\_modules/ +├── 05\_how-to\_guides/ +├── 06\_reference/ +├── 07\_tutorials/ +├── 08\_deployment\_guides/ +│ ├── container\_deployment/ +│ └── aws\_deployment/ +└── man/ \# For the RST source of man pages + +```` + +### Task 2.2: Configure `conf.py` for the New System + +To power our new features, we must update the Sphinx configuration file. + +**Action:** Edit `doc/source/conf.py` and ensure the following extensions are added to the `extensions` list. + +```python +# doc/source/conf.py + +extensions = [ + 'sphinx.ext.autodoc', + 'sphinx.ext.intersphinx', # For linking to external docs + 'sphinx.ext.githubpages', + 'sphinx_mermaid', # For Mermaid diagrams + 'sphinx_sitemap', # For generating sitemap.xml + # 'breathe', # Add this once Doxygen is set up +] + +# Add this configuration for intersphinx +intersphinx_mapping = { + 'agent': ('URL_TO_WINDOWS_AGENT_DOCS', None), +} + +# Add this for sitemap generation +html_baseurl = '[https://www.rsyslog.com/doc/](https://www.rsyslog.com/doc/)' + +# Add this for the "Edit on GitHub" link +html_context = { + "display_github": True, + "github_user": "rsyslog", + "github_repo": "rsyslog", + "github_version": "master", + "conf_py_path": "/doc/source/", +} +```` + +----- + +## 3\. Phase 2: Content Migration and Enhancement + +This phase involves moving existing content into the new system and refactoring it to meet our quality standards. + +### Task 3.1: Migrating Man Pages to RST + +**Goal:** Convert a legacy man page into a Sphinx-manageable RST file. + +**Steps:** + +1. **Create RST File:** For `rsyslogd.8`, create a new file at `doc/source/man/rsyslogd.rst`. +2. **Copy Content:** Copy the raw text content from the old man page into the new RST file. +3. **Add Metadata:** Add the required metadata fields at the top of the RST file. This is what the man page builder uses to create the file correctly. + ```rst + .. + :manpage_url: [https://www.rsyslog.com/doc/man/rsyslogd.8.html](https://www.rsyslog.com/doc/man/rsyslogd.8.html) + + rsyslogd + ######## + + :program:`rsyslogd` - reliable and extended syslogd + + .. + This is the rst source for the rsyslogd man page. + It is generated as part of the rsyslog build process. + + .. program:: rsyslogd + + .. sectionauthor:: Rainer Gerhards + + .. + (rest of man page content follows, formatted in RST) + ``` +4. **Update Makefile:** Ensure the `Makefile` contains a target to build the man pages. + ```makefile + # In your main Makefile + man: + @$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man + ``` +5. **Build and Validate:** Run `make man` and check the output in the `build/man` directory. + +### Task 3.2: Building the Validated Example Library + +**Goal:** Create a repository of tested, reliable configuration examples. + +**Steps:** + +1. **Identify Candidates:** Review the existing rsyslog testbench to identify self-contained, high-value configuration examples. +2. **Extract and Sanitize:** Copy these configurations into new, clean `.conf` files within the `doc/examples/` directory (e.g., `doc/examples/basic_forwarding.conf`). Remove any testbench-specific scaffolding. +3. **Create Validation Script:** Create a CI script that iterates through all files in `doc/examples/` and runs `rsyslogd -N1` against them to check for syntax validity. +4. **Refactor Existing Docs:** Search the documentation for inline `.. code-block::` sections that contain full configurations. Replace them with the `.. literalinclude::` directive pointing to the corresponding validated file. + ```rst + .. literalinclude:: ../examples/basic_forwarding.conf + :language: rsyslog-conf + ``` + +----- + +## 4\. Phase 3: Automation and CI Integration + +This phase involves the core engineering work to automate our quality gates. + +### Task 4.1: Implement the "Drift Detector" Linter + +**Goal:** Create a CI script that automatically detects technical inaccuracies in the documentation. + +**Specification:** + + * **Language:** Python (recommended, for ease of integration with Sphinx). + * **Input:** + 1. An "authoritative spec" file (`spec.json`) generated by the code analysis agent, containing a list of all valid parameters and modules. + 2. The set of all `.rst` files in the `doc/source/` directory. + * **Process:** + 1. The script reads `spec.json` into memory. + 2. It iterates through each `.rst` file. + 3. It uses regular expressions to find all potential rsyslog parameter names (e.g., `\$[a-zA-Z0-9_]+`, `[a-zA-Z]+\s*\(.*?\)`). + 4. For each term found, it checks if the term exists in the `spec.json`. + 5. If an unknown term is found, the script prints an error message indicating the term and the file/line number where it was found, then exits with a non-zero status code, failing the CI build. + +### Task 4.2: Implement the Code Analysis Agent + +**Goal:** Use Doxygen and Breathe to generate reference documentation directly from C code comments. + +**Steps:** + +1. **Define Comment Standard:** Establish a mandatory, standardized Doxygen comment block format for all developers to use. (See **Appendix B**). +2. **Configure Doxygen:** Create a `Doxyfile` configuration. Key settings: + * `GENERATE_XML = YES` + * `GENERATE_HTML = NO` + * `GENERATE_LATEX = NO` + * Set `INPUT` to the relevant source code directories. +3. **Integrate with Breathe:** + * Add `breathe` to the `extensions` list in `conf.py`. + * Configure the path to the Doxygen XML output: `breathe_projects = { "rsyslog": "path/to/doxygen/xml/" }` +4. **Create RST Stubs:** In the `06_reference/` directory, create stub `.rst` files that use Breathe directives to pull in the documentation. + ```rst + // In doc/source/06_reference/imfile.rst + + Module: imfile + ============== + + .. doxygenmodule:: imfile + :members: + :undoc-members: + ``` +5. **Generate `spec.json`:** The Doxygen XML output is machine-readable. Create a simple script that parses this XML and generates the `spec.json` file needed by the Drift Detector linter. + +### Task 4.3: Create the Full Documentation CI Pipeline + +**Goal:** A complete GitHub Actions workflow that automates all checks. + +**Action:** Create a file at `.github/workflows/docs.yml`. + +```yaml +name: 'Documentation CI' + +on: + pull_request: + paths: + - 'doc/**' + - '.github/workflows/docs.yml' + +jobs: + build_and_test_docs: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.x' + + - name: Install dependencies + run: | + python -m pip install -r doc/requirements.txt + # Add other dependencies like doxygen, etc. + + - name: Generate Authoritative Spec + run: | + # Command to run doxygen and the spec generator script + make doc-spec + + - name: Validate Examples + run: | + # Command to run the example validation script + python doc/ci/validate_examples.py + + - name: Run Drift Detector Linter + run: | + # Command to run the drift detector + python doc/ci/drift_detector.py + + - name: Build Sphinx Docs + run: | + make -C doc html +``` + +----- + +## 5\. Phase 4: Iterative Quality Improvement + +This phase is a continuous, long-term process. + +### Task 5.1: The RST Quality Rubric + +Use this checklist when refactoring an old documentation file to meet the new standard. + + * [ ] **Headings:** Are the heading levels consistent with the style guide (See **Appendix A**)? + * [ ] **Semantic Markup:** Have plain backticks (`` ` ``) been replaced with specific semantic roles where appropriate (e.g., `:program:`, `:file:`, `:option:`)? + * [ ] **Cross-References:** Are all mentions of other modules, concepts, or sections linked using the `:ref:` or `:doc:` role? + * [ ] **Examples:** Are all configuration examples either short, illustrative snippets or replaced with a `.. literalinclude::` directive pointing to a validated example? + * [ ] **Clarity:** Is the language clear, concise, and unambiguous? + +### Task 5.2: Prioritization Strategy + +It's impossible to refactor everything at once. Prioritize the work in this order: + +1. **High-Traffic Pages:** Start with the Getting Started guide, core configuration files, and the most popular modules. +2. **Community Pain Points:** Address pages that are frequently mentioned in GitHub issues or community forums as being confusing or incorrect. +3. **Core Concepts:** Ensure the files in `02_concepts/` are crystal clear, as they form the foundation of user understanding. + +----- + +## 6\. Appendices + +### Appendix A: Recommended RST Style Guide + + * **Page Title (H1):** `################` (over and under) + * **Chapter (H2):** `****************` (over and under) + * **Section (H3):** `================` (underline only) + * **Subsection (H4):** `----------------` (underline only) + * **Line Length:** Wrap all prose at 88 characters to improve readability in code editors. + +### Appendix B: Sample Standardized Code Comment (Doxygen) + +All new functions and configuration parameters in the C code should use this format. + +```c +/** + * @brief A brief, one-sentence description of the parameter. + * + * A more detailed, multi-sentence description can follow here. + * It should explain what the parameter does, its purpose, and any + * important side effects or interactions. + * + * @param[in] name The name of the parameter. (Example for a function) + * + * @default "default_value" + * @module-type output + * @supported-since 8.2404.0 + * @see another_function() + * + * @return A description of the return value. + */ +``` diff --git a/doc/README.md b/doc/README.md index 66f8997f2..ef9aded8c 100644 --- a/doc/README.md +++ b/doc/README.md @@ -31,17 +31,17 @@ provide dev and official release builds of the documentation. 1. Log in with a GitHub account. 2. Fork the official https://github.com/rsyslog/rsyslog repo. -3. Create a new branch off of the latest `master` branch. +3. Create a new branch off of the latest `main` branch. 4. Make your changes in the **`doc/`** subfolder. 5. Commit to the new branch in your fork. 6. Submit a Pull Request (PR) for review (https://github.com/rsyslog/rsyslog/pulls). **Note that PRs will be automatically AI reviewed.** 7. Stop making any changes to your new branch now that you've submitted a - Pull Request for review. Instead, create a new branch from your `master` + Pull Request for review. Instead, create a new branch from your `main` branch while you wait for feedback from the doc team. 8. A team member will review and offer feedback on your work. After feedback has been given and you've made all necessary changes, your - PR will be accepted and merged into the official `master` branch. + PR will be accepted and merged into the official `main` branch. 9. At this point, delete the branch you submitted the PR from and start a new one for your next round of work. @@ -148,12 +148,12 @@ later steps are identical, so we've covered those steps in one place. 1. `python -m pip install -r requirements.txt` 2. Clone the official Git repo: 1. `git clone https://github.com/rsyslog/rsyslog.git` -3. Check out either the current stable or development (aka, "master") branch: +3. Check out either the current stable or development (aka, "main") branch: 1. `cd rsyslog/doc` 1. `git checkout BRANCH_NAME_HERE` - Choose the `v8-stable` branch for coverage of features currently available in the latest stable release. - - Choose the `master` branch for coverage of upcoming features and fixes. + - Choose the `main` branch for coverage of upcoming features and fixes. 4. **Optional:** If you've previously cloned the repo, run `git pull` to update it with new changes before continuing. diff --git a/doc/STRATEGY.md b/doc/STRATEGY.md new file mode 100644 index 000000000..14c2e6061 --- /dev/null +++ b/doc/STRATEGY.md @@ -0,0 +1,111 @@ +# The Rsyslog Documentation Strategy + +**Version: 1.0** + +**Related Document:** `IMPLEMENTATION_PLAN.md` + +**Status: Active** + +This document outlines the official strategy for the rsyslog project's documentation. It is a living document, subject to revision as our tools and processes evolve. Its purpose is to ensure our documentation is **authoritative**, **sustainable**, and **intelligent**, serving both human system administrators and AI agents as a primary source of truth. + +## Core Principles + +1. **Docs-as-Code**: Documentation is treated with the same rigor as application code. It lives in the same repository, is version-controlled, and is integrated into our CI/CD pipelines. +2. **Single Source of Truth**: We will relentlessly eliminate information duplication. A piece of information should exist in one place and be referenced everywhere else. +3. **Automation First**: We will automate every process possible, from generation and validation to drift detection and contribution checks. +4. **Hybrid Model**: We recognize that great documentation requires both machine-driven accuracy for reference material and human-authored clarity for narrative content. + +--- + +## Pillar 1: The Automated Foundation (Code-Driven Reference) + +This pillar ensures that our reference documentation is always an accurate reflection of the source code, even if manual updates are missed during development. + +* **Authoritative Source**: The C source code is the ultimate source of truth for all modules, configuration parameters, and directives. +* **Generation Toolchain**: + * **Doxygen**: Parses standardized comment blocks and code structure from `.c` files into an intermediate XML format. + * **Sphinx**: The core documentation build system. + * **Breathe**: A Sphinx extension that bridges Doxygen and Sphinx, rendering the XML as clean HTML and man pages. +* **Handling Undocumented Changes**: + 1. A CI job runs nightly (or on commit) to analyze the source code and compare it to the current documentation. + 2. If this "code analysis agent" detects a new or changed parameter that is not documented, it will **not** fail the build. + 3. Instead, it will automatically create a **new issue** in the project's GitHub issue tracker, tagged with `documentation` and `autogenerated`. This creates a trackable "documentation debt" item. + +--- + +## Pillar 2: The Narrative Layer (Human-Authored Guides) + +This layer provides context, learning paths, and practical solutions. This content is manually authored but automatically validated. + +* **Validated Example Library**: + * All complete configuration examples will be stored in a dedicated `doc/examples/` directory. + * A CI job will test these examples for syntactic validity to ensure they are guaranteed to work. + * Tutorials and guides will **not** contain inline, hardcoded configurations. They will use Sphinx's `.. literalinclude:: ../examples/my_validated_example.conf` directive to pull in these tested examples. +* **Diagrams as Code**: + * All diagrams and schematics will be created using **Mermaid.js**. + * The `sphinx-mermaid` extension will be used to render these text-based diagrams directly within the documentation, making them version-controllable. +* **Content Migration**: High-value, existing content (such as from `rainer.gerhards.net`) will be migrated, converted to RST, and integrated directly into the `doc/source/tutorials/` section, enriching it with cross-references and standard formatting. + +--- + +## Pillar 3: Quality & Contribution Framework + +This pillar defines the processes that ensure our documentation is high-quality, trustworthy, and easy to contribute to. + +* **Documentation CI Pipeline**: Any pull request that modifies documentation will trigger a CI pipeline that: + 1. **Builds the Docs**: Runs `make html` to ensure no Sphinx errors. + 2. **Runs the "Drift Detector"**: A custom linter script scans all `.rst` files for rsyslog-specific technical terms (e.g., `$WorkDirectory`) and validates them against the authoritative list generated from the source code. An unknown term will fail the build, preventing typos and outdated information from being merged. +* **AI-Powered Review**: As a supplementary check, a GitHub Action can be configured to use an LLM to review the clarity, grammar, and style of documentation changes in a pull request, posting feedback as a comment. +* **User Feedback**: Every page will feature an "Edit on GitHub" link to provide a low-friction way for users to suggest improvements. This is configured in `conf.py` via the `html_context`. + +--- + +## Pillar 4: Distribution & Format Integration + +This pillar defines how documentation is delivered to end-users and how legacy formats are handled. + +* **Man Page Generation**: + * Man pages are not written manually. They are **generated** from the same RST source files as the HTML documentation. + * Content from existing man pages will be migrated to RST files under `doc/source/man/`. + * The `Makefile` will include the `make man` target, which uses Sphinx's man-page builder. The generated pages will be included in the release tarball. +* **Official Versioning Policy**: + > This online documentation reflects the latest development version (`main` branch) of rsyslog. It contains the most current information but may include features not yet available in a stable release. + > + > For stable versions, the authoritative documentation is bundled within the official release tarballs. Linux distributions like Red Hat, Ubuntu, and others build their documentation packages directly from these tarballs. **If you have installed rsyslog via a distribution's package manager, please refer to the documentation provided by that package as your primary source of truth.** + +--- + +## Pillar 5: Ecosystem Integration (External & Platform Docs) + +This pillar defines how we integrate with related external projects and platform-specific documentation. + +* **External Projects (e.g., Rsyslog Windows Agent)**: + * To link to related but separate documentation sets, we will use a combination of `git submodule` and the Sphinx `intersphinx` extension. + * This allows us to cross-reference content between projects seamlessly, making it feel like a single, unified documentation set to the user. +* **Cloud & Container Deployments**: + * A dedicated top-level section, `08_deployment_guides/`, will be created in the documentation. + * This section will contain sub-folders for major platforms (e.g., `container_deployment/`, `aws_deployment/`), providing comprehensive, validated guides for modern deployment scenarios. + +--- + +## Pillar 6: Content Amplification & Authority Building + +This pillar outlines our strategy to combat misinformation and establish the official documentation as the canonical source of truth. + +* **Technical SEO**: + * We will use the `sphinx-sitemap` extension to generate a `sitemap.xml` to improve indexing. + * We will embed `Schema.org` structured data (e.g., `TechArticle`, `HowTo`) into our HTML templates to provide explicit context to search engines and AI crawlers. +* **Strategic Syndication**: + * Key tutorials will be periodically published on high-authority external sites (e.g., dev.to). + * **Crucially**, these syndicated articles **must** use a `rel="canonical"` link pointing back to the original article in the official rsyslog documentation to centralize SEO authority. +* **Answer Hubs**: When answering questions on sites like Stack Overflow, community members are encouraged to provide a brief summary and then link directly to the relevant page in the official documentation. + +--- + +## Maintaining This Strategy + +This document is the "single source of truth" for our documentation process. + +* **Location**: It is stored at `doc/STRATEGY.md` within the main rsyslog repository. +* **Discoverability**: A `CONTRIBUTING.md` file at the repository root links directly to this strategy, and a pull request template (`.github/pull_request_template.md`) includes a checklist item requiring contributors to have read it. +* **Enforcement**: The strategy is primarily enforced through the automated CI checks. If the CI pipeline passes, the contribution is considered compliant.