Architectural Analysis of Rsyslog Core (#6390)

doc: add and improve code architecture doc

This adds information to both user- and AI Agent facing doc components. The Architecture was verified via code review, a joint effort between the Google Jules AI Agent and Rainer Gerhards, rsyslog lead maintainer.

AGENTS.md

@@ -49,7 +49,8 @@ Run a relevant test to verify your changes. The testbench allows test scripts to
 
   - **Primary Language**: C
   - **Build System**: autotools (`autogen.sh`, `configure`, `make`)
-  - **Modules**: Dynamically loaded from `plugins/`
+  - **Architecture**: Microkernel-like core (`runtime/`) with loadable plugins (`plugins/`)
+  - **Modules**: Dynamically loaded from `plugins/` (note: legacy modules like `omfile` reside in `tools/`)
   - **Contrib Modules**: Community-contributed under `contrib/`
   - **Contributions**: Additional modules and features are placed in `contrib/`, which contains community-contributed plugins not actively maintained by the core rsyslog team. These are retained in `contrib/` even if adopted later, to avoid disruptions in dependent software.
   - **Documentation**: Maintained in the doc/ subdirectory
@@ -77,6 +78,7 @@ Run a relevant test to verify your changes. The testbench allows test scripts to
   - **Inline comment conventions:** [`COMMENTING_STYLE.md`](./COMMENTING_STYLE.md)
   - **Module author checklist:** [`MODULE_AUTHOR_CHECKLIST.md`](./MODULE_AUTHOR_CHECKLIST.md)
   - **Developer overview:** [`DEVELOPING.md`](./DEVELOPING.md)
+  - **Architecture overview:** [`doc/source/development/architecture.rst`](./doc/source/development/architecture.rst)
   - **Commit prompt template:** [`ai/rsyslog_commit_assistant/base_prompt.txt`](./ai/rsyslog_commit_assistant/base_prompt.txt)
   - **Doc builder prompt template:** [`ai/rsyslog_code_doc_builder/base_prompt.txt`](./ai/rsyslog_code_doc_builder/base_prompt.txt)
 

Makefile.am

@@ -261,6 +261,7 @@ EXTRA_DIST += \
     doc/source/containers/standard.rst \
     doc/source/containers/user_images.rst \
     doc/source/dataflow.png \
+    doc/source/development/architecture.rst \
     doc/source/development/coding_practices.rst \
     doc/source/development/coding_practices/embedded_ipv4_tail_helper.rst \
     doc/source/development/coding_practices/error_logging_with_errno.rst \

README.md

@@ -2,7 +2,7 @@
 
 **Rsyslog** is a **r**ocket-fast **sys**tem for **log** processing pipelines.
 
-It offers high performance, advanced security features, and a modular design.  
+It offers high performance, advanced security features, and a **modular microkernel-like architecture**.
 Originally a regular syslogd, rsyslog has evolved into a highly versatile logging solution capable of ingesting data from numerous sources, transforming it, and outputting it to a wide variety of destinations.
 
 Rsyslog can deliver over one million messages per second to local destinations under minimal processing (based on v7, Dec 2013). Even with complex routing and remote forwarding, performance remains excellent.

doc/source/development/architecture.rst

@@ -0,0 +1,99 @@
+.. _dev_architecture:
+
+System Architecture
+===================
+
+.. meta::
+   :description: High-level architectural overview of rsyslog, describing the microkernel pattern, core components, and data flow pipeline.
+   :keywords: architecture, microkernel, plugins, data flow, pipeline, core engine, components
+
+.. page-summary-start
+
+High-level architectural overview of rsyslog, describing the microkernel pattern,
+core components, and data flow pipeline.
+
+.. page-summary-end
+
+Rsyslog follows a **Microkernel-like Architecture** (also known as a Plugin Architecture).
+The core system provides essential services—threading, queueing, configuration parsing,
+and object management—while the actual business logic of receiving, processing,
+and sending logs is implemented in dynamically loadable modules.
+
+Architectural Patterns
+----------------------
+
+Microkernel (Core vs. Plugins)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The system is cleanly divided into two layers:
+
+1.  **The Core (Runtime):**
+    Located in ``runtime/``, this acts as the "kernel." It knows nothing about
+    TCP, files, or Elasticsearch. It manages:
+
+    -   **Resources:** Worker threads (``wtp.c``), memory, and queues.
+    -   **Orchestration:** Loading plugins, parsing `rsyslog.conf`, and routing messages.
+    -   **Interfaces:** Defines the contract (``modules.h``) that all plugins must obey.
+
+2.  **The Plugins (Extensions):**
+    Located in ``plugins/`` (and some legacy ones in ``tools/``), these provide
+    functionality. They are classified by their prefix:
+
+    -   **Input (`im`):** Producers (e.g., `imtcp`, `imfile`).
+    -   **Output (`om`):** Consumers (e.g., `omelasticsearch`, `omfile`).
+    -   **Parser (`pm`):** Decoders (e.g., `pmrfc5424`).
+    -   **Message Modification (`mm`):** Transformers (e.g., `mmjsonparse`).
+    -   **Function (`fm`):** Extension functions for RainerScript.
+
+Data Flow Pipeline
+------------------
+The system operates as an **Event-Driven, Multi-threaded Producer-Consumer Pipeline**.
+
+.. mermaid::
+
+   flowchart LR
+     Source(("Log Source")) --> Input["Input Module<br>(imtcp)"]
+     Input -->|smsg_t| PreProc["Preprocessing"]
+     PreProc --> Queue["Main Queue<br>(qqueue_t)"]
+     Queue -->|Batch| Parser["Parsers"]
+     Parser --> Rules["Ruleset Engine"]
+     Rules -->|Filter| ActionQ["Action Queue"]
+     ActionQ --> Output["Output Module<br>(omfile)"]
+     Output --> Dest(("Destination"))
+
+1.  **Input (Producer):** An input module runs a thread (often an infinite loop like `runInput`) to listen for data. It creates a **Message Object** (``smsg_t``) and submits it to the core.
+2.  **Queue (Buffer):** The core ``qqueue_t`` buffers messages in memory or on disk to handle bursts and ensure reliability.
+3.  **Processing (Orchestration):** Worker threads dequeue messages. The parser chain decodes the raw message. The ruleset engine evaluates filters.
+4.  **Action (Consumer):** If a filter matches, the message is passed to an Action. The action invokes an Output Module via a standard interface (``doAction`` or Transactional).
+
+Component Map
+-------------
+
+- **``tools/``**: Contains the **Entry Point** (``rsyslogd.c``) and legacy built-in modules (e.g., `omfile.c`, `ompipe.c`).
+- **``runtime/``**: The **Core Engine**.
+
+    -   ``msg.c``: The Message object.
+    -   ``queue.c``: Queue implementation (In-Memory, Disk, DA).
+    -   ``modules.c``: Plugin loader and interface definitions.
+    -   ``obj.c``: The internal Object System.
+- **``plugins/``**: The **Extension Ecosystem**. Contains the majority of modern modules.
+- **``contrib/``**: Community-contributed modules. These are functionally identical to plugins but have different maintenance guarantees.
+
+The Object System
+-----------------
+Rsyslog implements a custom **Object System** in C (``runtime/obj.h``) to provide encapsulation and polymorphism.
+
+-   **Macros:** It relies heavily on macros like ``DEFobjCurrIf`` (Define Object Current Interface) and ``BEGINobjInstance`` to simulate classes.
+-   **Lifecycle:** Objects follow a strict constructor/destructor pattern:
+
+    -   ``Construct(&pThis)``: Allocate memory.
+    -   ``SetProperty(pThis, ...)``: Configure the instance.
+    -   ``ConstructFinalize(pThis)``: Ready the instance for use.
+    -   ``Destruct(&pThis)``: Teardown and free.
+
+See Also
+--------
+
+- :doc:`engine_overview`
+- :doc:`dev_action_threads`
+- :doc:`dev_queue`
+- :doc:`generic_design`

doc/source/development/index.rst

@@ -4,4 +4,5 @@ Development
 .. toctree::
    :glob:
 
+   architecture
    *