Merge pull request #5952 from rgerhards/doc-omkafka-env

doc: omkafka SASL-from-env example + backticks doc overhaul

doc/source/configuration/modules/omkafka.rst

@@ -258,3 +258,78 @@ comma-delimited list of values as shown here:
          )
 
 
+.. _omkafka-sasl-password-from-env:
+
+Set SASL password from an environment variable
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. versionchanged:: 8.2508.0
+
+   Backticks in RainerScript support the ``${VAR}`` form and adjacent text.
+   This enables a simpler inline configuration such as::
+
+      `echo sasl.password=${KAFKA_PASSWORD}`
+
+**Recommended (rsyslog v8.2508.0 and later)**
+
+Keep the secret out of *rsyslog.conf* and inject it via environment.
+Then build the ``key=value`` pair inline with backticks:
+
+.. code-block:: bash
+
+   # set by your service manager or a secure env file
+   export KAFKA_PASSWORD='supersecret'
+
+.. code-block:: rsyslog
+
+   action(
+     type="omkafka"
+     broker=["kafka.example.com:9093"]
+     confParam=[
+       "security.protocol=SASL_SSL",
+       "sasl.mechanism=SCRAM-SHA-512",
+       "sasl.username=myuser",
+       `echo sasl.password=${KAFKA_PASSWORD}`
+     ]
+   )
+
+Notes:
+
+- This relies on the enhanced backtick handling; it is **not** a general shell.
+  Only the documented backtick subset (notably ``echo`` and ``cat``) is supported.
+- The variable expansion happens at rsyslog parse time, using the process
+  environment of the rsyslog daemon.
+
+**Older rsyslog versions (before v8.2508.0)**
+
+Backticks did **not** understand ``${VAR}`` or adjacency. Inline forms like
+`` `echo sasl.password=$KAFKA_PASSWORD` `` could cause errors such as
+“missing equal sign in parameter”. Use a pre-composed environment variable that
+already contains the full ``key=value`` pair and echo **that**:
+
+.. code-block:: bash
+
+   export KAFKA_PASSWORD='supersecret'
+   # Pre-compose the full key=value (done *outside* rsyslog)
+   export SASL_PWDPARAM="sasl.password=${KAFKA_PASSWORD}"
+
+.. code-block:: rsyslog
+
+   action(
+     type="omkafka"
+     broker=["kafka.example.com:9093"]
+     confParam=[
+       "security.protocol=SASL_SSL",
+       "sasl.mechanism=SCRAM-SHA-512",
+       "sasl.username=myuser",
+       `echo $SASL_PWDPARAM`
+     ]
+   )
+
+Security guidance
+^^^^^^^^^^^^^^^^^
+
+- Prefer environment files or service manager mechanisms with strict permissions
+  over embedding secrets directly in *rsyslog.conf*.
+- Process environments may be visible to privileged users (e.g., via ``/proc``);
+  secure host access accordingly.

doc/source/rainerscript/constant_strings.rst

@@ -1,92 +1,255 @@
-String Constants
-================
+Rsyslog Parameter String Constants
+==================================
 
-String constants are necessary in any scripting language. They provide
-values that are evaluated at startup and never change during rsyslog's
-run.
+String constants provide values that are evaluated at startup and remain unchanged
+for the lifetime of the rsyslog process.
 
 Uses
 ----
-String constants are necessary in many places: comparisons,
-configuration parameter values and function arguments, to name a few
-important ones.
 
-In string constants, special characters are escaped by prepending a
-backslash in front of them -- just in the same way this is done in the C
-programming language or PHP.
+String constants are used in comparisons, configuration parameter values, and
+function arguments, among other places.
 
-If in doubt how to properly escape, use the `RainerScript String Escape
-Online
-Tool <http://www.rsyslog.com/rainerscript-constant-string-escaper/>`_.
+Escaping
+--------
+
+In string constants, special characters are escaped by prepending a backslash,
+similar to C or PHP.
+
+If in doubt how to escape properly, use the
+`RainerScript String Escape Online Tool
+<http://www.rsyslog.com/rainerscript-constant-string-escaper/>`_.
 
 Types
 -----
 
-Rsyslog provides different types of string constants, closely inspired
-by the shell:
+Rsyslog provides different types of string constants, inspired by common shell
+semantics.
 
-- single quotes
+Single quotes
+.............
 
-  Values are used unaltered, except for escape sequences, which are
-  escaped.
+Values are used unaltered, except that escape sequences are processed.
 
-- double quotes
+Double quotes
+.............
 
-  Right now, equivalent to single quotes, but $ signs need to be escaped.
-  If not escaped, a syntax error will be generated and rsyslog startup
-  be aborted in most cases.
-  The idea is to support environment variables just like the shell does
-  in later releases.
+Equivalent to single quotes; a literal dollar sign must be escaped (``\$``).
+If ``$`` is not escaped, a syntax error is raised and rsyslog usually refuses
+to start.
 
-- backticks
+Backticks
+.........
 
-  This was added in 8.33.0. The idea is to provide a useful subset of
-  what the shell does. Right now, only the following is supported:
+.. versionadded:: 8.33.0
+   Initial support for backticks (limited subset).
 
-  - `echo $VARNAME` - It will evaluate the environment variable and use
-    it as string constant.  If the variable is not found, an empty string
-    is generated (this is **not** an error).
+.. versionchanged:: 8.37.0
+   Multiple environment variable expansions and constant text between them.
 
-    Starting with 8.37.0, the `echo` case has been enhanced. It is now
-    more along the lines of what bash does. It supports multiple
-    environment variable expansions as well as constant text
-    between them.
+.. versionchanged:: 8.2508.0
+   Support for brace-style variables (``${VAR}``), correct termination of
+   unbraced ``$VAR`` at the first non-``[A-Za-z0-9_]`` character, and support
+   for variable text **adjacent** to static text (e.g., key/value pairs).
 
-    An example:
+Backticks intentionally implement a **small, safe subset** of shell-like
+behavior. Currently, only the following forms are supported:
 
-    * env SOMEPATH is set to "/var/log/custompath"
-    * config is: param=echo $SOMEPATH/myfile
-    * param than is expanded to "/var/log/custompath/myfile"
+- ``echo …`` — evaluate simple text with environment variable expansion.
+- ``cat <filename>`` — include the contents of a single file.
+  If the file is unreadable, the result is an empty string.
 
-    Note, however, that some common bash features are not supported.
-    Environment variables are terminated once a character not belonging
-    to the set ``[A-Za-z0-9_]`` is encountered or by the closing brace in
-    ``${VAR}`` syntax. Constructs like ``$(pwd)`` remain unsupported. The idea
-    of this parameter is not to provide a
-    full-blown bash-equivalent, but provide some functionality that is
-    usually considered useful for customizing rsyslog configuration with
-    outside data.  For example, an expression like::
+Any other construct results in a parse error. There must be **exactly one space**
+between ``echo`` or ``cat`` and the following argument.
 
-        `echo Log-$HOSTNAME!`
+Environment variable expansion rules
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-    expands to ``Log-<yourhost>!`` where ``$HOSTNAME`` is replaced and the
-    exclamation mark follows as literal text. The braces form ``${HOSTNAME}``
-    can also be used, e.g. `` `echo {${HOSTNAME}}` ``, which results in
-    ``{<yourhost>}``. That said, we are still
-    interested in extending the coverage if clear need and reasoning is
-    provided.
+- Both ``$VAR`` and ``${VAR}`` are supported.
+- For unbraced ``$VAR``, the variable name ends at the first character
+  **not** in ``[A-Za-z0-9_]``; that character is emitted literally.
+  This makes expressions like ``$VAR!`` or ``$VAR.suffix`` work as expected.
+- For braced ``${VAR}``, expansion ends at the matching ``}``, which naturally
+  supports **adjacent** static text (e.g., ``prefix${VAR}suffix``).
+- Unknown variables expand to the empty string (not an error).
+- Command substitutions and other shell features (e.g., ``$(pwd)``) are
+  **not supported** by design.
 
-  - `cat filename` - It will evaluate to the content of the given file.
-    Only a single file name is supported. If the file is not readable,
-    it will evaluate to an empty string.
+Examples
+========
 
-  Any other construct will currently lead to an error message.
-  Note that there must be exactly one space between "echo" or "cat" and
-  the other parameter.
+Simple expansion
+----------------
 
-  Backticks are especially useful for configuration files that are
-  auto-generated but need to contain a small set of special functionality.
+Given ``SOMEPATH=/var/log/custompath``:
 
-  For an example of this in action, have a look at the rsyslog docker
-  appliance available at
-  https://github.com/rsyslog/rsyslog-docker/tree/master/appliance/alpine.
+.. code-block:: rsyslog
+
+   param = `echo $SOMEPATH/myfile`
+   # expands to: /var/log/custompath/myfile
+
+Brace form and adjacent text
+----------------------------
+
+Given ``HOSTNAME=myhost``:
+
+.. code-block:: rsyslog
+
+   title = `echo Log-${HOSTNAME}!`
+   # expands to: Log-myhost!
+
+Key/value pairs (common with module parameters)
+-----------------------------------------------
+
+Given ``KAFKA_PASSWORD=supersecret`` and rsyslog >= 8.2508.0:
+
+.. code-block:: rsyslog
+
+   action(
+     type="omkafka"
+     broker=["kafka.example.com:9093"]
+     confParam=[
+       "security.protocol=SASL_SSL",
+       "sasl.mechanism=SCRAM-SHA-512",
+       "sasl.username=myuser",
+       `echo sasl.password=${KAFKA_PASSWORD}`
+     ]
+   )
+
+.. index::
+   single: config.enabled
+   single: environment variables; config.enabled
+   single: systemd; environment variables
+   single: imtcp; conditional loading
+   pair: containers; environment-style configuration
+
+Conditional enablement of configuration objects via ``config.enabled``
+----------------------------------------------------------------------
+
+You can toggle modules, inputs, actions, etc. based on environment variables by
+setting the generic boolean parameter ``config.enabled``. The example below loads
+``imtcp`` and starts a TCP listener only if ``LOAD_IMTCP`` evaluates to a
+truthy value.
+
+**systemd service (recommended on most distributions)**
+
+Use a drop-in to set the environment variable in rsyslog’s service context:
+
+.. code-block:: bash
+
+   sudo mkdir -p /etc/systemd/system/rsyslog.service.d
+   sudo tee /etc/systemd/system/rsyslog.service.d/10-imtcp.conf >/dev/null <<'EOF'
+   [Service]
+   # Enable
+   Environment=LOAD_IMTCP=on
+   EOF
+
+   sudo systemctl daemon-reload
+   sudo systemctl restart rsyslog
+
+Alternatively, you can point to an ``EnvironmentFile=`` that contains
+``LOAD_IMTCP=…``.
+
+.. code-block:: ini
+
+   # /etc/systemd/system/rsyslog.service.d/10-imtcp.conf
+   [Service]
+   EnvironmentFile=-/etc/default/rsyslog
+
+   # /etc/default/rsyslog
+   LOAD_IMTCP=on
+
+**Bash / non-systemd service environment**
+
+If rsyslog is started from a shell wrapper (e.g., SysV init or container entrypoint):
+
+.. code-block:: bash
+
+   # enable (any of 1, true, on, yes will do)
+   export LOAD_IMTCP=on
+
+   # or disable
+   # export LOAD_IMTCP=off
+
+**RainerScript (rsyslog.conf)**
+
+.. code-block:: rsyslog
+
+   # Load the module conditionally. If LOAD_IMTCP is unset/empty/0/false,
+   # the statement is disabled and the module is not loaded.
+   module(
+     load="imtcp"
+     config.enabled=`echo ${LOAD_IMTCP}`
+   )
+
+   # Start a TCP input only when enabled. This also works if you prefer
+   # to always load the module but gate the input itself.
+   input(
+     type="imtcp"
+     port="514"
+     ruleset="in_tcp"
+     config.enabled=`echo ${LOAD_IMTCP}`
+   )
+
+   ruleset(name="in_tcp") {
+     action(type="omfile" file="/var/log/remote/tcp.log")
+   }
+
+Accepted truthy value for ``config.enabled`` is ``on``. Any other value,
+including no value, is treated as false.
+
+.. note::
+
+   In containerized deployments, this pattern is often the simplest way to pass
+   environment-style toggles into a static configuration. For example:
+
+   .. code-block:: bash
+
+      docker run --rm \
+        -e LOAD_IMTCP=on \
+        -v /path/to/rsyslog.conf:/etc/rsyslog.conf:ro \
+        my-rsyslog-image
+
+
+Including a file’s content
+--------------------------
+
+.. code-block:: rsyslog
+
+   token = `cat /etc/rsyslog.d/token.txt`
+
+Compatibility note for older releases
+-------------------------------------
+
+On versions **before 8.2508.0**, backticks did **not** support ``${VAR}`` or
+adjacent text reliably. If you need to build a ``key=value`` pair, pre-compose
+it outside rsyslog and expand the already-complete string, e.g.:
+
+In bash or init script:
+
+.. code-block:: bash
+
+   export KAFKA_PASSWORD='supersecret'
+   export SASL_PWDPARAM="sasl.password=${KAFKA_PASSWORD}"
+
+In rsyslog conf:
+
+.. code-block:: rsyslog
+
+   action(
+     type="omkafka"
+     broker=["kafka.example.com:9093"]
+     confParam=[
+       "security.protocol=SASL_SSL",
+       "sasl.mechanism=SCRAM-SHA-512",
+       "sasl.username=myuser",
+       `echo $SASL_PWDPARAM`
+     ]
+   )
+
+Rationale
+---------
+
+Backticks are intended to provide just enough functionality to customize
+configuration with external data while avoiding the complexity and risks of a
+general shell interpreter.