rsyslog/plugins/external/INTERFACE.md

What?

This file describes the interface between rsyslog and external output plugins.

Basic Facts

Rsyslog uses stdin and stdout to communicate with the external plugin. This is an established mode of interprocess communication and well supported by all decent languages. Parameters, if any, will be passed in via command line arguments, which should also be easy to obtain in (almost) all languages. Where this is not the case, it is suggested to either use an external config file or hardcode the parameters inside the plugin code.

How the plugin receives messages

Rsyslog pushes messages via stdin. Each message is terminated by a LF. So a plugin can obtain a full message simply by reading a line.

This can cause problems with multi-line messages. There are some cures for this. The recommended one is to use JSON message format (more on message formats below). This will encode LF as "\n" (by JSON RFC requirements) and thus will ensure there are no embedded LFs even in multiline messages. An alternative is to use a message format which contains some other delimiter and program the plugin to watch for this delimiter. With the near-universal availability of JSON libraries in languages these days, we strongly think that the JSON approach is superior.

The message format is generated by standard rsyslog methods, that is via a template. This gives full flexibility over what the plugin is fed. Unfortunately this also means the necessary template definitions are needed. See the rsyslog doc for how to do that (in the future, this file will also contain some samples).

Providing Feedback to Rsyslog

The plugin may convey error information to rsyslog. To do this, set the confirmMessages flag to on in the omprog action configuration (this flag is disabled by default). When this flag is enabled, rsyslog will wait for a confirmation from the plugin after sending every log message to it.

The plugin must confirm the message by writing a line with the word OK to its standard output. That is, the plugin must write the characters O, K and LF (line feed) to stdout.

If the plugin writes a line to stdout containing anything else (for example, the string Error: could not connect to the database followed by a LF), rsyslog will consider that the plugin has not been able to process the message. The message will be retried later, according to the retry settings configured for the omprog action (e.g. the action.resumeInterval setting). When debugging is enabled in rsyslog, the line returned by the plugin will be included in the debug logs.

If the plugin terminates, the message is also considered as non-processed. The plugin will later be restarted, and the message retried, according to the configured retry settings.

When starting the plugin, if confirmMessages is on, rsyslog will also wait for the plugin to confirm its initialization. The plugin must write an OK line to stdout just after starting. If it writes anything else or terminates, rsyslog will consider the plugin initialization has failed, and will try to restart it later.

Example of exchanged messages

The following sequence illustrates the message exchanges between rsyslog and the plugin. A right arrow (=>) indicates a message read by the plugin from its stdin, and a left arrow (<=) indicates a message written by the plugin to its stdout. Note that the arrows themselves are not read or written. Each line is terminated by a LF (\n).

<= OK
=> log message 1
<= OK
=> log message 2
<= OK
...
=> log message N
<= OK

Note that the first OK does not confirm any message, but that the plugin has correctly started and is ready to receive messages. When the plugin receives an end-of-file (EOF), it must silently terminate.

Writing to stderr

Aside from confirming messages via stdout, at any moment the plugin may write anything it wants to stderr. The output setting of the omprog action allows capturing the plugin's stderr to a file, which can be useful for debugging. Apart from this facility, rsyslog will ignore the plugin's stderr.

Note: When the output setting is specified and confirmMessages is set to off, rsyslog will capture both the stdout and stderr of the plugin to the specified file. You can use this to debug your plugin if you think it is not confirming the messages as expected.

Example implementation

See this Python plugin skeleton for a featured example on how a plugin can provide feedback to rsyslog.

Batching of Messages (Transactions)

You can write a plugin that processes the messages in batches (also called transactions), instead of individually. For a general explanation on how rsyslog handles the batching of messages, see http://www.rsyslog.com/doc/v8-stable/development/dev_oplugins.html.

How to process the messages in batches (transactions)

To enable transactions, set the useTransactions flag to on in the omprog action configuration. When this flag is enabled, rsyslog will send a special message line to the plugin's stdin to indicate that a batch of log messages is going to be sent. This special message is BEGIN TRANSACTION by default, although it can be customized using the beginTransactionMark setting of omprog.

After the BEGIN TRANSACTION line, rsyslog will send the log messages in the batch, each one in its own line, and then another special message COMMIT TRANSACTION to indicate the batch has ended. (The later can be customized via the commitTransactionMark setting.)

That is:

BEGIN TRANSACTION
log message 1
log message 2
...
log message N
COMMIT TRANSACTION
BEGIN TRANSACTION
...
COMMIT TRANSACTION
BEGIN TRANSACTION
...
COMMIT TRANSACTION
...

(with a LF at the end of each line)

When transactions are enabled, rsyslog will always send log messages within a transaction block, never outside it, even when the batch consists of a single message.

How to provide feedback when using transactions

You can enable both the useTransactions and confirmMessages settings in the omprog action, only one of them, or none of them. When both settings are set to on, the plugin must confirm the BEGIN TRANSACTION and COMMIT TRANSACTION messages, and the log messages within the transaction. The log messages within the transaction can be confirmed with any of the following status codes (which the plugin must write to stdout):

• OK
• DEFER_COMMIT
• PREVIOUS_COMMITTED

Refer to http://www.rsyslog.com/doc/v8-stable/development/dev_oplugins.html for an explanation on the meaning of these status codes. You will typically need to return the DEFER_COMMIT status code, since the other codes imply a partial commit, and do not guarantee that the COMMIT TRANSACTION will be received.

The following sequence illustrates the exchanges between rsyslog and the plugin when transactions and message confirmations are enabled, and the plugin confirms the log messages within each transaction with DEFER_COMMIT:

<= OK
=> BEGIN TRANSACTION
<= OK
=> log message 1
<= DEFER_COMMIT
=> log message 2
<= DEFER_COMMIT
...
=> log message 5
<= DEFER_COMMIT
=> COMMIT TRANSACTION
<= OK
=> BEGIN TRANSACTION
<= OK
=> log message 6
<= DEFER_COMMIT
=> log message 7
<= DEFER_COMMIT
...
=> log message 10
<= DEFER_COMMIT
=> COMMIT TRANSACTION
<= OK

Example implementation

For a reference example of a plugin with transaction support, see this Python plugin skeleton.

Threading Model

Write your plugin as you would do in a single threaded environment. Rsyslog automatically detects when it is time to spawn additional threads. If it decides so, it will also spawn another instance of your script and feed it concurrently. Note that rsyslog will also terminate instances that it knows are no longer needed.

If your plugin for some reason cannot be run in multiple instances, there are ways to tell rsyslog to work with a single instance. But it is strongly suggested to not restrict rsyslog to do that. Multiple instances in almost all cases do NOT mean any burden to the plugin developer. Just think of two (or more) independent instances of your program running in different console windows. If that is no problem, rsyslog running multiple instances of it is also no problem.

Future Enhancements

Interfaces for external input, filter and message modification plugins are planned. Most probably, they will become available in the order mentioned in the last sentence.

External Message Modification Modules

The external plugin will use stdin to receive the message that it potentially can modify. The message will be LF-terminated, and no LF must be present within the message itself. By default, the MSG part of the message is provided as input. The "interface.input" parameter can be used to modify this. It may has the following values:

• "msg" (the default)
• "rawmsg", which is the complete message (including header) as received by rsyslog
• "json", which is the complete message object (with all properties broken out) as a JSON object string. This is the "jsonmesg" dynamic message property.

Note: if multi-line messages are to be processed, JSON representation must be used, otherwise errors will happen.

The ability to use non-JSON representations is primarily a performance enhancement. Building the JSON representation causes some overhead, and very often access to either msg or rawmsg is fully sufficient.

The plugin will emit a JSON representation of those properties that need to be modified and their new values to stdout. Again, this is delimited by LF, with no LF permitted inside the JSON representation. Only properties that are to be changed must be included in the response. Unchanged properties should NOT be included in the response, as this would increase processing cost. If no property is to be modified, an empty JSON representation is to be provided.

The plugin must emit one response line for each message (line) received, and must do so in the same order in which the messages were put in stdin. Note that like the output module interface, multiple instances of the plugin may be activated. See above for more information.

Most message properties can be modified. Modifiable are:

• rawmsg
• msg
• syslogtag
• syslogfacility
• syslogseverity
• msgid
• procid
• structured-data
• hostname (aliased "source")
• fromhost
• fromhost-ip
• all message variable ("$!" tree)

If the message variable tree is modified, new variables may also be added. Deletion of message variables is not directly supported. If this is desired, it is suggested to set the variable in question to the empty string ("").

Implementation

The plugin interface is implemented via the "mmexternal" native plugin. See its documentation on how to tie your plugin into rsyslog's procesing flow.