What? ===== This file describes the interface between rsyslog and external output plugins. Basic Facts =========== Rsyslog uses stdin and stdout to communicate with the external plugin. This is an established mode of interprocess communication and well supported by all decent languages. Parameters, if any, will be passed in via command line arguments, which should also be easy to obtain in (almost) all languages. Where this is not the case, it is suggested to either use an external config file or hardcode the parameters inside the plugin code. How the plugin receives messages -------------------------------- Rsyslog pushes messages via stdin. Each message is terminated by a LF. So a plugin can obtain a full message simply by reading a line. This can cause problems with multi-line messages. There are some cures for this. The recommended one is to use JSON message format (more on message formats below). This will encode LF as "\n" (by JSON RFC requirements) and thus will ensure there are no embedded LFs even in multiline messages. An alternative is to use a message format which contains some other delimiter and program the plugin to watch for this delimiter. With the near-universal availability of JSON libraries in languages these days, we strongly think that the JSON approach is superior. The _message format_ is generated by standard rsyslog methods, that is via a template. This gives full flexibility over what the plugin is fed. Unfortunately this also means the necessary template definitions are needed. See the rsyslog doc for how to do that (in the future, this file will also contain some samples). Providing Feedback to Rsyslog ============================= The plugin may convey error information to rsyslog. To do this, set the `confirmMessages` flag to `on` in the `omprog` action configuration (this flag is disabled by default). When this flag is enabled, rsyslog will wait for a confirmation from the plugin after sending every log message to it. The plugin must confirm the message by writing a line with the word `OK` to its standard output. That is, the plugin must write the characters `O`, `K` and LF (line feed) to stdout. If the plugin writes a line to stdout containing anything else (for example, the string `Error: could not connect to the database` followed by a LF), rsyslog will consider that the plugin has not been able to process the message. The message will be retried later, according to the retry settings configured for the `omprog` action (e.g. the `action.resumeInterval` setting). When debugging is enabled in rsyslog, the line returned by the plugin will be included in the debug logs. If the plugin terminates, the message is also considered as non-processed. The plugin will later be restarted, and the message retried, according to the configured retry settings. When starting the plugin, if `confirmMessages` is `on`, rsyslog will also wait for the plugin to confirm its initialization. The plugin must write an `OK` line to stdout just after starting. If it writes anything else or terminates, rsyslog will consider the plugin initialization has failed, and will try to restart it later. Example of exchanged messages ----------------------------- The following sequence illustrates the message exchanges between rsyslog and the plugin. A right arrow (`=>`) indicates a message read by the plugin from its stdin, and a left arrow (`<=`) indicates a message written by the plugin to its stdout. Note that the arrows themselves are not read or written. Each line is terminated by a LF (\n). <= OK => log message 1 <= OK => log message 2 <= OK ... => log message N <= OK Note that the first `OK` does not confirm any message, but that the plugin has correctly started and is ready to receive messages. When the plugin receives an end-of-file (EOF), it must silently terminate. Writing to stderr ----------------- Aside from confirming messages via stdout, at any moment the plugin may write anything it wants to stderr. The `output` setting of the `omprog` action allows capturing the plugin's stderr to a file, which can be useful for debugging. Apart from this facility, rsyslog will ignore the plugin's stderr. Note: When the `output` setting is specified and `confirmMessages` is set to `off`, rsyslog will capture both the stdout and stderr of the plugin to the specified file. You can use this to debug your plugin if you think it is not confirming the messages as expected. Example implementation ---------------------- See [this Python plugin skeleton](skeletons/python/plugin-with-feedback.py) for a featured example on how a plugin can provide feedback to rsyslog. Batching of Messages (Transactions) =================================== You can write a plugin that processes the messages in batches (also called _transactions_), instead of individually. For a general explanation on how rsyslog handles the batching of messages, see http://www.rsyslog.com/doc/v8-stable/development/dev_oplugins.html. How to process the messages in batches (transactions) ----------------------------------------------------- To enable transactions, set the `useTransactions` flag to `on` in the `omprog` action configuration. When this flag is enabled, rsyslog will send a special message line to the plugin's stdin to indicate that a batch of log messages is going to be sent. This special message is `BEGIN TRANSACTION` by default, although it can be customized using the `beginTransactionMark` setting of `omprog`. After the `BEGIN TRANSACTION` line, rsyslog will send the log messages in the batch, each one in its own line, and then another special message `COMMIT TRANSACTION` to indicate the batch has ended. (The later can be customized via the `commitTransactionMark` setting.) That is: BEGIN TRANSACTION log message 1 log message 2 ... log message N COMMIT TRANSACTION BEGIN TRANSACTION ... COMMIT TRANSACTION BEGIN TRANSACTION ... COMMIT TRANSACTION ... (with a LF at the end of each line) When transactions are enabled, rsyslog will always send log messages within a transaction block, never outside it, even when the batch consists of a single message. How to provide feedback when using transactions ----------------------------------------------- You can enable both the `useTransactions` and `confirmMessages` settings in the `omprog` action, only one of them, or none of them. When both settings are set to `on`, the plugin must confirm the `BEGIN TRANSACTION` and `COMMIT TRANSACTION` messages, and the log messages within the transaction. The log messages within the transaction can be confirmed with any of the following status codes (which the plugin must write to stdout): * `OK` * `DEFER_COMMIT` * `PREVIOUS_COMMITTED` Refer to http://www.rsyslog.com/doc/v8-stable/development/dev_oplugins.html for an explanation on the meaning of these status codes. You will typically need to return the `DEFER_COMMIT` status code, since the other codes imply a partial commit, and do not guarantee that the `COMMIT TRANSACTION` will be received. The following sequence illustrates the exchanges between rsyslog and the plugin when transactions and message confirmations are enabled, and the plugin confirms the log messages within each transaction with `DEFER_COMMIT`: <= OK => BEGIN TRANSACTION <= OK => log message 1 <= DEFER_COMMIT => log message 2 <= DEFER_COMMIT ... => log message 5 <= DEFER_COMMIT => COMMIT TRANSACTION <= OK => BEGIN TRANSACTION <= OK => log message 6 <= DEFER_COMMIT => log message 7 <= DEFER_COMMIT ... => log message 10 <= DEFER_COMMIT => COMMIT TRANSACTION <= OK Example implementation ---------------------- For a reference example of a plugin with transaction support, see [this Python plugin skeleton](skeletons/python/plugin-with-feedback.py). Threading Model =============== Write your plugin as you would do in a single threaded environment. Rsyslog automatically detects when it is time to spawn additional threads. If it decides so, it will also spawn another instance of your script and feed it concurrently. Note that rsyslog will also terminate instances that it knows are no longer needed. If your plugin for some reason cannot be run in multiple instances, there are ways to tell rsyslog to work with a single instance. But it is strongly suggested to not restrict rsyslog to do that. Multiple instances in almost all cases do NOT mean any burden to the plugin developer. Just think of two (or more) independent instances of your program running in different console windows. If that is no problem, rsyslog running multiple instances of it is also no problem. Future Enhancements =================== Interfaces for external input, filter and message modification plugins are planned. Most probably, they will become available in the order mentioned in the last sentence. External Message Modification Modules ------------------------------------- The external plugin will use stdin to receive the message that it potentially can modify. The message will be LF-terminated, and no LF must be present within the message itself. By default, the MSG part of the message is provided as input. The "interface.input" parameter can be used to modify this. It may has the following values: * "msg" (the default) * "rawmsg", which is the complete message (including header) as received by rsyslog * "json", which is the complete message object (with all properties broken out) as a JSON object string. This is the "jsonmesg" dynamic message property. **Note**: if multi-line messages are to be processed, JSON representation **must** be used, otherwise errors will happen. The ability to use non-JSON representations is primarily a performance enhancement. Building the JSON representation causes some overhead, and very often access to either msg or rawmsg is fully sufficient. The plugin will emit a JSON representation of those properties that **need to be modified** and their new values to stdout. Again, this is delimited by LF, with no LF permitted inside the JSON representation. Only properties that are to be changed must be included in the response. Unchanged properties should NOT be included in the response, as this would increase processing cost. If no property is to be modified, an empty JSON representation is to be provided. The plugin must emit one response line for each message (line) received, and must do so in the same order in which the messages were put in stdin. Note that like the output module interface, multiple instances of the plugin may be activated. See above for more information. Most message properties can be modified. Modifiable are: * rawmsg * msg * syslogtag * syslogfacility * syslogseverity * msgid * procid * structured-data * hostname (aliased "source") * fromhost * fromhost-ip * all message variable ("$!" tree) If the message variable tree is modified, new variables may also be *added*. Deletion of message variables is not directly supported. If this is desired, it is suggested to set the variable in question to the empty string (""). Implementation -------------- The plugin interface is implemented via the "mmexternal" native plugin. See its documentation on how to tie your plugin into rsyslog's procesing flow.