This commit adds the initial implementation of the `mmaitag` module,
a pluggable AI-based message classification processor.
The module supports two providers:
- `gemini`: uses the Gemini API via libcurl and JSON-C
- `gemini_mock`: a deterministic in-memory mock used for testing
Each log message is classified individually and tagged with a label
stored in a configurable message variable (default: `$.aitag`).
Included:
- Provider abstraction layer (`ai_provider.h`)
- Gemini provider with API key support (inline or via file)
- Prompt customization and input property selection
- Two regression tests: basic and invalid-key scenarios
- Sphinx documentation under `doc/configuration/modules`
Note: mmaitag is in its infancy and primarily a PoC. Future work
will improve batching, performance, and add more providers.
Refs:
- https://github.com/rsyslog/rsyslog/issues/5747 (AI-first strategy)
With the help of AI Agents: Codex, Gemini
The omsendertrack module is designed to track and report statistics for
message senders across all inputs in Rsyslog. It periodically outputs a
JSON file containing information about each sender.
Note: This commit provides minimalistic basic functionality as a PoC.
We will check it's usefulness in practice and expect follow-up PRs
to enhance functionality and include feedback from early testing.
However, this module is solid, just feature limited.
see also: https://github.com/rsyslog/rsyslog/issues/5599
- Extracted basic OpenSSL helper functions into own module net_ossl.h/net_ossl.c
Both are compiled into lmnsd_ossl.
- Cleanup of OpenSSL code, fixed minor compiler and linking issues.
- Added DTLS Sender option DTLS into tcpflood for testbench.
- Add initial implementation of imdtls input module. Added to configure and makefile
- Add initial implementation of omdtls output module. Added to configure and makefile
- Add multiple basic tests for imdtls receiving data by using tcpflood.
- Add multiple send-receive test for imdtls and omdtls based on existing tls tests.
- Add timeout and sessionbreak tests for imdtls stress testing.
closes: https://github.com/rsyslog/rsyslog/issues/5211
The output module uses Apache "Qpid Proton C API" which is a solid
AMQP protocol library implementation that can be integrated
very well into the rsyslog dev environment.
- Implemented Delivery with submitted and accepted state checking
- impstatscounter used in testcases
- saving of failed messages in a failed list with support of saving
and restoring.
- Add testcases (requires ENV variables) to testbench
- Using application/octect-stream (binary) to send messages based on
Microsoft Code Sample:
https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-c-getstarted-send
* Note original Microsoft Samplecode is not working anymore, we are using
* QPID Proton Proactor based on
https://github.com/apache/qpid-proton/blob/main/c/examples/send.c
- requires QPID-PROTON Version 0.13 or higher because of the proactor API
- Add EventProperties configuration parameters
- using internal array instead of linkedlist for better performance
- using single byte helper to store message status (ubsubmit/submitted...)
- trigger wake up if messages need to be resubmitted (rejected)
- Slow down when sender credit reaches zero (10ns).
- Add enhanced performance stress test omazureeventhubs-stress.sh
- Add support for static library linking of qpid-proton
This is needed to build the module from source and remove
library package dependencies.
- adjusted valgrind suppressions
This commit adds a new rainerscript function to unflatten keys
in a JSON tree. It provides a way to expand dot separated fields.
<result> = unflatten(<source-tree>, <key-separator-character>);
It allows for instance to produce this:
{ "source": { "ip": "1.2.3.4", "port": 443 } }
from this source data:
{ "source.ip": "1.2.3.4", "source.port": 443 }
Here is a sample use case:
set $!source.ip = "1.2.3.4";
set $!source.bytes = 3258;
set $!source.geo.country_iso_code = "FR";
set $!destination.ip = "4.3.2.1";
set $.unflatten = unflatten($!, ".");
if (script_error() == 0) then {
unset $!;
set $! = $.unflatten;
unset $.unflatten;
}
{
"source": {
"ip": "1.2.3.4",
"bytes": 3258,
"geo": {
"country_iso_code": "FR"
}
},
"destination": {
"ip": "4.3.2.1"
}
}
This patch should address (as a post-processing workaround) the needs
expressed in rsyslog/liblognorm#105: permit specifying a multi-level
name. I believe this is somehow what @rgerhards suggested in a comment
on the issue.
The patch was originally written to address a lack of functionality in
Elasticsearch ingest pipeline processors. Some processors would not
accept dotted fields as input (or output, can't remember exactly, at
least in 7.8.1). The dot expander processor can be used to unflatten
fields but this is not generic (and thus, difficult to maintain) as
you have to apply the processor on a known list of fields.
- uses http library to provide http input.
user would need to configure an 'endpoint' as input, along
with a ruleset, defining how the input should be routed in
rsyslog.
bugfix, free dynamic buf if created, don't echo to client data.
do data framing using newline character.
more input options support:
- input name
- flowcontrol
add support for gzip content support, and other options
- gzip content
- parse linefeeds by default, option to ignore linefeeds
add support for gzip content support, and other options
- gzip content
- parse linefeeds by default, option to ignore linefeeds
WIP - misc edits enable port, documentroot module parameter
fix overloaded stack issue
update tests to use available port instead of hard-coded one.
Add 'octet counted framing' support
- option is "SupportOctetCountedFraming", currently 'off' by
default.
update imhttp-getrequest-file.sh test to use $srcdir
imhttp - support multi-threaded connection contexts.
- tests for large data posts
- Add header data as metadata option
move mg_start into activatecnf instead of in runinput
This file is a systemd unit file. Over the past months, we
received numerous complaints from the RH'ish community because
of the "wrong" content of this file. Thus, we changed it to
silence these complaints. Now, very similar complains come
from the Debian'ish community
(https://github.com/rsyslog/rsyslog/pull/4317#discussion_r445907950).
The root cause of these problems is that this is not really
a ryslog-related file. It is a systemd unit file, and obviously
very highly depending on distro policies. It was an excellent
idea to add this file back in the early days of systemd when
nobody knew about unit files. Nowadays, however, a single
"proposed" unit file causes more trouble than it solves.
As such, we will remove the file in rsyslog's project root.
Instead, we will provide distro-specific sample files in the
./platform subdirectory.
This way each distro can maintain its (considerably different)
rsyslog.service without the rsyslog interfering with it.
closes https://github.com/rsyslog/rsyslog/issues/4333
This was accidentally set to "no". Test for imfile should by
default run when imfile is enabled.
We also temporarily disable the failing tests - this makes
it posible to run the remaining tests during CI and prevent
future errors. That is better than nothing. Tests are to be
re-enabled when the imfile bug mentioned below is fixed.
see also https://github.com/rsyslog/rsyslog/issues/4120
Signed-off-by: Rainer Gerhards <rgerhards@adiscon.com>
Adds a python based SNMP Trap Receiver which outputs received
SNMP Traps by UDP onyl at the moment. Can be extended later.
Tests results are verified by content_check.
The tests have to be enabed with ./configure --enable-snmp-tests
Test sndrcv_omsnmpv1_udp_dynsource.sh checks the new snmpv1dynsource
option.
In order to work, the following python packages needs to be installed:
pip install pysnmp
On Ubuntu 18, we need these packages to be installed:
apt install snmp libsnmp-dev snmp-mibs-downloader
closes: https://github.com/rsyslog/rsyslog/issues/3985
This brings omamqp1 up-to-date with the latest qpid-proton-c
api version. This also adds a test for the plugin, to test
the basic functionality. The test requires the user to
install qdrouterd and the python qpid-proton library in order
to use the simple_recv.py test program.
- plugin will get (docker) container logs from a host as well as filling out some
basic container metadata as id, name, image, labels.
- requirements: curl-7.40.0+, for unix_domain_socket option (Docker API listens on a
local socket). Additionally, plugin will handle the issue of log lines larger than
16KB begin split by Docker.
- include imdocker-unittests in testbench
- enable imdocker tests in osx travis tests
- use curl master branch for imdocker-tests
- fix cflags for pthread
- enforce minimum curl version during configuration - to 7.40.0 for imdocker
- container polls after the initial one are automatically filtered utilizing the 'since' option as per docker api
- imdocker tests refactored.
- add escapeLF config option
Test bench test added:
- imdocker-basic: checks for completeness.
- imdocker-basic-vg: same as basic with valgrind enabled.
- imdocker-long-logline: checks imdocker can handle 16K+ log lines.
- imdocker-long-logline-vg: same as long-logline test with valgrind enabled.
- add option 'retrieveNewLogsFromStart' which will cause imdocker to ignore the 'tail'
option when retrieving container logs for newly activated containers. Containers that
were already active when imdocker first starts still uses the tail option.
- imdocker-new-logs-from-start: checks that new containers will ignore the tail option
- imdocker-new-logs-from-start-vg: valgrind enabled
- multi-line support via docker label: imdocker.startregex
added multi-line tests for testbench
Travis test bench related history and notes:
- Travis tests, enable imdocker
- imdocker travis tests not enabled in precise or trusty ubuntu versions of travis-test.
Assuming travis tests will run in ubuntu 16
- tests: harden container name against strange file system pathes
RSYSLOG_DYNNAME contains the file system path, which is NOT guarnateed
to be suitable for things other than file names.
- tests: bugfix and guard against false negative
- content_check_with count was done before rsyslog was terminated, so
it was uncertain how many messages were written at that spot
(especially on slow machines)
- so far, only the number of lines was checked, not their content. So
any problems with the actual messages would not have been detected.
- improved support for listcontainer options, fix racy imdocker tests
This output module adds the posibility to send
INSERT querys to a Clickhouse database.
The messages are sent via a REST interface.
This commit also adds support of the testbench
for clickhouse tests, as well as various tests.
Closes https://github.com/rsyslog/rsyslog/issues/2272
We run only some distcheck test that we cannot yet run inside
the containers on travis. This reduces the amount of redundant
work done, speeding up Travis runtime.
Any missing checks are still detected by buildbot part of CI.
Also a couple of changes to testbench worth mentioning:
* use cp -f to ensure files can be overwirtten in VBUILD
* fix issue of missing include test file in EXTRA_DIST
* new supressions
* testbench: try to use local system dependency cache
avoid going to Internet repos if not absolutely necessary. For
development containers, they should be pre-populated with the
important dependencies.
* do not enable libfaketime if ASAN is selected
unfortunately, libfaketime does not work in that case
see also https://github.com/rsyslog/rsyslog/issues/174
veeeeery old testing capability, no longer functional but
causes build to fail if enabled. Replaced by ASAN/valgrind.
Issue detected while testing some other CI settings.
function modules add functions to rainerscript dynamically,
change http_request into such a module (enabled by default).
This module can be used as a sample for future function modules.
