mirror of
https://github.com/rsyslog/rsyslog.git
synced 2026-03-19 14:09:30 +01:00
178 lines
11 KiB
HTML
178 lines
11 KiB
HTML
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
|
<html>
|
|
<head>
|
|
<meta content="en" http-equiv="Content-Language" />
|
|
<title>Elasticsearch Output Module</title>
|
|
</head>
|
|
<body>
|
|
<p>
|
|
<a href="rsyslog_conf_modules.html">back</a></p>
|
|
<h1>
|
|
Elasticsearch Output Module</h1>
|
|
<p>
|
|
<b>Module Name: omelasticsearch</b></p>
|
|
<p>
|
|
<b>Author: </b>Rainer Gerhards <rgerhards@adiscon.com></p>
|
|
<p>
|
|
<b>Available since: </b>6.4.0+</p>
|
|
<p>
|
|
<b>Description</b>:</p>
|
|
<p>
|
|
This module provides native support for logging to <a href="http://www.elasticsearch.org/">Elasticsearch</a>.</p>
|
|
<p>
|
|
<b>Action Parameters</b>:</p>
|
|
<ul>
|
|
<li>
|
|
<b>server</b><br />
|
|
Host name or IP address of the Elasticsearch server. Defaults to "localhost"</li>
|
|
<li>
|
|
<b>serverport</b><br />
|
|
HTTP port to connect to Elasticsearch. Defaults to 9200</li>
|
|
<li>
|
|
<b>searchIndex</b><br />
|
|
<a href="http://www.elasticsearch.org/guide/appendix/glossary.html#index">Elasticsearch index</a> to send your logs to. Defaults to "system"</li>
|
|
<li>
|
|
<b>dynSearchIndex </b><on/<b>off</b>><br />
|
|
Whether the string provided for <strong>searchIndex</strong> should be taken as a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a>. Defaults to "off", which means the index name will be taken literally. Otherwise, it will look for a template with that name, and the resulting string will be the index name. For example, let's assume you define a template named "date-days" containing "%timereported:1:10:date-rfc3339%". Then, with dynSearchIndex="on", if you say searchIndex="date-days", each log will be sent to and index named after the first 10 characters of the timestamp, like "2013-03-22".</li>
|
|
<li>
|
|
<b>searchType</b><br />
|
|
<a href="http://www.elasticsearch.org/guide/appendix/glossary.html#type">Elasticsearch type</a> to send your index to. Defaults to "events"</li>
|
|
<li>
|
|
<b>dynSearchType</b> <on/<strong>off</strong>><br />
|
|
Like <strong>dynSearchIndex</strong>, it allows you to specify a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a> for <strong>searchType</strong>, instead of a static string.</li>
|
|
<li>
|
|
<strong>asyncrepl </strong><on/<strong>off</strong>><br />
|
|
By default, an indexing operation returns after all <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#replica_shard">replica shards</a> have indexed the document. With asyncrepl="on" it will return after it was indexed on the <a href="http://www.elasticsearch.org/guide/appendix/glossary.html#primary_shard">primary shard</a> only - thus trading some consistency for speed.</li>
|
|
<li>
|
|
<strong>timeout</strong><br />
|
|
How long Elasticsearch will wait for a primary shard to be available for indexing your log before sending back an error. Defaults to "1m".</li>
|
|
<li>
|
|
<strong>template</strong><br />
|
|
This is the JSON document that will be indexed in Elasticsearch. The resulting string needs to be a valid JSON, otherwise Elasticsearch will return an error. Defaults to:</li>
|
|
</ul>
|
|
<pre>
|
|
$template JSONDefault, "{\"message\":\"%msg:::json%\",\"fromhost\":\"%HOSTNAME:::json%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"timereported\":\"%timereported:::date-rfc3339%\",\"timegenerated\":\"%timegenerated:::date-rfc3339%\"}"
|
|
</pre>
|
|
<p>
|
|
Which will produce this sort of documents (pretty-printed here for readability):</p>
|
|
<ul>
|
|
</ul>
|
|
<pre>
|
|
{
|
|
"message": " this is a test message",
|
|
"fromhost": "test-host",
|
|
"facility": "user",
|
|
"priority": "info",
|
|
"timereported": "2013-03-12T18:05:01.344864+02:00",
|
|
"timegenerated": "2013-03-12T18:05:01.344864+02:00"
|
|
}</pre>
|
|
<ul>
|
|
<li>
|
|
<strong>bulkmode </strong><on/<strong>off</strong>><br />
|
|
The default "off" setting means logs are shipped one by one. Each in its own HTTP request, using the <a href="http://www.elasticsearch.org/guide/reference/api/index_.html">Index API</a>. Set it to "on" and it will use Elasticsearch's <a href="http://www.elasticsearch.org/guide/reference/api/bulk.html">Bulk API</a> to send multiple logs in the same request. The maximum number of logs sent in a single bulk request depends on your queue settings - usually limited by the <a href="http://www.rsyslog.com/doc/node35.html">dequeue batch size</a>. More information about queues can be found <a href="http://www.rsyslog.com/doc/node32.html">here</a>.</li>
|
|
<li>
|
|
<strong>parent</strong><br />
|
|
Specifying a string here will index your logs with that string the parent ID of those logs. Please note that you need to define the <a href="http://www.elasticsearch.org/guide/reference/mapping/parent-field.html">parent field</a> in your <a href="http://www.elasticsearch.org/guide/reference/mapping/">mapping</a> for that to work. By default, logs are indexed without a parent.</li>
|
|
<li>
|
|
<strong>dynParent </strong><on/<strong>off</strong>><br />
|
|
Using the same parent for all the logs sent in the same action is quite unlikely. So you'd probably want to turn this "on" and specify a <a href="http://www.rsyslog.com/doc/rsyslog_conf_templates.html">template</a> that will provide meaningful parent IDs for your logs.</li>
|
|
<li>
|
|
<strong>uid</strong><br />
|
|
If you have basic HTTP authentication deployed (eg: through the <a href="https://github.com/Asquera/elasticsearch-http-basic">elasticsearch-basic plugin</a>), you can specify your user-name here.</li>
|
|
<li>
|
|
<strong>pwd</strong><br />
|
|
Password for basic authentication.</li>
|
|
</ul>
|
|
<p>
|
|
<b>Samples:</b></p>
|
|
<p>
|
|
The following sample does the following:</p>
|
|
<ul>
|
|
<li>
|
|
loads the omelasticsearch module</li>
|
|
<li>
|
|
outputs all logs to Elasticsearch using the default settings</li>
|
|
</ul>
|
|
<pre>
|
|
module(load="omelasticsearch")
|
|
*.* action(type="omelasticsearch")</pre>
|
|
<p>
|
|
The following sample does the following:</p>
|
|
<ul>
|
|
<li>
|
|
loads the omelasticsearch module</li>
|
|
<li>
|
|
defines a template that will make the JSON contain the following properties (more info about what properties you can use <a href="http://www.rsyslog.com/doc/property_replacer.html">here</a>):
|
|
<ul>
|
|
<li>
|
|
RFC-3339 timestamp when the event was generated</li>
|
|
<li>
|
|
the message part of the event</li>
|
|
<li>
|
|
hostname of the system that generated the message</li>
|
|
<li>
|
|
severity of the event, as a string</li>
|
|
<li>
|
|
facility, as a string</li>
|
|
<li>
|
|
the tag of the event</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
outputs to Elasticsearch with the following settings
|
|
<ul>
|
|
<li>
|
|
host name of the server is myserver.local</li>
|
|
<li>
|
|
port is 9200</li>
|
|
<li>
|
|
JSON docs will look as defined in the template above</li>
|
|
<li>
|
|
index will be "test-index"</li>
|
|
<li>
|
|
type will be "test-type"</li>
|
|
<li>
|
|
activate bulk mode. For that to work effectively, we use an in-memory queue that can hold up to 5000 events. The maximum bulk size will be 300</li>
|
|
<li>
|
|
retry indefinitely if the HTTP request failed (eg: if the target server is down)</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
<pre>
|
|
module(load="omelasticsearch")
|
|
template(name="testTemplate"
|
|
type="list"
|
|
option.json="on") {
|
|
constant(value="{")
|
|
constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
|
|
constant(value="\",\"message\":\"") property(name="msg")
|
|
constant(value="\",\"host\":\"") property(name="hostname")
|
|
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
|
|
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
|
|
constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
|
|
constant(value="\"}")
|
|
}
|
|
*.* action(type="omelasticsearch"
|
|
server="myserver.local"
|
|
serverport="9200"
|
|
template="testTemplate"
|
|
searchIndex="test-index"
|
|
searchType="test-type"
|
|
bulkmode="on"
|
|
queue.type="linkedlist"
|
|
queue.size="5000"
|
|
queue.dequeuebatchsize="300"
|
|
action.resumeretrycount="-1")</pre>
|
|
<p>
|
|
</p>
|
|
<pre>
|
|
</pre>
|
|
<p>
|
|
[<a href="rsyslog_conf.html">rsyslog.conf overview</a>] [<a href="manual.html">manual index</a>] [<a href="http://www.rsyslog.com/">rsyslog site</a>]</p>
|
|
<p>
|
|
<font size="2">This documentation is part of the <a href="http://www.rsyslog.com/">rsyslog</a> project.<br />
|
|
Copyright © 2008-2012 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and <a href="http://www.adiscon.com/">Adiscon</a>. Released under the ASL 2.0.</font></p>
|
|
</body>
|
|
</html>
|
|
|