Filebeat Multiline Configuration Changes for Object, StackTrace and XML


Multiline configuration is required if need to handle multilines on filebeat server end. That will help for logs type like stackTrace for exception, print objects, XML, JSON etc. where standard log4j format does’t work  so this type of lines can be combined with previous line where log4j format was applied.

Below are filebeat configuration for multiline.

multiline.pattern: The regexp Pattern that has to be matched. The example pattern matches all lines starting with [DEBUG,ALERT,TRACE,WARNING log level that can be customize according to your logs line format. But that is generic one that will help most of cases.
multiline.pattern: ‘^[([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)’

Default is false for negate. Defines if the pattern match should be negated or not.
multiline.negate: true

multiline.match define if pattern not match with above pattern where these line need to append. Possible values are “after” or “before”.

multiline.match: after

If you will set this max line after these number of multiline all will ignore
multiline.max_lines: 50

For Example :

multiline.pattern: ‘^\[([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)’

multiline.negate: true
multiline.match: after
#multiline.max_lines: 50

Sample Configuration file for multiline configuration.

Sample filebeat.yml file for Prospectors,Multiline and Logging Configuration

Integration

Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Read More

To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial  and  Filebeat Issues. To know more about YAML follow link as YAML Tutorial.

Leave you feedback to enhance more on this topic so that make it more helpful for others.

Advertisements

13 thoughts on “Filebeat Multiline Configuration Changes for Object, StackTrace and XML”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s