Filebeat Multiline Configuration Changes for Object, StackTrace and XML


Multiline configuration is required if need to handle multilines on filebeat server end. That will help for logs type like stackTrace for exception, print objects, XML, JSON etc. where standard log4j format does’t work  so this type of lines can be combined with previous line where log4j format was applied.

Below are filebeat configuration for multiline.

multiline.pattern: The regexp Pattern that has to be matched. The example pattern matches all lines starting with [DEBUG,ALERT,TRACE,WARNING log level that can be customize according to your logs line format. But that is generic one that will help most of cases.
multiline.pattern: ‘^[([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)’

Default is false for negate. Defines if the pattern match should be negated or not.
multiline.negate: true

multiline.match define if pattern not match with above pattern where these line need to append. Possible values are “after” or “before”.

multiline.match: after

If you will set this max line after these number of multiline all will ignore
multiline.max_lines: 50

For Example :

multiline.pattern: ‘^\[([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)’

multiline.negate: true
multiline.match: after
#multiline.max_lines: 50

Sample Configuration file for multiline configuration.

Sample filebeat.yml file for Prospectors,Multiline and Logging Configuration

Integration

Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Read More

To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial  and  Filebeat Issues.

Leave you feedback to enhance more on this topic so that make it more helpful for others.

 

About Saurabh Gupta

My Name is Saurabh Gupta, I have approx. 10 Year of experience in Information Technology World manly in Java/J2EE. During this time I have worked with multiple organization with different client, so many technology, frameworks etc.
This entry was posted in Filebeat and tagged , , . Bookmark the permalink.