Logstash provide some predefined grok pattern for some standard cases like URL , INT, GREEDYDATA, WORD etc. We can customize and define our own grok pattern also.

Why do we need customize Grok Pattern?

If our requirement is define our own grok pattern because need to configure on multiple configuration files for same pattern so that in future any thing change on pattern on log format just need to update on one place only and will reflect on all files.

How to define own Grok Pattern?

• Go to Logstash installation directory and follow below path to edit grok-pattern file.

Logstash-Installation-directory/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-4.0.2/patterns

• Grok-Pattern file define grok  in below form and same way we can define our own grok pattern.

Name regular expression for same

• Consume define Grok Pattern  in your logstash configuration file for grok filter as given in below example.

Example : Suppose our requirement is to parse below log line and retrieve all information like Loglevel, timestamp, ClassName, threadNumber and logContent.

Log statement :

[DEBUG|20161226 134758 956] (ElasticManagerImpl@ExecuteThread: '297' for queue: 'weblogic.kernel.Default') {Using Weblogic-specific timeout values for context request. RequestTimeout: 7200000 RMIClientTimeout: 7200000}

As per our requirement  divide complete log line in sub part with different fields like as below.

logLevel:DEBUG

timestamp: 20161226 134758 956

className: ElasticManagerImpl

threadNumber:297

logContent: Using Weblogic-specific timeout values for context request. RequestTimeout: 7200000 RMIClientTimeout: 7200000

for above parse information grok predefine patterns are there like LOGLEVEL for logs level , INT for thread number , WORD for className and GREEDYDATA for logContent but there is no grok pattern matching for timestamp so we can define our own pattern in grok-pattern file.

LOG_TIMESTAMP %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{SPACE}%{HOUR}%{MINUTE}%{SECOND}%{SPACE}%{INT:milliseconds}

Grok Pattern for Logstash:

In Logstash configuration file will define grok pattern filter as given below.

grok{

match => {"message" => "(?m)^\[%{LOGLEVEL:loglevel}%{SPACE}*\|%{LOG_TIMESTAMP:timestamp \]\]%{SPACE}\(%{GREEDYDATA:className}@%{GREEDYDATA}%{NUMBER:threadNumber}%{GREEDYDATA}\)%{SPACE}\{+?%{GREEDYDATA:logContent\}" }

}

Issues Solution

For more Logstash issues solution follow link Common Logstash Issues.