Filebeat, Kafka, Logstash, Elasticsearch and Kibana Integration is used for big organizations where applications deployed in production on hundreds/thousands of servers and scattered around different locations and need to do analysis on data from these servers on real time.
This integration helps mostly for log level analysis , tracking issues, anomalies with data and alerts on events of particular occurrence and where accountability measures.
By using these technology provide scalable architecture to enhance systems and decoupled of each other individually.
Why these Technology?
Filebeat :
- Lightweight agent for shipping logs.
- Forward and centralize files and logs.
- Robust (Not miss a single beat)
- Open source distributed, Steam Processing, Message Broker platform.
- process stream data or transaction logs on real time.
- fault-tolerant, high throughput, low latency platform for dealing real time data feeds.
- Open source, server-side data processing pipeline that accept data from a different sources simultaneously.
- Parse, Format, Transform data and send to different output sources.
- Elasticsearch is open source, distributed cross-platform.
- Built on top of Lucene which provide full text search and provide NRT(Near real Time) search results.
- Support RESTFUL search by Elasticsearch REST
Kibana:
- Open source
- Provide window to view Elasticsearch data in form different charts and dashboard.
- Provide way searches and operation of data easily with respect to time interval.
- Easily Imported by any web application by embedded dashboards.
How Data flow works ?
In this integration filebeat will install in all servers where your application is deployed and filebeat will read and ship latest logs changes from these servers to Kafka topic as configured for this application.
Logstash will subscribe log lines from kafka topic and perform parsing on these lines make relevant changes, formatting, exclude and include fields then send this processed data to Elasticsearch Indexes as centralize location from different servers.
Kibana is linked with Elasticsearch indexes which will help to do analysis by search, charts and dashboards .
Design Architecture
In below configured architecture considering my application is deployed on three servers and each server having current log file name as App1.log . Our goal is read real time data from these servers and do analysis on these data.
Steps to Installation, Configuration and Start
Here first we will install Kafka and Elasticsearch run individually rest of tools will install and run sequence to test with data flow. Initially install all in same machine and test with sample data with below steps and at end of this post will tell about what changes need to make according to your servers.
- Kafka Installation, Configuration and Start
- Elasticsearch Installation,Configuration and Start
- Filebeat Installation,Configuration and Start
- Logstash Installation,Configuration and Start
- Kibana Installation,Start and display.
Pre-Requisite
These Filebeat,Logstash, Elasticsearch and Kibana versions should be compatible better use latest from https://www.elastic.co/downloads.
- Java 8+
- Linux Server
- Filebeat 5.XX
- Kafka 2.11.XX
- Logstash 5.XX
- Elasticsearch 5.XX
- Kibana 5.XX
Note : Make sure JDK 8 should be install and JAVA_HOME environment variable point to JDK 8 home directory wherever you want in install Elasticsearch, Logstash,Kibana and Kafka.
Window : My computer ->right click-> Properties -> Advance System Settings->System Variable
Linux : Go to your home directory/ sudo directory and below line as below .
export JAVA_HOME=/opt/app/facingissuesonit/jdk1.8.0_66
Sample Data
For testing we will use these sample log line which is having debug as well as stacktrace of logs and grok parsing of this example is designed according to it. For real time testing and actual data you can point to your server log files but you have to modify grok pattern in Logstash configuration accordingly.
2013-02-28 09:57:56,662 WARN CreateSomethingActivationKey - WhateverException for User 49-123-345678 {{rid,US8cFAp5eZgAABwUItEAAAAI_dev01_443}{realsid,60A9772A136B9912B6FF0C3627A47090.dev1-a}}
2013-02-28 09:57:56,663 INFO LMLogger - ERR1700 - u:null failures: 0 - Technical error {{rid,US8cFAp5eZgAABwUItEAAAAI_dev01_443}{realsid,60A9772A136B9912B6FF0C3627A47090.dev1-a}}
2013-02-28 09:57:56,668 ERROR SomeCallLogger - ESS10005 Cpc portalservices: Exception caught while writing log messege to MEA Call: {}
java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
2013-02-28 10:04:35,723 INFO EntryFilter - Fresh on request /portalservices/foobarwhatever {{rid,US8dogp5eZgAABwXPGEAAAAL_dev01_443}{realsid,56BA2AD41D9BB28AFCEEEFF927EE61C2.dev1-a}}
Create App1.log file in same machine where filebeat need to install and copy above logs lines in App1.log file.
Kafka Installation , Configuration and Start
Download latest version of Kafka from below link and use command to untar and installation in Linux server or if window just unzip downloaded file.
Download Link : https://kafka.apache.org/downloads
tar -zxvf kafka_2.11-0.10.0.0
For more configuration and start options follow Setup Kafka Cluster for Single Server/Broker
After download and untar/unzip file it will have below files and directory structure.
ls- l drwxr-xr-x 3 facingissuesonit Saurabh 4096 Apr 3 05:18 bin drwxr-xr-x 2 facingissuesonit Saurabh 4096 May 8 11:05 config drwxr-xr-x 74 facingissuesonit Saurabh 4096 May 27 20:00 kafka-logs drwxr-xr-x 2 facingissuesonit Saurabh 4096 Apr 3 05:17 libs -rw-r--r-- 1 facingissuesonit Saurabh 28824 Apr 3 05:17 LICENSE drwxr-xr-x 2 facingissuesonit Saurabh 487424 May 27 20:00 logs -rw-r--r-- 1 facingissuesonit Saurabh 336 Apr 3 05:18 NOTICE drwxr-xr-x 2 facingissuesonit Saurabh 4096 Apr 3 05:17 site-docs
For more details about all these files,configuration option and other integration options follow Kafka Tutorial.
Make below changes in files config/zookeeper.properties and config/server.properties
config/zookeeper.properties clientPort=2181
config/server.properties: broker.id=0 listeners=PLAINTEXT://:9092 log.dir=/kafka-logs zookeeper.connect=localhost:2181
Now Kafka is configured and ready to run. Use below command to start zookeeper and Kafka server as background process.
screen -d -m bin/zookeeper-server-start.sh config/zookeeper.properties screen -d -m bin/kafka-server-start.sh config/server.properties
To test Kafka install successfully you can check by running Kafka process on Linux “ps -ef|grep kafka” or steps for consumer and producer to/from topic in Setup Kafka Cluster for Single Server/Broker.
Elasticsearch Installation,Configuration and Start
Download latest version of Elasticsearch from below link and use command to untar and installation in Linux server or if window just unzip downloaded file.
Download Link : https://www.elastic.co/downloads/elasticsearch
tar -zxvf elasticsearch-5.4.0.tar.gz
It will show below files and directory structure for Elasticsearch.
drwxr-xr-x 2 facingissuesonit Saurabh 4096 Apr 25 19:20 bin drwxr-xr-x 3 facingissuesonit Saurabh 4096 May 13 17:27 config drwxr-xr-x 3 facingissuesonit Saurabh 4096 Apr 24 15:56 data drwxr-xr-x 2 facingissuesonit Saurabh 4096 Apr 17 10:55 lib -rw-r--r-- 1 facingissuesonit Saurabh 11358 Apr 17 10:50 LICENSE.txt drwxr-xr-x 2 facingissuesonit Saurabh 4096 May 28 05:00 logs drwxr-xr-x 12 facingissuesonit Saurabh 4096 Apr 17 10:55 modules -rw-r--r-- 1 facingissuesonit Saurabh 194187 Apr 17 10:55 NOTICE.txt drwxr-xr-x 2 facingissuesonit Saurabh 4096 Apr 17 10:55 plugins -rw-r--r-- 1 facingissuesonit Saurabh 9540 Apr 17 10:50 README.textile
Before going to start Elasticsearch need to make some basic changes in config/elasticsearch.yml file for cluster and node name. You can configure it based on you application or organization name.
cluster.name: FACING-ISSUE-IN-IT node.name: TEST-NODE-1 #network.host: 0.0.0.0 http.port: 9200
Now we are ready with elasticsearch configuration and time start elasticsearch. We can use below command to run elasticsearch in background.
screen -d -m /bin/elasticsearch
For checking elasticsearch starts successfully you can use below url on browser to know cluster status . You will get result like below.
http://localhost:9200/_cluster/health?pretty or as below if network.host configured http://elasticseverIp:9200/_cluster/health?pretty
Result :
{
"cluster_name" : "FACING-ISSUE-IN-IT",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Filebeat Installation, Configuration and Start
Download latest version of filebeat from below link and use command to untar and installation in Linux server. or if window just unzip downloaded file.
Download Link : https://www.elastic.co/downloads/beats/filebeat
tar -zxvf filebeat-<version>.tar.gz
For more configuration and start options follow Filebeat Download,Installation and Start/Run
After download and untar/unzip file it will have below files and directory structure.
ls- l -rwxr-xr-x 1 facingissuesonit Saurabh 14908742 Jan 11 14:11 filebeat -rw-r--r-- 1 facingissuesonit Saurabh 31964 Jan 11 14:11 filebeat.full.yml -rw-r--r-- 1 facingissuesonit Saurabh 3040 Jan 11 14:11 filebeat.template-es2x.json -rw-r--r-- 1 facingissuesonit Saurabh 2397 Jan 11 14:11 filebeat.template.json -rw-r--r-- 1 facingissuesonit Saurabh 4196 Jan 11 14:11 filebeat.yml -rw-r--r-- 1 facingissuesonit Saurabh 811 Jan 11 14:10 README.md drwxr-xr-x 2 facingissuesonit Saurabh 4096 Jan 11 14:11 scripts
For more details about all these files,configuration option and other integration options follow Filebeat Tutorial.
Now filebeat is installaed and need to make below changes in filebeat.full.yml file
- Inside prospectors section change paths to your log file location as
paths: -/opt/app/facingissuesonit/App1.log
- Comment out Elasticsearch Output default properties as below
#output.elasticsearch: #hosts: ["localhost:9200"]
- Configure multiline option as below so that all stacktrace line which are not starting with date can we consider as single line.
multiline.pattern: ^\d multiline.negate: true multiline.match: after
For learn more on filebeat multiline configuration follow Filebeat Multiline Configuration Changes for Object, StackTrace and XML
- Inside Kafka Output section update these properties hosts and topic. if Kafka on same machine then use localhost else update with IP of kafka machine.
output.kafka: hosts: ["localhost:9092"] topic: APP-1-TOPIC
For more on Logging configuration follow link Filebeat, Logging Configuration.
Now filebeat is configured and ready to start with below command, it will read from configured prospector for file App1.log continiously and publish log line events to Kafka . It will also create topic as APP-1-TOPIC in Kafka if not exist.
./filebeat -e -c filebeat.full.yml -d "publish"
On console it will display output as below for sample lines.
2017/05/28 00:24:27.991828 client.go:184: DBG Publish: {
"@timestamp": "2017-05-28T00:24:22.991Z",
"beat": {
"hostname": "sg02870",
"name": "sg02870",
"version": "5.1.2"
},
"input_type": "log",
"message": "2013-02-28 09:57:56,662 WARN CreateSomethingActivationKey - WhateverException for User 49-123-345678 {{rid,US8cFAp5eZgAABwUItEAAAAI_dev01_443}{realsid,60A9772A136B9912B6FF0C3627A47090.dev1-a}}",
"offset": 194,
"source": "/opt/app/facingissuesonit/App1.log",
"type": "log"
}
2017/05/28 00:24:27.991907 client.go:184: DBG Publish: {
"@timestamp": "2017-05-28T00:24:22.991Z",
"beat": {
"hostname": "sg02870",
"name": "sg02870",
"version": "5.1.2"
},
"input_type": "log",
"message": "2013-02-28 09:57:56,663 INFO LMLogger - ERR1700 - u:null failures: 0 - Technical error {{rid,US8cFAp5eZgAABwUItEAAAAI_dev01_443}{realsid,60A9772A136B9912B6FF0C3627A47090.dev1-a}}",
"offset": 375,
"source": "/opt/app/facingissuesonit/App1.log",
"type": "log"
}
2017/05/28 00:24:27.991984 client.go:184: DBG Publish: {
"@timestamp": "2017-05-28T00:24:22.991Z",
"beat": {
"hostname": "sg02870",
"name": "sg02870",
"version": "5.1.2"
},
"input_type": "log",
"message": "2013-02-28 09:57:56,668 ERROR SomeCallLogger - ESS10005 Cpc portalservices: Exception caught while writing log messege to MEA Call: {}\njava.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist\n\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)",
"offset": 718,
"source": "/opt/app/facingissuesonit/App1.log",
"type": "log"
}
2017/05/28 00:24:27.991984 client.go:184: DBG Publish: {
"@timestamp": "2017-05-28T00:24:22.992Z",
"beat": {
"hostname": "sg02870",
"name": "sg02870",
"version": "5.1.2"
},
"input_type": "log",
"message": "2013-02-28 10:04:35,723 INFO EntryFilter - Fresh on request /portalservices/foobarwhatever {{rid,US8dogp5eZgAABwXPGEAAAAL_dev01_443}{realsid,56BA2AD41D9BB28AFCEEEFF927EE61C2.dev1-a}}",
"offset": 902,
"source": "/opt/app/facingissuesonit/App1.log",
"type": "log"
}
Now you can see from above filebeat debug statements publish event 3 is having multiline statements with stacktrace exception and each debug will have these fields like.
@timestamp: Timestamp of data shipped.
beat.hostname : filebeat machine name from where data is shipping.
beat.version: which version of filebeat installed on server that help for compatibility check on target end.
message : Log line from logs file or multline log lines
offset: it’s represent inode value in source file
source : it’s file name from where logs were read
Now time to check data is publish to Kafka topic or not. For this go to below directory and you will see two files as xyz.index and xyz.log for maintaining data offset and messages.
{Kafka_home}/kafk_logs/APP-1-TOPIC
00000000000000000000.log
00000000000000000000.index
Now your server log lines are in Kafka topic for reading and parsing by Logstash and send it to elasticsearch for doing analysis/search on this data.
Logstash Installation, Configuration and Start
Download latest version of Logstash from below link and use command to untar and installation in Linux server or if window just unzip downloaded file.
Download Link : https://www.elastic.co/downloads/logstash
tar -zxvf logstash-5.4.0.tar.gz
It will show below file and directory structure.
drwxr-xr-x 2 facingissuesonit Saurabh 4096 Apr 20 11:27 bin -rw-r--r-- 1 facingissuesonit Saurabh 111569 Mar 22 23:49 CHANGELOG.md drwxr-xr-x 2 facingissuesonit Saurabh 4096 Apr 20 11:27 config -rw-r--r-- 1 facingissuesonit Saurabh 2249 Mar 22 23:49 CONTRIBUTORS drwxr-xr-x 3 facingissuesonit Saurabh 4096 Apr 20 12:07 data -rw-r--r-- 1 facingissuesonit Saurabh 3945 Mar 22 23:55 Gemfile -rw-r--r-- 1 facingissuesonit Saurabh 21544 Mar 22 23:49 Gemfile.jruby-1.9.lock drwxr-xr-x 5 facingissuesonit Saurabh 4096 Apr 20 11:27 lib -rw-r--r-- 1 facingissuesonit Saurabh 589 Mar 22 23:49 LICENSE drwxr-xr-x 2 facingissuesonit Saurabh 4096 May 21 00:00 logs drwxr-xr-x 4 facingissuesonit Saurabh 4096 Apr 20 11:27 logstash-core drwxr-xr-x 3 facingissuesonit Saurabh 4096 Apr 20 11:27 logstash-core-event-java drwxr-xr-x 3 facingissuesonit Saurabh 4096 Apr 20 11:27 logstash-core-plugin-api drwxr-xr-x 3 facingissuesonit Saurabh 4096 Apr 20 11:27 logstash-core-queue-jruby -rw-r--r-- 1 facingissuesonit Saurabh 28114 Mar 22 23:56 NOTICE.TXT drwxr-xr-x 4 facingissuesonit Saurabh 4096 Apr 20 11:27 vendor
Before going to start Logstash need to create configuration file for taking input data from Kafka and parse these data in respected fields and send it elasticsearch. Create file logstash-app1.conf in logstash bin directory with below content.
/bin/logstash-app1.conf
input {
kafka {
bootstrap_servers => 'localhost:9092'
topics => ["APP-1-TOPIC"]
codec => json {}
}
}
filter
{
//parse log line
grok
{
match => {"message" => "\A%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:loglevel}\s+(?<logger>(?:[a-zA-Z0-9-]+\.)*[A-Za-z0-9$]+)\s+(-\s+)?(?=(?<msgnr>[A-Z]+[0-9]{4,5}))*%{DATA:message}({({[^}]+},?\s*)*})?\s*$(?<stacktrace>(?m:.*))?" }
}
#Remove unused fields
#mutate { remove_field =>["beat","@version" ]}
}
output {
#Output result sent to elasticsearch and dynamically create array
elasticsearch {
index => "app1-logs-%{+YYYY.MM.dd}"
hosts => ["localhost:9200"]
sniffing => false
}
#Sysout logs
stdout
{
codec => rubydebug
}
}
To test your configuration file you can use below command.
./logstash -t -f logstash-app1.conf
If we get result OK from above command run below to start reading and parsing data from Kafka topic.
./logstash -f logstash-app1.conf
For design your own grok pattern for you logs line formatting you can follow below link that will help to generate incrementally and also provide some sample logs grok.
http://grokdebug.herokuapp.com and http://grokconstructor.appspot.com/
Logstash console will show parse data as below and you can remove unsed fields for storing in elasticsearch by uncomment mutate section from configuration file.
{
"@timestamp" => 2017-05-28T23:47:42.160Z,
"offset" => 194,
"loglevel" => "WARN",
"logger" => "CreateSomethingActivationKey",
"beat" => {
"hostname" => "zlp0287k",
"name" => "zlp0287k",
"version" => "5.1.2"
},
"input_type" => "log",
"@version" => "1",
"source" => "/opt/app/facingissuesonit/App1.log",
"message" => [
[0] "2013-02-28 09:57:56,662 WARN CreateSomethingActivationKey - WhateverException for User 49-123-345678 {{rid,US8cFAp5eZgAABwUItEAAAAI_dev01_443}{realsid,60A9772A136B9912B6FF0C3627A47090.dev1-a}}",
[1] "WhateverException for User 49-123-345678 "
],
"type" => "log",
"timestamp" => "2013-02-28 09:57:56,662"
}
{
"msgnr" => "ERR1700",
"@timestamp" => 2017-05-28T23:47:42.160Z,
"offset" => 375,
"loglevel" => "INFO",
"logger" => "LMLogger",
"beat" => {
"hostname" => "zlp0287k",
"name" => "zlp0287k",
"version" => "5.1.2"
},
"input_type" => "log",
"@version" => "1",
"source" => "/opt/app/facingissuesonit/App1.log",
"message" => [
[0] "2013-02-28 09:57:56,663 INFO LMLogger - ERR1700 - u:null failures: 0 - Technical error {{rid,US8cFAp5eZgAABwUItEAAAAI_dev01_443}{realsid,60A9772A136B9912B6FF0C3627A47090.dev1-a}}",
[1] "ERR1700 - u:null failures: 0 - Technical error "
],
"type" => "log",
"timestamp" => "2013-02-28 09:57:56,663"
}
{
"offset" => 718,
"logger" => "SomeCallLogger",
"input_type" => "log",
"message" => [
[0] "2013-02-28 09:57:56,668 ERROR SomeCallLogger - ESS10005 Cpc portalservices: Exception caught while writing log messege to MEA Call: {}\njava.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist\n\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)",
[1] "ESS10005 Cpc portalservices: Exception caught while writing log messege to MEA Call: "
],
"type" => "log",
"msgnr" => "ESS10005",
"@timestamp" => 2017-05-28T23:47:42.160Z,
"stacktrace" => "\njava.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist\n\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:445)\n\tat oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)",
"loglevel" => "ERROR",
"beat" => {
"hostname" => "zlp0287k",
"name" => "zlp0287k",
"version" => "5.1.2"
},
"@version" => "1",
"timestamp" => "2013-02-28 09:57:56,668"
}
{
"@timestamp" => 2017-05-28T23:47:42.160Z,
"offset" => 903,
"loglevel" => "INFO",
"logger" => "EntryFilter",
"beat" => {
"hostname" => "zlp0287k",
"name" => "zlp0287k",
"version" => "5.1.2"
},
"input_type" => "log",
"@version" => "1",
"message" => [
[0] "2013-02-28 10:04:35,723 INFO EntryFilter - Fresh on request /portalservices/foobarwhatever {{rid,US8dogp5eZgAABwXPGEAAAAL_dev01_443}{realsid,56BA2AD41D9BB28AFCEEEFF927EE61C2.dev1-a}}\n",
[1] "Fresh on request /portalservices/foobarwhatever "
],
"type" => "log",
"timestamp" => "2013-02-28 10:04:35,723"
}
To test on elasticsearch end your data sent successfully you can use this url
http://localhost:9200/_cat/indices on your browser and will display created index with current date.
yellow open app1-logs-2017.05.28 Qjs6XWiFQw2zsiVs9Ks6sw 5 1 4 0 47.3kb 47.3kb
Kibana Installation, Configuration and Start
Download latest version of Kibana from below link and use command to untar and installation in Linux server or if window just unzip downloaded file.
Download Link : https://www.elastic.co/downloads/kibana
tar -zxvf kibana-5.4.0.tar.gz
It will show below files and directory structure for kibana.
ls -l drwxr-xr-x 2 facingissuesonit Saurabh 4096 May 22 14:23 bin drwxr-xr-x 2 facingissuesonit Saurabh 4096 Apr 25 18:58 config drwxr-xr-x 2 facingissuesonit Saurabh 4096 Apr 25 11:54 data -rw-r--r-- 1 facingissuesonit Saurabh 562 Apr 17 12:04 LICENSE.txt drwxr-xr-x 6 facingissuesonit Saurabh 4096 Apr 17 12:04 node drwxr-xr-x 485 facingissuesonit Saurabh 20480 Apr 17 12:04 node_modules -rw-r--r-- 1 facingissuesonit Saurabh 660429 Apr 17 12:04 NOTICE.txt drwxr-xr-x 3 facingissuesonit Saurabh 4096 Apr 17 12:04 optimize -rw-r--r-- 1 facingissuesonit Saurabh 702 Apr 17 12:04 package.json drwxr-xr-x 2 facingissuesonit Saurabh 4096 May 22 12:29 plugins -rw-r--r-- 1 facingissuesonit Saurabh 4909 Apr 17 12:04 README.txt drwxr-xr-x 10 facingissuesonit Saurabh 4096 Apr 17 12:04 src drwxr-xr-x 3 facingissuesonit Saurabh 4096 Apr 17 12:04 ui_framework drwxr-xr-x 2 facingissuesonit Saurabh 4096 Apr 17 12:04 webpackShims
Before going to start Kibana need to make some basic changes in config/kibana.yml file make below changes after uncomment these properties file.
server.port: 5601 server.host: localhost elasticsearch.url: "http://localhost:9200"
Now we are ready with Kibana configuration and time start Kibana. We can use below command to run Kibana in background.
screen -d -m /bin/kibana
Kibana take time to start and we can test it by using below url in browser
http://localhost:5601/
For checking this data in Kibana open above url in browser go to management tab on left side menu -> Index Pattern -> Click on Add New
Enter Index name or pattern and time field name as in below screen and click on create button.
Now go to Discover Tab and select index as app1-log* will display data as below.
Now make below changes according to your application specification .
Filebeat :
- update prospector path to your log directory current file
- Move Kafka on different machine because Kafka will single location where receive shipped data from different servers. Update localhost with same IP of kafka server in Kafka output section of filebeat.full.yml file for hosts properties.
- Copy same filebeat setup on all servers from where you application deployed and need to read logs.
- Start all filebeat instances on each Server.
Elasticsearch :
- Uncomment network.host properties from elasticsearch.yml file for accessing by IP address.
Logstash:
- Update localhost in logstash-app1.conf file input section with Kafka machine IP.
- change grok pattern in filter section according to your logs format. You can take help from below url for incrementally design. http://grokdebug.herokuapp.com and http://grokconstructor.appspot.com/
- Update localhost output section for elasticsearch with IP if moving on different machine.
Kibana:
- update localhost in kibana.yml file for elasticsearch.url properties with IP if kibana on different machine.
Conclusion :
In this tutorial considers below points :
- Installation of Filebeat, Kafka, Logstash, Elasticsearch and Kibana.
- Filebeat is configured to shipped logs to Kafka Message Broker.
- Logstash configured to read logs line from Kafka topic , Parse and shipped to Elasticsearch.
- Kibana show these Elasticsearch information in form of chart and dashboard to users for doing analysis.
Read More
To read more on Filebeat, Kafka, Elasticsearch configurations follow the links and Logstash Configuration,Input Plugins, Filter Plugins, Output Plugins, Logstash Customization and related issues follow Logstash Tutorial and Logstash Issues.
Hope this blog was helpful for you.
Leave you feedback to enhance more on this topic so that make it more helpful for others.
Reference :
https://www.elastic.co/products
You must be logged in to post a comment.