Tag Archives: Start in background

Logstash Installation, Configuration and Start


Download latest version of Logstash from below link and use command to unTar and installation in Linux server or if window just unzip downloaded file.

Download Link : https://www.elastic.co/downloads/logstash

tar -zxvf logstash-5.4.0.tar.gz

It will show below file and directory structure.

drwxr-xr-x 2 facingissuesonit Saurabh   4096 Apr 20 11:27 bin
-rw-r--r-- 1 facingissuesonit Saurabh 111569 Mar 22 23:49 CHANGELOG.md
drwxr-xr-x 2 facingissuesonit Saurabh   4096 Apr 20 11:27 config
-rw-r--r-- 1 facingissuesonit Saurabh   2249 Mar 22 23:49 CONTRIBUTORS
drwxr-xr-x 3 facingissuesonit Saurabh   4096 Apr 20 12:07 data
-rw-r--r-- 1 facingissuesonit Saurabh   3945 Mar 22 23:55 Gemfile
-rw-r--r-- 1 facingissuesonit Saurabh  21544 Mar 22 23:49 Gemfile.jruby-1.9.lock
drwxr-xr-x 5 facingissuesonit Saurabh   4096 Apr 20 11:27 lib
-rw-r--r-- 1 facingissuesonit Saurabh    589 Mar 22 23:49 LICENSE
drwxr-xr-x 2 facingissuesonit Saurabh   4096 May 21 00:00 logs
drwxr-xr-x 4 facingissuesonit Saurabh   4096 Apr 20 11:27 logstash-core
drwxr-xr-x 3 facingissuesonit Saurabh   4096 Apr 20 11:27 logstash-core-event-java
drwxr-xr-x 3 facingissuesonit Saurabh   4096 Apr 20 11:27 logstash-core-plugin-api
drwxr-xr-x 3 facingissuesonit Saurabh   4096 Apr 20 11:27 logstash-core-queue-jruby
-rw-r--r-- 1 facingissuesonit Saurabh  28114 Mar 22 23:56 NOTICE.TXT
drwxr-xr-x 4 facingissuesonit Saurabh   4096 Apr 20 11:27 vendor

Before going to start Logstash need to create configuration file for taking input from different input sources like file, csv, jdbc, json, kafka, filebeat etc.  and parse these data in respected fields and send it to output like elasticsearch, file, kafka etc.

Logstash Configuration file will follow below syntax  as i have created file logstash-app1.conf in logstash bin directory . Please follow Logstash Tutorial for more Input, Filter and Output plugin Examples.

/bin/logstash-app1.conf

input {
     kafka {
            ....
          }
    jdbc {
            ....
          }
}
filter
{
//parse log line or data...
      grok
	{
	....
	}
}
output {
    #Output result sent to elasticsearch
    elasticsearch {
       ....
  	}
     #Sysout to console
     stdout
       {
         codec => rubydebug
       }
}

To test your configuration file you can use below command.


./logstash -t -f logstash-app1.conf

If  we get result OK means no any syntax, compile time issue with configuation file from above command.Now run below to start reading and parsing data from different sources.


./logstash -f logstash-app1.conf

To run logstash in background follow command as so that when close your console your Logstash process will keep running.


screen -d -m ./logstash -f logstash-app1.conf

Summary

In above detail cover about below points:

  • How to Install Logstash on Linux Environment.
  • Configuration file Syntax and validation.
  • Start Logstash for configuration file.
  • Start Logstash on background for configuration file.

Read More

To read more on Logstash Configuration,Input Plugins, Filter Plugins, Output Plugins, Logstash Customization and related issues follow Logstash Tutorial and Logstash Issues.

Hope this blog was helpful for you.

Leave you feedback to enhance more on this topic so that make it more helpful for others.

Advertisements

Elasticsearch Installation in Window


For  Elasticsearch installation in window download .zip file from below  link.

Latest Version : elasticsearch-5.4.0

Download Link : https://www.elastic.co/downloads/elasticsearch

Pre-Requisite : Java 8 +

How to Install Elasticsearch?

Unzip downloaded .zip file in directory where you want to install elasticsearch and your unzip directory structure world look as below.

Elasticsearch installation directory window

Below is description about all these directories.

Directory Description
 bin  Binary Script directory which keeps files to start elasticseach, install plugin
 config  Keeps elasticsearch.yml file elasticseach configuration and jvm.options to make jvm related setting like heap etc.
 data This is default directory set in elasticsearch for keeping node data that can be configure by changing path.data in elasticsearch.yml file.
 lib  Keeps all jar files for elasticsearch
 logs  This is default directory set in elasticsearch for keeping logs  that can be configure by changing path.logs in elasticsearch.yml file
 modules  It keeps all functionality and processors required for data.
 plugins  All installed plugin will store in plugins directory

Elasticsearch Configuration

Before going to start Elasticsearch need to make some basic changes in config/elasticsearch.yml file.

cluster.name: FACING-ISSUE-IN-IT
node.name: TEST-NODE-1
#network.host: 0.0.0.0
http.port: 9200

cluster.name : Elasticsearch Cluster name is unique name with in network which links all nodes.

node.name :  Each node in cluster have unique name by which Cluster identify nodes.

http.port: Default http port for Elasticsearch is 9200 . You can update it .

network.host: This property will  set when elasticsearch need to  access other machine  or by IP. For more on network.host follow link Why network.host?

Now Elasticsearch is ready for start.

How to start Elasticsearch?

Click on elasticsearch.bat file inside /bin directory and you will get below console sysout when successfully started.

elasticsearch start

Now you can see from above screen your Elasticsearch is started successfully on port 9200.

For testing elasticsearch cluster status copy below link try on your browser address bar. You will get result like below.

http://localhost:9200/_cluster/health?pretty

or as below if network.host configured

http://elasticseverIp:9200/_cluster/health?pretty

Result :

{
  "cluster_name" : "FACING-ISSUE-IN-IT",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

For learn more about Cluster follow link Cluster Configuration and Health

Same way you can get detail about your configured Node from below URLs.

http://localhost:9200/_nodes?pretty

or as below if network.host configured

http://elasticsearverip:9200/_nodes?pretty

For learn more about Node follow link Node Type,Configuration and status etc. Go to below link for Elasticsearch Tutorials.

Read More

To read more on Elasticsearch Configuration, sample Elasticsearch REST clients, Queries Type configurations with example follow link Elasticsearch Tutorial and Elasticsearch Issues.

Hope this blog was helpful for you.

Leave you feedback to enhance more on this topic so that make it more helpful for others.