Tag Archives: commandline

Filebeat

Filebeat, Commandline Arguments Configuration

12 Comments

Environment and Commandline arguments setting to filebeat.yml

We can use environment variables and arguments from command line  references in the filebeat.yml file to set values or fields that need to be configurable during deployment. To configure fields and values use like this:

${VAR1}

Where VAR1 is the name of the environment variable or passing values from command line.

These configured values will replaced on start up by the value of environment variable or command line arguments. These configured fields are case sensitive if value not found will fail on parsing so better configure values with default value as given below.

${VAR1:default_value}

We pass command line arguments to set configured values with option -E with argument name.

We have configured in below file variable fields for server, kafkaHost and topic in below sample configuration file that way we can set other fields also as per requirement.

Use below command to run this sample configuration file:

./filebeat -c filebeat.yml -d publish -E server=server1 -E kafkaHost=IP:PORT -E topicName=QC-TEST

For example above commandline arguments will pass to filebeat.yml file for below settings.

Prospectors Settings :

filebeat.prospectors:
- input_type: log
paths:
- /opt/app/${server}/logs/app1-debug*.log*
fields_under_root: true

Kafka Output Settings:

output.kafka:
#enabled: true
hosts: ["${kafkaHost}"]
topic: ${topicName}

Integration

Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Read More

To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial  and  Filebeat Issues.To Know more about YAML follow link YAML Tutorials.

Leave you feedback to enhance more on this topic so that make it more helpful for others.

ELK, Filebeat, Kafka, Logstash

Filebeat Download,Installation and Start/Run

14 Comments

Filebeat Latest Version : 6.2.4

Filebeat Download Link :  https://www.elastic.co/downloads/beats/filebeat

 Download filebeat from above link according to your Operating System and copy to directory where you want to install.

 Installation on Linux :  Go to directory where tar file was copied and use below    command to install it.


tar –zxvf filebeat--linux-x86.tar.gz

Installation on Windows: Go to directory where zip file was copied and unzip  file.


unzip file filebeat--window-xxx.zip file

Before Test and Run filebeat installation need to make below configuration changes in filbeat.yml file for prospectors,Output ,logging etc. Prospectors changes are required rest of changes optional and decide based on application requirements.

Required Change:

 Optional Change : Based on your Application Requirement

 Run/Start Filebeat On Linux:


./filebeat -e -c filebeat.yml -d "publish"

For running filebeat in background add “screen –d –m” as given below:


screen -d -m ./filebeat -e -c filebeat.yml -d "publish"

For Logging filebeat output to log file remove –e option from command as given below and follow link Filebeat Configuration Changes for Logging  for more info.


./filebeat  -c filebeat.yml -d "publish"

screen -d -m ./filebeat  -c filebeat.yml -d "publish"

Filebeat 5 added new features of passing command line arguments while start filebeat. This is really helpful because no change required in filebeat.yml configuration file specifics to servers and and pass server specific information over command line. If in future your servers scaling and changes in output port and machine IP for elasticsearch or kafka or logstash. Then configuration team need to update only command line arguments for specific information and no change in configuration file.

Run/Start Filebeat with command line Arguments in Forground:


./filebeat -c filebeat.yml -d publish -E server= -E file=app1.log  -E tz=CDT -E                     kafkaHost=IP:PORT

Run In Background :


screen –d –m ./filebeat -c filebeat.yml -d publish -E server= -E  file=app1.log  -E tz=CDT -E kafkaHost=IP:PORT

Here -E option represents argument values are passing from command line will set in respective position in filebeat.yml configuration file. Follow link Filebeat Commandline Arguments setting in configuration file .

Integration

Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Read More

To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial  and  Filebeat Issues. To know more about YAML/YML follow YAML Tutorials

Leave you feedback to enhance more on this topic so that make it more helpful for others.