Tag Archives: compression

ELK, Filebeat, Kafka

Filebeat, Kafka Output Configuration

13 Comments

If need  to shipped server logs lines  directly to Kafka. Follow below steps:

Pre-Requisite :

  • Start Kafka  before start filebeat to listen publish events and configure filebeat with same kafka server port

Kafka Output  Required Configuration :

  • Comment out output.elasticsearch output section and uncomment output.kafka section
  • Set enabled value is true to make kafka output as enabled
  • Set host  of server where Kafka is running for listening  by default port for Kafka is 9092 if any change use same port value.
output.kafka:
 enabled:true
 #configure topic as per your application need
 hosts:["kafkaserver:9092"]
 topic:QC-TEST

Kafka Credentials Settings: Set below credentials if any for Kafka broker.

 username:"userid"
 password:"password"

Other Optional Configurations:

Kafka Output Compression Configuration:

Default value for compression is gzip. We can also set other compression codec like snappy, gzip or none.

compression:gzip

Logstash Output Performance Configuration:

worker:  we can configure number of worker for each host publishing events to elasticsearch which will do load balancing.

Kafka Broker Topic Partition Configuration:

key: Default no key setting. But we can use formatted key settings.

partition.hash: Default partition strategy is ‘hash’ using key values set. If not set key value will randomly distribute publish events.

reachable_only: Default value  is false. If reach_only enabled event will publish only reachable kafka brokers.

hash: [] Default value is empty list. Configure alternative event field names used to compute the hash value. If empty output.kafka.key setting will be used.

version: Kafka Broker version to configure so that filebeat can check compatibility with that.

Meta Data Configuration: Meta data information is required for broker event publishing so that filebeat can take  #decision based on status of brokers.

metadata:

retry.max: Defaults value for max 3 retries selection of available brokers.

retry.backoff: Default value is 250ms. Will wait for specified time before make next retries.
refresh_frequency: Will update meta data information  in every 10 minutes.

max_retries: Default value is 3. If set less than 0 filebeat will retry continuously as logs as events not publish.

bulk_max_size: The Default value is 2048.It shows max number of batch events will publish to Kafka in one request.

Kafka Reliability Setting:

#Default Value is 1 for ACK for reliability. Possible values can be :

#0=no response , Message can be lost on some error happens

#1=wait for local commit

#-1=wait for all replicas to commit.
required_acks: 1
timeout: The default value is 30 second. It will timeout if not hear any response from Kafka broker with in specified time.
broker_timeout: Default is value is 10 seconds. During this max duration broker will wait for number #of required acknowledgement.
channel_buffer_size: Default value is 256 for buffered message for Kafka broker.
keep_alive: Default value is 0 seconds  as keep alive is disabled and if this value set will keep alive active network connection for that time.
max_message_bytes: Default value is 1000000 bytes . If Json value is more than configured max message bytes event will dropped.

flush_interval: Waiting Interval between new events and previous events for read logs.

client_id: Default value is beat. We can set values for this field that will help for analysis and auditing purpose.

Sample configuration file

Sample filebeat.yml file for Kafka Output Configuration

Integration

Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Read More

To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial  and  Filebeat Issues. To Know more about YAML follow link YAML Tutorials.

Leave you feedback to enhance more on this topic so that make it more helpful for others.

ELK, Filebeat, Logstash

Filebeat, Logstash Output Configuration

12 Comments

If need  to shipped server logs lines  directly to Logstash. Follow below steps:

Pre-Requisite :

  • Create Logstash Configuration file  with input section mentioned same port as configured in filebeat for logstash listener. Default port for logstash is 5044.
  • Start Logstash with same configuration file.

Logstash Output  Required Configuration :

  • Comment out output.elasticsearch output section and uncomment output.logstash section
  • Set enabled value is true to make logstash output as enabled
  • Set host  of server where Logstash is running for listening  by default port for Logstash is 5044 if any change use same port value.
output.logstash:
 enabled:true
#use localhost if on same machine and same port                                                                    useby  logstash listener
 hosts:["logstashserver:5044"]

Other Optional Configurations:

Logstash Output Compression Configuration:

Filebeat provide gzip compression level which varies from 1 to 9. As compression level increase processing speed will reduce but network speed increase.By default compression level disable and value is 0.

compress_level:0

Logstash Output Performance Configuration:

worker:  we can configure number of worker for each host publishing events to elasticseach which will do load balancing.

loadbalance: Default value is false.  If set to true will check status of hosts if unresponsive will send to another available host. if false filebeat will select random host and send events to it.

pipelining: Default value is 0 means pipeline disabled. Configure value decide of pipeline  batches to send to logstash asynchronously and wait for response. If pipeline value is written means output will blocking.

Logstash Output Proxy Configuration: Filebeat use SOCKS5 protocol to communicate with logstash servers. If any proxy configure for this protocol on server end then we can overcome by setting below details.

proxy_url:socks5://userid:pwd@socks5-server:2233

proxy_use_local_resolver: Default value is false means resolve host name resolution on  proxy server. If value is set as true Logstash host name resolution locally for proxy.

Sample configuration file

Sample filebeat.yml file for Logstash Output

Integration

Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Read More

To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial  and  Filebeat Issues. To know more about YML follow link YAML Tutorials.

Leave you feedback to enhance more on this topic so that make it more helpful for others.

Elasticsearch, ELK, Filebeat

Filebeat,Elasticsearch Output Configuration

12 Comments

If we need  to shipped server logs lines  directly to elasticseach  over HTTP by filebeat . We have set below fields for elasticsearch output according to your elasticsearch server configuration and follow below steps.

  1.  Uncomment output.elasticsearch in filebeat.yml file Elasticsearch
  2.  Set host and port in hosts line
  3.  Set index name as you want. If it’s not set filebeat will create default index as “filebeat-%{+yyyy.MM.dd}” .
output.elasticsearch :

   enabled:true
   hosts:["localhost:9200"]
   index:app1-logs-%{+yyyy.MM.dd}"

Elasticsearch server credentials configuration if any 

  1.  Set user name and password
  2.  Set protocol if https because default protocol is http
    username:userid
    password:pwd

Elasticsearch Index Template Configuration: We can update elasticsearch index template from filebeat which will define settings and mappings to determine field analysis.

Auto Index Template Loading: Filebeat package will load default template filebeat.template.json to elasticsearch if no any template configuration for template and will not overwrite template.

Customize Index Template Loading: We can upload our user define template and update version also by using below configuration.

#(if set as false template need to upload manually)
template.enabled:true
#default value is filebeat
template.name:"app1"
#default value is filebeat.template.json.
template.path:"app1.template.json"
#default value is false
template.overwrite:false

By default, template.overwrite value is false and will not overwrite index template if already exist on elasticsearch.  For overwriting index template make this flag as true in filebeat.yml configuraton file.

Latest Template Version Loading from Filebeat: Set template.overwrite as true and if need to update template file version as 2.x then set path of Latest template file with below configuration.


template.overwrite:true
template.versions.2x.enabled: true
template.versions.2x.path:"${path.config}/app1.template-es2x.json"

Manually Index Template Loading : for manually index loading please refer Elasticsearch Index Template Management.

Compress Elasticsearch Output :  Filebeat provide gzip compression level which varies from 1 to 9. As compression level increase processing speed will reduce but network speed increase.By default compression level disable and value is 0.


compress_level: 0

Other configuration Options:

bulk_max_size : Default values is 50. If filebeat is generating events more than configure batch max size it will split events in configure size batches and send to elasticsearch. As much as batch size will increase performance will improve but require more buffring. It can cause other issue like connection, errors, timeout for requests.

Never set value of bulk size as 0 because there would not be any buffering for events and filebeat will send each event directly to elasticsearch.

timeout: Default value is 90 seconds. If no response http request will timeout.

flush_interval: waiting time for new events for bulk requests. If bulk request max size sent before this specified time, new bulk index request created.

max_retries: Default value is 3. When max retry reach specified limit and evens not published all events will drop. Filebeat also provide option to retry until all events are published by setting value as less than 0.

worker:  we can configure number of worker for each host publishing events to elasticseach which will do load balancing.

 Sample Filebeat Configuration file:

Sample filebeat.yml file to Integrate Filebeat with Elasticsearch

Integration

Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Read More

To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial  and  Filebeat Issues. To know more about YAML follow link YAML tutorials.

Leave you feedback to enhance more on this topic so that make it more helpful for others.