If need to shipped server logs lines directly to Logstash. Follow below steps:
- Create Logstash Configuration file with input section mentioned same port as configured in filebeat for logstash listener. Default port for logstash is 5044.
- Start Logstash with same configuration file.
Logstash Output Required Configuration :
- Comment out output.elasticsearch output section and uncomment output.logstash section
- Set enabled value is true to make logstash output as enabled
- Set host of server where Logstash is running for listening by default port for Logstash is 5044 if any change use same port value.
output.logstash: enabled:true #use localhost if on same machine and same port useby logstash listener hosts:["logstashserver:5044"]
Other Optional Configurations:
Logstash Output Compression Configuration:
Filebeat provide gzip compression level which varies from 1 to 9. As compression level increase processing speed will reduce but network speed increase.By default compression level disable and value is 0.
Logstash Output Performance Configuration:
worker: we can configure number of worker for each host publishing events to elasticseach which will do load balancing.
loadbalance: Default value is false. If set to true will check status of hosts if unresponsive will send to another available host. if false filebeat will select random host and send events to it.
pipelining: Default value is 0 means pipeline disabled. Configure value decide of pipeline batches to send to logstash asynchronously and wait for response. If pipeline value is written means output will blocking.
Logstash Output Proxy Configuration: Filebeat use SOCKS5 protocol to communicate with logstash servers. If any proxy configure for this protocol on server end then we can overcome by setting below details.
proxy_use_local_resolver: Default value is false means resolve host name resolution on proxy server. If value is set as true Logstash host name resolution locally for proxy.
Sample configuration file
Sample filebeat.yml file for Logstash Output
Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana
To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial and Filebeat Issues. To know more about YML follow link YAML Tutorials.
Leave you feedback to enhance more on this topic so that make it more helpful for others.
12 thoughts on “Filebeat, Logstash Output Configuration”