Filebeat Prospectors Configuration Changes for Read Log files


Filebeat Prospectors Configuration

Filebeat can read logs from multiple files parallel and apply different condition, pass additional fields for different files, multiline and include_line, exclude_lines etc. based on different log files.

Filebeat allows multiline prospectors on same filebeat.yml file.

How to decide number of prospectors in configuration file?

We can decide number of prospectors after categorizing same type logs file based on their format format and operation need to perform based on business need. I have find out some steps to divide in prospectors.

  • Read only: First decide what are files/files from need to read by filebeat. If that’s need to read and shipped output to some other system then only one prospectors is enough. If require any below case then require more prospectors for each category.
  • Multiline : If require multiline handling on filebeat end then divide selected files from above step to different category based on same file log format and where same multiline pattern  can  apply. Go to link for more information about Filebeat Configuration Changes for Multiline Logs Handling
  • Fields Handling: If need to pass some additional fields over shipping data from filebeat to Output System. If field detail are same for prospectors then no more prospectors required if different then again sub categories according to required field detail and define more prospectors.

How to define Prospectors?

Filebeat allow two type of prospector’s input_type log and stdin. Prospector setting start from filebeat.prospectors and each prospector implement with input_type. Here in below example will consider as input type of log.

Multiline Prospectors Example:

filebeat.prospectors:


#Prospectors 1 : Only reading logs line
input_type: log
paths:
- /var/app1/backend/debug-log.log
- /var/app1/frontend/debug-log.log
- /var/app1/backend/server.log
- /var/app1/frontend/server.log

#Prospector 2 : reading and sending some additional field
input_type: log
paths:
-/var/app2/log/*-debug.log
fields:
apache: true

#Prospectors 3 : reading, multiline and sending some additional fields
input_type: log
paths:
-/var/app2/log/*-debug.log

multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
#multiline.max_lines: 50

fields:
tz: EST

fields_under_root: true

Above example having three prospectors as given below

Prospector 1: reading logs files and shipped to output system.
Prospector 2: reading logs files and also sending additional fields like apache.
Prospectors 3: reading logs, multiline and also sending additional field for timezone.

Integration

Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Read More

To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial  and  Filebeat Issues. To know more about YAML follow link as YAML Tutorial

Leave you feedback to enhance more on this topic so that make it more helpful for others.

Advertisements

About Saurabh Gupta

My Name is Saurabh Gupta, I have approx. 11 Year of experience in Information Technology World manly in Java/J2EE. During this time I have worked with multiple organization with different client, so many technology, frameworks etc.
This entry was posted in Filebeat, Filebeat Issues and tagged , , , , . Bookmark the permalink.

12 Responses to Filebeat Prospectors Configuration Changes for Read Log files

  1. Pingback: Sample filebeat.yml file for Prospectors, Elasticsearch Output and Logging Configuration | Facing Issues On IT

  2. Pingback: Filebeat, Commandline Arguments Configuration | Facing Issues On IT

  3. Pingback: Sample filebeat.yml file for Prospectors ,Kafka Output and Logging Configuration | Facing Issues On IT

  4. Pingback: Sample filebeat.yml file for Prospectors,Multiline and Logging Configuration | Facing Issues On IT

  5. Pingback: Sample filebeat.yml file for Prospectors and Logging Configuration | Facing Issues On IT

  6. Pingback: Filebeat,Elasticsearch Output Configuration | Facing Issues On IT

  7. Pingback: Filebeat Multiline Configuration Changes for Object, StackTrace and XML | Facing Issues On IT

  8. Pingback: Filebeat Configuration Changes for Kafka Output | Facing Issues On IT

  9. Pingback: Filebeat Configuration Changes for Logstash Output | Facing Issues On IT

  10. Pingback: Filebeat Introduction | Facing Issues On IT

  11. Pingback: Filebeat Configuration Changes for Logging | Facing Issues On IT

  12. Pingback: Steps to Download,Installation and Start/Run Filebeat | Facing Issues On IT

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s