Filebeat Prospectors Configuration

Filebeat can read logs from multiple files parallel and apply different condition, pass additional fields for different files, multiline and include_line, exclude_lines etc. based on different log files.

Filebeat allows multiline prospectors on same filebeat.yml file.

How to decide number of prospectors in configuration file?

We can decide number of prospectors after categorizing same type logs file based on their format format and operation need to perform based on business need. I have find out some steps to divide in prospectors.

• Read only: First decide what are files/files from need to read by filebeat. If that’s need to read and shipped output to some other system then only one prospectors is enough. If require any below case then require more prospectors for each category.
• Multiline : If require multiline handling on filebeat end then divide selected files from above step to different category based on same file log format and where same multiline pattern  can  apply. Go to link for more information about Filebeat Configuration Changes for Multiline Logs Handling
• Fields Handling: If need to pass some additional fields over shipping data from filebeat to Output System. If field detail are same for prospectors then no more prospectors required if different then again sub categories according to required field detail and define more prospectors.

How to define Prospectors?

Filebeat allow two type of prospector’s input_type log and stdin. Prospector setting start from filebeat.prospectors and each prospector implement with input_type. Here in below example will consider as input type of log.

Multiline Prospectors Example:

filebeat.prospectors:

#Prospectors 1 : Only reading logs line
input_type: log
paths:
- /var/app1/backend/debug-log.log
- /var/app1/frontend/debug-log.log
- /var/app1/backend/server.log
- /var/app1/frontend/server.log

#Prospector 2 : reading and sending some additional field
input_type: log
paths:
-/var/app2/log/*-debug.log
fields:
apache: true

#Prospectors 3 : reading, multiline and sending some additional fields
input_type: log
paths:
-/var/app2/log/*-debug.log

multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
#multiline.max_lines: 50

fields:
tz: EST

fields_under_root: true

Above example having three prospectors as given below

Prospector 1: reading logs files and shipped to output system.
Prospector 2: reading logs files and also sending additional fields like apache.
Prospectors 3: reading logs, multiline and also sending additional field for timezone.

Integration

Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Read More

To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial  and  Filebeat Issues. To know more about YAML follow link as YAML Tutorial

Leave you feedback to enhance more on this topic so that make it more helpful for others.