Filebeat Prospectors Configuration
Filebeat can read logs from multiple files parallel and apply different condition, pass additional fields for different files, multiline and include_line, exclude_lines etc. based on different log files.
Filebeat allows multiline prospectors on same filebeat.yml file.
How to decide number of prospectors in configuration file?
We can decide number of prospectors after categorizing same type logs file based on their format format and operation need to perform based on business need. I have find out some steps to divide in prospectors.
- Read only: First decide what are files/files from need to read by filebeat. If that’s need to read and shipped output to some other system then only one prospectors is enough. If require any below case then require more prospectors for each category.
- Multiline : If require multiline handling on filebeat end then divide selected files from above step to different category based on same file log format and where same multiline pattern can apply. Go to link for more information about Filebeat Configuration Changes for Multiline Logs Handling
- Fields Handling: If need to pass some additional fields over shipping data from filebeat to Output System. If field detail are same for prospectors then no more prospectors required if different then again sub categories according to required field detail and define more prospectors.
How to define Prospectors?
Filebeat allow two type of prospector’s input_type log and stdin. Prospector setting start from filebeat.prospectors and each prospector implement with input_type. Here in below example will consider as input type of log.
Multiline Prospectors Example:
#Prospectors 1 : Only reading logs line input_type: log paths: - /var/app1/backend/debug-log.log - /var/app1/frontend/debug-log.log - /var/app1/backend/server.log - /var/app1/frontend/server.log #Prospector 2 : reading and sending some additional field input_type: log paths: -/var/app2/log/*-debug.log fields: apache: true #Prospectors 3 : reading, multiline and sending some additional fields input_type: log paths: -/var/app2/log/*-debug.log multiline.pattern: '^\[' multiline.negate: true multiline.match: after #multiline.max_lines: 50 fields: tz: EST fields_under_root: true
Above example having three prospectors as given below
Prospector 1: reading logs files and shipped to output system.
Prospector 2: reading logs files and also sending additional fields like apache.
Prospectors 3: reading logs, multiline and also sending additional field for timezone.
Leave you feedback to enhance more on this topic so that make it more helpful for others.