Here is you will know about configuration for Elasticsearch Ingest Node, Creation of pipeline and processors for Ingest Node. You will see to configuration for filebeat to shipped logs to Ingest Node.
Below is some sample logs line which will be shipped through filebeat to Elasticsearch Ingest Node.
Sample Logs
2016-06-01 05:14:51,921 INFO main [com.mchange.v2.log.MLog] - MLog clients using log4j logging.
2016-06-01 05:14:52,065 INFO main [com.mchange.v2.c3p0.C3P0Registry] - Initializing c3p0-0.9.1.2 [built 21-May-2007 15:04:56; debug? true; trace: 10]
2016-06-01 05:14:52,728 INFO net.sf.ehcache.CacheManager@204119 [net.sf.ehcache.util.UpdateChecker] - New update(s) found: 2.4.7 [http://www.terracotta.org/confluence/display/release/Release+Notes+Ehcache+Core+2.4]. Please check http://ehcache.org for the latest version.
2016-06-01 05:14:53,198 WARN main [com.ibatis.sqlmap.engine.builder.xml.SqlMapParser] - Duplicate -include 'rowSubSelectEnd' found.
2016-06-01 05:14:53,199 WARN main [com.ibatis.sqlmap.engine.builder.xml.SqlMapParser] - Duplicate -include 'activitiesSelect' found.
2016-06-01 05:14:56,172 ERROR main [com.fsvps.clientPortal.service.clientReport.ReportsServiceImpl] - Invalid/unreadable reportsBasePath dir: C:homeclientReportsBase
2016-06-01 05:15:51,784 INFO main [com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource] - Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 1, acquireRetryAttempts -> 0, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2x2tdj9h36ggie1v52d93|144b457, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> oracle.jdbc.driver.OracleDriver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2x2tdj9h36ggie1v52d93|144b457, idleConnectionTestPeriod -> 140, initialPoolSize -> 0, jdbcUrl -> jdbc:oracle:thin:@10.226.168.6:1521:FSVD2, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 150, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 1, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> select 1, properties -> {user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ]
2016-06-01 05:16:13,317 INFO com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0 [com.mchange.v2.resourcepool.BasicResourcePool] - An exception occurred while acquiring a poolable resource. Will retry.
java.sql.SQLException: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:412)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:531)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:221)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:503)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:359)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:422)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:672)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:237)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1042)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:301)
... 12 more
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:141)
at oracle.net.nt.ConnOption.connect(ConnOption.java:123)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:337)
... 17 more
2016-06-01 05:16:13,317 INFO com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2 [com.mchange.v2.resourcepool.BasicResourcePool] - An exception occurred while acquiring a poolable resource. Will retry.
java.sql.SQLException: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:412)
at oracle.jdbc.driver.PhysicalConnection.(PhysicalConnection.java:531)
at oracle.jdbc.driver.T4CConnection.(T4CConnection.java:221)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:503)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:359)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:422)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:672)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:237)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1042)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:301)
... 12 more
Filebeat Configuration
To check this example put above sample logs in file app.log and configure filebeat.yaml file as below:
Prospectors Configuration
filebeat.prospectors:
- type: log
- D:your-directory-path*.log*
#Multiline configutaion as per above logs to stack trace
multiline.pattern: '^[([0-9]{4})-([0-9]{2})-([0-9]{2})'
multiline.negate: true
multiline.match: after
#Elasticsearch Ingest Node Pipe Line configuration
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
pipeline: grok-pipeline
Below is complete sample filebeat.yaml file
######Filebeat Configuration Example ########
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
# Below are the prospector specific configurations.
- type: log
# Change to true to enable this prospector configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- D:fsv_logs*.log*
#- c:programdataelasticsearchlogs*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: ['.gz
In above filbeat.yaml you can check for configuration for filbeat, multiline processing logs and Ingest Node pipe line.
Ingest Node pipeline Creation and configure Processors
You can execute below commands by using curl or Kibana Adsense /Dev tools to create pipeline to execute processors.
Here I have configured below processors with in pipeline:
grok processor : grok processor will parse logs message to fields values which will help to do analysis.
date processor : date processor will change @timestamp values corresponding timestamp of each logs line. Which will help while indexing and sorting of logs based on timestamp.
date_index_name : date_index_name processor will create elasticsearch indexes with suffix as date so that easily maintain indexes as per durability.
Kibana adsense
GET _template/
GET _ingest/pipeline/grok-pipeline
GET _ingest/pipeline/grok-pipelin?filter_path=*.version
DELETE _ingest/pipeline/my-pipeline-id
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description" : "grok parse of log message",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["^\[%{TIMESTAMP_ISO8601:timestamp}\]\s+\[%{LOGLEVEL:level}\]\s+\[%{GREEDYDATA:ip}\]\s+\[%{GREEDYDATA:CARD_ID}\]\s+\[%{GREEDYDATA:thread}\]\s+\[%{GREEDYDATA:CLASS_NAME}\]\s+-%{GREEDYDATA:complete_message}({({[^}]+},?\s*)*})?\s*$(?(?m:.*))?"]
}
}
]
},
"docs":[
{
"_source": {
"message": "[2014-03-27 13:30:01,106] [INFO] [0:0:0:0:0:0:0:1] [] [Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@79eb1cad] [com.mchange.v2.resourcepool.BasicResourcePool] - An exception occurred while acquiring a poolable resource. Will retry."
}
}
]
}
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description" : "grok parse of log message",
"processors": [
{
"date": {
"field": "timestamp",
"target_field" : "timestamp",
"formats": ["%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND},%{MILLISECOND}"],
"timezone" : "America/Chicago"
}
}
]
},
"docs":[
{
"_source": {
"timestamp": "2014-03-27 13:30:01,106"
}
}
]
}
PUT _ingest/pipeline/grok-pipeline
{
"description" : "grok parse of log message",
"version" : 123,
"processors": [
{
"grok": {
"field": "message",
"patterns": ["^\[%{TIMESTAMP_ISO8601:timestamp}\]\s+\[%{LOGLEVEL:level}\]\s+\[%{GREEDYDATA:ip}\]\s+\[%{GREEDYDATA:CARD_ID}\]\s+\[%{GREEDYDATA:thread}\]\s+\[%{GREEDYDATA:CLASS_NAME}\]\s+-%{GREEDYDATA:complete_message}({({[^}]+},?\s*)*})?\s*$(?(?m:.*))?"]
},
"date": {
"field": "timestamp",
"target_field" : "timestamp",
"formats": ["yyyy-MM-dd HH:mm:ss,SSS"],
"timezone" : "America/Chicago"
},
"date_index_name" : {
"field" : "timestamp",
"index_name_prefix" : "filebeat-",
"date_rounding" : "d"
}
}
]
}
PUT _ingest/pipeline/date-pipeline
{
"description" : "grok parse of timestamp to @timestamp",
"processors": [
{
"date": {
"field": "timestamp",
"target_field" : "timestamp",
"formats": ["TIMESTAMP_ISO8601"],
"timezone" : "America/Chicago"
}
}
]
}
delete filebeat*
GET _search
{
"query": {
"match_all": {}
}
}
GET _ingest/processor/grok
Elasticsearch
bootstrap.memory_lock: false
cluster.name: FIOT_CLUSTER
http.port: 9200
node.data: true
node.ingest: true
node.master: true
node.max_local_storage_nodes: 1
node.name: FIOT_NODE_1
path.data: D:ElasticsearchDatadata
path.logs: D:ElasticsearchDatalogs
transport.tcp.port: 9300
# Optional additional fields. These fields can be freely picked # to add additional information to the crawled log files for filtering #fields: # level: debug # review: 1 ### Multiline options # Mutiline can be used for log messages spanning multiple lines. This is common # for Java Stack Traces or C-Line Continuation # The regexp Pattern that has to be matched all lines starting with [ multiline.pattern: '^[([0-9]{4})-([0-9]{2})-([0-9]{2})' # Defines if the pattern set under pattern should be negated or not. Default is false. multiline.negate: true # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern # that was (not) matched before or after or as long as a pattern is not matched based on negate. # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash multiline.match: after #============================= Filebeat modules =============================== filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}modules.d*.yml # Set to true to enable config reloading reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s #==================== Elasticsearch template setting ========================== setup.template.settings: index.number_of_shards: 3 #index.codec: best_compression #_source.enabled: false #================================ Outputs ===================================== # Configure what output to use when sending the data collected by the beat. #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"] pipeline: grok-pipeline # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme"
In above filbeat.yaml you can check for configuration for filbeat, multiline processing logs and Ingest Node pipe line.
Ingest Node pipeline Creation and configure Processors
You can execute below commands by using curl or Kibana Adsense /Dev tools to create pipeline to execute processors.
Here I have configured below processors with in pipeline:
grok processor : grok processor will parse logs message to fields values which will help to do analysis.
date processor : date processor will change @timestamp values corresponding timestamp of each logs line. Which will help while indexing and sorting of logs based on timestamp.
date_index_name : date_index_name processor will create elasticsearch indexes with suffix as date so that easily maintain indexes as per durability.
Facing Issues On IT 
You must be logged in to post a comment.