Tag Archives: topic

Logstash Input and Output to/from Kafka Example


Logstash Input and Output to/from Kafka Example Continue reading

Advertisements
Posted in Kafka, Logstash | Tagged , , , , , | 6 Comments

Integrate Filebeat with Kafka


Kafka can consume messages published by Filebeat based on configuration  filebeat.yml file for Kafka Output. Filebeat Kafka Output Configuration Filebeat.yml  required below fields to connect and publish message to Kafka for configured topic. Kafka will create Topics dynamically based on filebeat requirement. … Continue reading

Posted in Filebeat, Kafka | Tagged , , , , , , | 6 Comments

Setup Kafka Cluster for Multi/Distributed Servers/Brokers


For setting up Kafka Cluster for Multi Broker/ Server on single Machine follow below steps: In below example we will create Kafka cluster with three brokers on single machine. All steps are same as configured for Kafka Cluster with Single Server … Continue reading

Posted in Kafka | Tagged , , , , , , | 6 Comments

Integrate Java with Kafka


Below examples are for Kafka Logs Producer and Consumer by Kafka Java API. Where Producer is sending logs from file to Topic1 on Kafka server and same logs Consumer is subscribing from Topic1. While Kafka Consumer can subscribe logs from … Continue reading

Posted in Kafka | Tagged , , , , , | 6 Comments

Kafka Introduction and Architecture


Kafka is Open source distributed, Steam Processing, Message Broker platform written in Java and Scala developed by Apache Software Foundation. Kafka is massively use for enterprise infrastructure to process stream data or transaction logs on real time. Kafka provide unified, … Continue reading

Posted in Kafka | Tagged , , , , , , , , , | 7 Comments

Filebeat, Kafka Output Configuration


If need  to shipped server logs lines  directly to Kafka. Follow below steps: Pre-Requisite : Start Kafka  before start filebeat to listen publish events and configure filebeat with same kafka server port Kafka Output  Required Configuration : Comment out output.elasticsearch … Continue reading

Posted in Filebeat, Kafka | Tagged , , , | 12 Comments