Tag Archives: Proxy

Elasticsearch REST Java Connection Client


For Elasticsearch connection, Elasticsearch REST Java API provide RestClient and RestClient.build() to get connection.

Below connection client class covers all the ways of connectivity to elastic search depend of elastic servers configuration and  accessibility accordingly you can uncomment below methods in customize Http Client.  I made this class as Singleton because Elasticsearch client keep connection persistent.

public class ElasticSearchConnectionRestTest {
	private RestClient client = null;
	private static ElasticSearchConnectionRestTest esc = null;
	private ElasticSearchConnectionRestTest() {
	}

	public static synchronized ElasticSearchConnectionRestTest getInstance() {
		if (esc == null) {
			esc = new ElasticSearchConnectionRestTest();
		}
		return esc;
	}

	public RestClient getClient() {
		if (client == null) {
			getElasticSearchClient();
		}
		return client;
	}

private RestClient getElasticSearchClient() {

//Basic credential settings
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("usrid", "password"));

client = RestClient.builder(new HttpHost("elasticserchhost1",
Integer.parseInt("elasticsearchport1")),
new HttpHost("elasticserchhost2",
Integer.parseInt("elasticsearchport2"))))
.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
 //Security Settings
@override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
	return httpClientBuilder
//disable preemptive authentication so that same request done when next
//request
.disableAuthCaching()

//Credentials
.setDefaultCredentialsProvider(credentialsProvider)

//Proxy server settings
.setProxy(new HttpHost("one.proxy.att.com", 8080))

//setting for key store for JKS	SSL
//.setSSLContext(sslcontext)

//Number of threads will execute
//.setDefaultIOReactorConfig(IOReactorConfig.custom().setIoThreadCount(1).build())
//connection timeout
//.setConnectTimeout(5000)
//socket connection timeout
//.setSocketTimeout(60000)
								;
}
)
//Max retry timeout
.setMaxRetryTimeoutMillis(60000).build();
return client;
}

private void closeConnnection()
{
try
{
client.close();
}catch(IOException ex)
{
ex.printStackTrace();
}
}
}

Details of API’s used by Elasticsearch REST Connection

setHttpClientConfigCallback 

This callback method allows to modify the http client configuration like  encrypted ,proxy, communication over ssl, socket timeout etc.  By using customizeHttpClient we can configure all these values.

setDefaultHeaders

We can set default header if need to sent some value with in  every request.

setMaxRetryTimeoutMillis

The timeout value if request do multiple attempts for same request.

setFailureLister

This is listener to get notification whenever get any node fails and need to get any action for same.

Methods of CustomizeHttpClient Builder

Timeout Configuration

connectTimeout: Default value is 1 seconds

socketTimeout: Default 30 seconds

maxRetryTimeoutMilliseconds: Default 30 seconds

Thread Configuration

IoThreadCount:  Client start with default one thread and a number of worker threads used by connection manager, as many as the number of locally detected processors.

Authentication Configuration

setDefaultCredentialsProvider :  This method require basic credentials for authentication.

httpClientBuilder.disableAuthCaching(): We can disable authentication caching and sent in  every request headers to elasticsearch  if it will accepted and,  if it get failed by receiving a HTTP 401 response message, it will resend the exact same request again with the basic authentication header.

Encrypted Communication

setSSLContext:  Set this value for SSL Context for encrypted communication.

Read More on Elasticsearch REST

Integration

Integrate Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Logstash Connection Refused by Elasticsearch Over Proxy or AIC


We faced connection refused issue if  trying to Logstash output data to Elasticsearch over HTTP that happen because of Proxy configuration or if Elasticsearch on cloud environment.

Generally we faced below exception

[2017-04-24T10:45:32,933][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#}
[2017-04-24T10:45:32,934][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_second
Logstash allow proxy declaration in configuration file for Elasticsearch Output as given below in field proxy. For user userid and password is having any special symbol than you have to use ASCII value. For example my password is music@123 then after converting to ASCII value for that is %40 my password become music%40123. Refer this link ASCII CODE for getting ASCII value corresponding to each character.
proxy => "http://userid:passowrd@proxyhost:8080"

For  example my userid and password is “smart” and “music@123” below proxy configuration like

proxy => "http://smart:music%40123@proxyhost:8080"

How to set Proxy in Logstash Configuration for Elasticsearch Output?

output {
    elasticsearch {
       index => "app1-logs-%{+YYYY.MM.dd}"
       proxy => "http://smart:music%40123@proxyhost:8080"
       hosts => ["elasticServerHost:elasticServerPort"]
       }
}

Issues Solution

For more Logstash issues solution follow link Common Logstash Issues.

Filebeat, Logstash Output Configuration


If need  to shipped server logs lines  directly to Logstash. Follow below steps:

Pre-Requisite :

  • Create Logstash Configuration file  with input section mentioned same port as configured in filebeat for logstash listener. Default port for logstash is 5044.
  • Start Logstash with same configuration file.

Logstash Output  Required Configuration :

  • Comment out output.elasticsearch output section and uncomment output.logstash section
  • Set enabled value is true to make logstash output as enabled
  • Set host  of server where Logstash is running for listening  by default port for Logstash is 5044 if any change use same port value.
output.logstash:
 enabled:true
#use localhost if on same machine and same port                                                                    useby  logstash listener
 hosts:["logstashserver:5044"]

Other Optional Configurations:

Logstash Output Compression Configuration:

Filebeat provide gzip compression level which varies from 1 to 9. As compression level increase processing speed will reduce but network speed increase.By default compression level disable and value is 0.

compress_level:0

Logstash Output Performance Configuration:

worker:  we can configure number of worker for each host publishing events to elasticseach which will do load balancing.

loadbalance: Default value is false.  If set to true will check status of hosts if unresponsive will send to another available host. if false filebeat will select random host and send events to it.

pipelining: Default value is 0 means pipeline disabled. Configure value decide of pipeline  batches to send to logstash asynchronously and wait for response. If pipeline value is written means output will blocking.

Logstash Output Proxy Configuration: Filebeat use SOCKS5 protocol to communicate with logstash servers. If any proxy configure for this protocol on server end then we can overcome by setting below details.

proxy_url:socks5://userid:pwd@socks5-server:2233

proxy_use_local_resolver: Default value is false means resolve host name resolution on  proxy server. If value is set as true Logstash host name resolution locally for proxy.

Sample configuration file

Sample filebeat.yml file for Logstash Output

Integration

Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Read More

To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial  and  Filebeat Issues. To know more about YML follow link YAML Tutorials.

Leave you feedback to enhance more on this topic so that make it more helpful for others.

Filebeat,Elasticsearch Output Configuration


If we need  to shipped server logs lines  directly to elasticseach  over HTTP by filebeat . We have set below fields for elasticsearch output according to your elasticsearch server configuration and follow below steps.

  1.  Uncomment output.elasticsearch in filebeat.yml file Elasticsearch
  2.  Set host and port in hosts line
  3.  Set index name as you want. If it’s not set filebeat will create default index as “filebeat-%{+yyyy.MM.dd}” .
output.elasticsearch :

   enabled:true
   hosts:["localhost:9200"]
   index:app1-logs-%{+yyyy.MM.dd}"

Elasticsearch server credentials configuration if any 

  1.  Set user name and password
  2.  Set protocol if https because default protocol is http
    username:userid
    password:pwd

Elasticsearch Index Template Configuration: We can update elasticsearch index template from filebeat which will define settings and mappings to determine field analysis.

Auto Index Template Loading: Filebeat package will load default template filebeat.template.json to elasticsearch if no any template configuration for template and will not overwrite template.

Customize Index Template Loading: We can upload our user define template and update version also by using below configuration.

#(if set as false template need to upload manually)
template.enabled:true
#default value is filebeat
template.name:"app1"
#default value is filebeat.template.json.
template.path:"app1.template.json"
#default value is false
template.overwrite:false 

By default, template.overwrite value is false and will not overwrite index template if already exist on elasticsearch.  For overwriting index template make this flag as true in filebeat.yml configuraton file.

Latest Template Version Loading from Filebeat: Set template.overwrite as true and if need to update template file version as 2.x then set path of Latest template file with below configuration.


template.overwrite:true
template.versions.2x.enabled: true
template.versions.2x.path:"${path.config}/app1.template-es2x.json"

Manually Index Template Loading : for manually index loading please refer Elasticsearch Index Template Management.

Compress Elasticsearch Output :  Filebeat provide gzip compression level which varies from 1 to 9. As compression level increase processing speed will reduce but network speed increase.By default compression level disable and value is 0.


compress_level: 0

Other configuration Options:

bulk_max_size : Default values is 50. If filebeat is generating events more than configure batch max size it will split events in configure size batches and send to elasticsearch. As much as batch size will increase performance will improve but require more buffring. It can cause other issue like connection, errors, timeout for requests.

Never set value of bulk size as 0 because there would not be any buffering for events and filebeat will send each event directly to elasticsearch.

timeout: Default value is 90 seconds. If no response http request will timeout.

flush_interval: waiting time for new events for bulk requests. If bulk request max size sent before this specified time, new bulk index request created.

max_retries: Default value is 3. When max retry reach specified limit and evens not published all events will drop. Filebeat also provide option to retry until all events are published by setting value as less than 0.

worker:  we can configure number of worker for each host publishing events to elasticseach which will do load balancing.

 Sample Filebeat Configuration file:

Sample filebeat.yml file to Integrate Filebeat with Elasticsearch

Integration

Complete Integration Example Filebeat, Kafka, Logstash, Elasticsearch and Kibana

Read More

To read more on Filebeat topics, sample configuration files and integration with other systems with example follow link Filebeat Tutorial  and  Filebeat Issues. To know more about YAML follow link YAML tutorials.

Leave you feedback to enhance more on this topic so that make it more helpful for others.