Questions tagged [elastic-stack]

2

votes
1

answer
69

Views

Reuse/cache script fields to use other parts of the query

I am working on a project where we need to show distance the listings are from a user location. To show the distance, when lat/lon is given in the input, we calculate distance using a script field called "distance" "script_fields" : { "distance" : { "script" : { "lang": "painless", "source": "Math.r...
Anam
2

votes
0

answer
17

Views

What is the nest way to bulk index(around 40 k files of type .docx) using ingest-attachment?

I am fairly familiar with the ELK stack and currently using Elastic search 6.6. Our use case is content search for about 40K .docx files (uploaded by Portfolio managers as research reports. Max file size allowed is 10 MB, but mostly file sizes are in few Kb). I have used the ingest attachment plu...
Priyanka Shah
1

votes
0

answer
13

Views

where can I find ELK version in REST API?

I would like to get ELK version through REST API or parse html. I search in API documentation without finding anything Re-edit: In python ... i'm not found better than re.findall(r"version":"(\d\.\d\.\d)"", requests.get(my_elk).content.decode())[0]
ggens
1

votes
1

answer
739

Views

Recommended RAM ratios for ELK with docker-compose

I have a production server with 8GB RAM. Im looking to host elastic,logstash and kibana on the server. Using documer compose. What would be the recommended java sizes memory sizes for each of the containers. How might I configure this. My docker-compose looks like the following --- version: '3' se...
Robbo_UK
1

votes
2

answer
55

Views

Time mismatch in kibana

We are having ELK setup with Kibana version 5.6.10. We are facing a time mismatch in displaying logs from different servers. We are fetching log from 8 IIS server and parsing via Logstash to Elastic search Kibana. While filtering logs for past hour we could notice only 2 server logs were displayed....
Arumugaperumal
1

votes
1

answer
12

Views

script filter field value is in array

Here is a part of my query: must_not: { script: { script: { source: "doc['type.keyword'].value=='CHANNEL' ? doc['id'].value == 0 : doc['id'].value.contains(['1','3','7'])", lang: 'painless' } } } How can I check if doc['id'].value has a value from an array?
Crina Grigore
1

votes
3

answer
2.8k

Views

How do I increment my count field of my document from logstash?

I want to update one field of my document/log in elasticsearch from logstash. My logstash conf file input { http { host => "127.0.0.1" # default: 0.0.0.0 port => 31311 # default: 8080 } } output { stdout { codec => json }, elasticsearch { action => "update" bind_host => "127.0.0.1" bind_port => 92...
nm10
1

votes
2

answer
516

Views

Docker - Cannot connect to the Docker Daemon to rm containers

Playing with ELK and docker, I needed to restart every services. docker ps told me that I haven't any containers up. docker run -it --rm [...] --name es elasticsearch -> Error response from daemon. The name "es" is already use by container [...] So I try to remove all container : docker ps -a -q | x...
Ragnar
1

votes
1

answer
4.3k

Views

Send filebeat output to multiple Logstash servers without load balancing

I am trying to send the same logs from Filebeat to two different servers (one Logstash and one Graylog server) without load balancing. We are testing ELK and Graylog at our company and for testing purposes, we'd like to send the logs to two different stacks. However, on the filebeat.yml file, I only...
barsha shrestha
1

votes
2

answer
337

Views

Amazon elasticsearch interpretation of FreeStorageSpace metrics

I have 6 instances of type m3.large.elasticsearch and storage type instant. I don't really get what does Average, Minimum, Maximum ..mean here? I am not getting any logs into my cluster right now although it shows FreeStorageSpace as 14.95GB here: But my FreeStorageSpace graph for "Minimum" has reac...
PuRaK
0

votes
1

answer
12

Views

How do I refer to multiple nesting levels in an Elastic Search's Filter Aggregation?

Let's call my root level foo and my child level events. I want to aggregate on the events level but with a filter that EITHER the event has color "orange" OR the parent foo has customerId "35". So, I want to have a filter aggregation that's inside a nested aggregation. In this filter's query clause,...
K. M
1

votes
2

answer
1.1k

Views

How to aggregate fields based particular date in every year in elasticsearch?

I'm having data in my index from 2010 to 2015. I have used following code to get the aggregated first name details in every year from 2010 to 2015, it works as expected POST profile/_search { "size": "0", "aggs": { "count_by_year": { "date_histogram": { "field": "logdate", "interval": "year", "forma...
Pandiyan Cool
0

votes
0

answer
5

Views

"No mapping found for [@timestamp] in order to sort on- Failed Query

my logstash say that some querys fail, because there is no mapping in the specific index. There are several questions like mine I know, but my problem just occurs if I enable the X-Pack Security. Without there are no problems. So I guess it have something to do with the X-Pack indices. Here My custo...
Ricardo
18

votes
6

answer
21.2k

Views

Docker apps logging with Filebeat and Logstash

I have a set of dockerized applications scattered across multiple servers and trying to setup production-level centralized logging with ELK. I'm ok with the ELK part itself, but I'm a little confused about how to forward the logs to my logstashes. I'm trying to use Filebeat, because of its loadbalan...
Gianluca
2

votes
1

answer
2.2k

Views

LogStash - Failed to instantiate type net.logstash.logback.appender.LogstashTcpSocketAppender

I am working on Springboot Microservcies & for monitoring Im using ELK Stack. I am using docker containers for running ELK as per this guide. ELK is up and running, I am starting my Logstash by, docker run -d -it --name logstash -p 5000:5000 logstash -e 'input { tcp { port => 5000 codec => "json" }...
John Seen
2

votes
1

answer
432

Views

ElasticSearch 6, copy_to with dynamic index mappings

Maybe I'm missing something simple, but still could not figure out the following thing: As of ES 6.x the _all field is deprecated, and instead it's suggested to use the copy_to instruction (https://www.elastic.co/guide/en/elasticsearch/reference/current/copy-to.html). However, I got an impression th...
Vyacheslav
3

votes
1

answer
2.6k

Views

error=>“Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)”}

ELK with salesforce URL:http://localhost:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"} docker-compose.yml version: '2'...
Jain
5

votes
3

answer
8.7k

Views

Logstash creates enormous local log files and (although it sends then to Elasticsearch)

The problem I have a machine with logstash on it, and another Elasticsearch-Kibana machine which stores the logs written from logstash on the first machine. Naturally, I want no logs to be kept on the origin machine and handle logging only on the Elasticsearch cluster. Unfortunately, logstash create...
Adam Matan
2

votes
1

answer
4.4k

Views

Optimal way to set up ELK stack on three servers

I am looking to set up an ELK stack and have three servers to do so. While I have found plenty of documentation and tutorials about how to actually install, and configure elasticsearch, logstash, and kibana, I have found less information about how I should set up the software across my servers to ma...
jeromefroe
5

votes
2

answer
3.7k

Views

Logstash not working with multiple files wildcard path

Logstash doesn't seem to read path with wildcard here is my config file input { file { path => "C:\logs\app*.log" type => "MyType" } } filter { } output { elasticsearch { } }
Amir Katz
2

votes
0

answer
1.2k

Views

Docker ELK 5.4.0 - Kibana did not load properly. Check the server output for more information

I'm experiencing troubles on kibana official docker ELK stack. I'm using docker compose to startup a complete stack made of elasticsearch, logstash and kibana. The stack seems fine to me and everything is ok in logstash and elasticsearch. i can find everithing as expected inside the relative logs...
2

votes
2

answer
657

Views

Displaying n-greatest in Kibana

In my elasticsearch backend, I have a index stats with a numerical field and a @name field. I would like to display a pie chart (or some other panel type) in Kibana where you can list the top N @name where the percentage in the pie represents the percent of the total fields are represented. For exa...
user3249763
2

votes
1

answer
697

Views

How to get Stdout from a service into logstash

I understand that a similar question was asked here: How can I configure logtash to use "stdout" as input?, but I wanted to generalize the question a little. What I have currently configured is the basic ELK stack(elasticSearch, logstash, kibana) on a Linux VM and I want to feed a bit of standard ou...
TKoch
4

votes
2

answer
8.9k

Views

Kibana Regular expression search

I am newbie to ELK. I want to search for docs based on order of occurrence of words in a field. For example, In doc1, my_field: "MY FOO WORD BAR EXAMPLE" In doc2, my_field: "MY BAR WORD FOO EXAMPLE" I would like to query in Kibana for docs where "FOO" is followed by "BAR" and not the opposite. So,...
Krishna Chaitanya
2

votes
2

answer
199

Views

Elasticsearch query not giving exact match

Am searching elasticsearch with the below match query, which is not giving me the exact match instead its giving some more irrevalant match also. am using elasticsearch 6.2.3 Please find my query below get items/_search { "query" : { "match" : { "code" : "7000-8900" } } } Please find the response am...
Karthikeyan
5

votes
1

answer
1k

Views

Unable to install Search Guard plugin for Elasticsearch-5.x

Due to the restrictions, I was not allowed to install any packages from internet. So, This command is not useful for me inorder to install search-guard. bin/elasticsearch-plugin install -b com.floragunn:search-guard-ssl: However, I am able to install Search Guard successfully on a different network...
hello world
2

votes
1

answer
404

Views

ElasticSearch Java RestClient - listener timeout after waiting f or [30000] ms

Am querying elasticsearch index documents which is having more than 100K documents via java code. Am using RestClient for that. While am trying to fetching documents am getting Exception in thread "main" java.io.IOException: listener timeout after waiting f or [30000] ms Please find the error below...
Karthikeyan
1

votes
2

answer
2.2k

Views

using query_string query with bool in elastic search causing parsing exception

Why is this query giving me a parsing exception? If I remove the bool it does seem to work. But I need the bool there with the query_string. How can I make this work? { "query": { "filtered": { "query": { "bool": { "must": [ { "terms": { "status_type": [ "5" ] } } ] } }, "filter": { "query_string":...
Horse Voice
12

votes
3

answer
7.6k

Views

Logging from Java app to ELK without need for parsing logs

I want to send logs from a Java app to ElasticSearch, and the conventional approach seems to be to set up Logstash on the server running the app, and have logstash parse the log files (with regex...!) and load them into ElasticSearch. Is there a reason it's done this way, rather than just setting up...
Rob
6

votes
1

answer
3.1k

Views

analyzed or not_analyzed, what to choose

I'm using only kibana to search ElasticSearch and i have several fields that can only take a few values (worst case, servername, 30 different values). I do understand what analyze do to bigger, more complex fields like this, but the small and simple ones i fail to understand the advance/disadvantage...
higuita
2

votes
1

answer
2.9k

Views

Percent Metric in Kibana 4?

Hey I'm using the elk stack to store and view test results. I want to see a nice display something like this: _Pass_____Fail_ 99.99%___00.01% However I can't find any way to do this. Is this possible in Kibana 4? I feel like this is a very basic feature that should be available...
abferm
6

votes
1

answer
1.3k

Views

Change type and reindex in Elasticsearch

I recently upgraded my ELK stack (logstash 2.3.4 using redis 3.2.3, Elasticsearch 2.3.5 and Kibana 4.5.4) from (logstash 1.4.1/1.4.2 using redis 2.8.24, Elasticsearch 1.2.2 and Kibana 3.1.1). The upgrade went well but after the upgrade I had some fields that had conflicting types. This specific fi...
Mike
6

votes
0

answer
3.9k

Views

org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 30662099 larger than 30662028)

I am trying to push data from Flume channels to Kafka cluster using Kafka sink and i can see related data into associated topic but simultaneously i am observing below mentioned exception trace in Kafka logs too frequently , [2017-03-21 16:47:56,250] WARN Unexpected error from /10.X.X.X; closing co...
Ritesh Sharma
12

votes
2

answer
17k

Views

Kibana Alternatives

I am having an issue in kibana. It does not show any results in the Discover tab. Please look here for more information. Do we have any Kibana alternatives that the community has used? I searched on the internet and I could find only Head elasticSearch plugin. If nothing works, then I will work on...
Ajit Goel
12

votes
1

answer
17.2k

Views

Elasticsearch: No handler for type [keyword] declared on field [hostname]

I get above Mapper Parsing Error on Elasticsearch when indexing log from filebeat. I tried both Filebeat -> Elasticserach and Filebeat -> Logstash -> Elasticsearch approach. I have followed their own documentations, I installed filebeat template as per instructed and verified from Loading the Index...
rayhan
2

votes
1

answer
1.8k

Views

Filebeat > is it possible to send data to Elasticsearch by means of Filebeat without Logstash

I am a newbie of ELK. I installed first Elasticsearch and Filebeat without Logstash, and I would like to send data from Filebeat to Elasticsearch. After I installed the Filebeat and configured the log files and Elasticsearch host, I started the Filebeat, but then nothing happened even though there a...
Rui
1

votes
1

answer
176

Views

Logstash date filter configuration

I am trying to replace @timestamp with tstamp but it is not working. What am I missing here? Here is my configuration date { match => ["tstamp","yyyy-MM-dd'T'HH:mm:ss.SSSZ"] target => "@timestamp" } and the sample date entry is like this 2016-04-02T09:29:50.348Z 2016-04-02T08:52:49Z 2016-04-02T02:52...
user3847894
2

votes
2

answer
4.4k

Views

Log storage location ELK stack

I am doing centralized logging using logstash. I am using logstash-forwarder on the shipper node and ELK stack on the collector node.I wanted to know the location where the logs are stored in elasticsearch i didn't see any data files created where the logs are stored.Do anyone has idea about this?
Anand j. Kadhi
3

votes
2

answer
1.1k

Views

Kibana 5.5.1 behind a nginx 1.13 proxy (dockerized)

Goal: I want to run the elk stack in a docker container. To be able to access the ELK Stack over a nginx proxy to bypass the individual ports for the services. The Kibana service (default port 5601) http://.com:5601 should be reachable over the following address: http://.com/kibana Problem: The prob...
neutron
5

votes
1

answer
4.3k

Views

Application logging with ELK stack

Using NLog with Elasticsearch target to forward logs to AWS Elasticsearch as a Service cluster for visualisations in Kibana. This works fine but I am concerned about using this in production due to ES cluster availability and the impact a cluster failover has, when the logs are sent using the elasti...
thedev

View additional questions