Questions tagged [elastic-stack]

1

votes
1

answer
348

Views

Logstash, how to use grok patterns coming from event data

I have an ELK stack deployed on kubernetes used to collect containers' data. Among all the rest, it is using a grok filter to parse the actual log line based on a pattern. My wish is to be able to setup this pattern by using an annotation in the kubernetes pod. I added an annotation called elk-grok-...
whites11
1

votes
0

answer
30

Views

Relation based search with elasticsearch / kibana

I'm running a software where a client-application communicates with a backend-service. In order to be able to track what the client does I want to analyze my logs in the backend. Therefore I've setup an ELK stack in the backend. Setup and application behaves as follows: a) filebeat delivers logfiles...
BK_
1

votes
1

answer
441

Views

Converting from SQL to elasticsearch query

Elasticsearch noob and need help with a query. I have the following SQL query that I need to convert to a query to Elasticsearch SELECT COUNT(*) FROM table WHERE Message LIKE '%Communication has failed.%' AND [Date] > CONVERT( CHAR(8), GetDate(), 112) + ' 07:40:00' AND [Date] < CONVERT( CHAR(8),...
flalar
1

votes
0

answer
178

Views

Getting logstash to run on Windows

I downloaded the zip version of the logstash from LogStash Download Link I unziped it and ensured the folder path does not have any space. Also I ensured that the JDK 64bit is available in the path. But when I try to start logstash, I am getting 'could not find jruby in C:\ELK\logstash-6.1.3\vendor...
vinSan
1

votes
0

answer
31

Views

filebeat can't get log written at php by “fwrite”

My filebeat can't get the log file that written on php by 'fwrite' Here is my php coding: $date = date('Y-M-d'); $log_path = JSON_LOG_PATH.'_'.$date.'_'.$eventid.'.log'; if (!is_file($log_path)) { # code... if( ($jsonLog=fopen ($log_path,'w')) === FALSE){ return false; } $json_string = json_encode($...
vincent wang
1

votes
1

answer
509

Views

Filebeat not pushing logs to Elasticsearch

I am new to docker and all this logging stuff so maybe I'm making a stuipd mistake so thanks for helping in advance. I have ELK running a a docker container (6.2.2) via Dockerfile line: FROM sebp/elk:latest In a separate container I am installing and running Filebeat via the folling Dockerfile line...
Flash Death
1

votes
0

answer
200

Views

How to collect more than 22 event ids with winlogbeat?

I've got a task to collect over 500 events from DC with winlogbeat. But windows got a limit 22 events to query. I'm using version 6.1.2. I've tried with processors like this: winlogbeat.event_logs: - name: Security processors: - drop_event.when.not.or: - equals.event_id: 4618 ... but with these sett...
FanteG
1

votes
0

answer
18

Views

Sort Aggregation on decay function in Elasticsearch

this is my decay function with aggregation for getting topics count. GET goal,user,summary,tip/_search { 'query': { 'function_score': { 'exp': { 'created': { 'origin': 'now', 'scale': '1d', 'decay' : 0.5 } } } }, 'aggs' : { 'topics': { 'terms': { 'field': 'topics.keyword' } } } } I want my answer...
Saad Ahmed
1

votes
0

answer
110

Views

Error with logstash Date filter yyyy-MM-dd HH:mm:ss Z format

I have a ruby (Grape API) application that's logging its dates like the following 2018-04-07 15:20:30 -0700 unfortunately, no matter what mutations, date filters, or grok filters that I attempt it seems that the - format of the date is causing issues, and resulting in the value remaining a string a...
ochero
1

votes
0

answer
48

Views

Querying Multiple Indexes (of different field mappings) causing Failed Shards Exception

I am trying to create Coordinate Map Visualization in Kibana, using an Index Pattern which covers two Indices of ElasticSearch. One Index contains the location (lat/lon) info whereas the other contains live data from IoT devices relevant to these locations. Both these indices have different fields o...
waleed ali
1

votes
0

answer
114

Views

Adding filebeat to ebextensions alongside a war file created with AWS Code Pipelines

I have a Spring application that automatically builds and deploys using AWS Code Pipeline and Elastic Beanstalk. In order to get that to work, I had to create an .ebextensions file fix-path.config as follows container_commands: fix_path: command: 'echo Hello world' command: 'unzip MyApp.war 2>&1 >...
jpetrichsr
1

votes
0

answer
215

Views

How to add Kibana 5 into your angular4 application

I have a default port of kibana5 and i want to integrate it in my angular4 application. I have no idea how to do that. I want the guiding steps . My application uses - angular4 node.js mysql And I have a dummy dashboard in my app and i want to include kibana dashboard in it, the UI like this - I g...
1

votes
1

answer
159

Views

Xunit test results to ELK stack - anyone tried this or know of projects?

I am planning to put my organization's test run results into the ELK stack for analysis and have had no luck finding any code to leverage. I've downloaded and analyzed a variety of xunit xml outputs and have noted that there are variations in the format, which is kind of a bummer. I am using python...
chrismead
1

votes
1

answer
294

Views

Plugin not Working with FileBeat in ElasticSearch

Blockquote2018-05-31T16:51:02.494+0530 ERROR fileset/factory.go:93 Error loading pipeline: Error loading pipeline for fileset apache2/access: This module requires the following Elasticsearch plugins: ingest-user-agent, ingest-geoip. You can install them by running the following commands on all t...
Harshit Yadav
1

votes
0

answer
220

Views

How to check if a field value exists before inserting in elasticsearch?

I have a working ELK with input coming from filebeat prospecting several log files and sending them to logstash. Logstash retrieves the stream, filters that in order to match lines with some fields and then sends them into elasticsearch. Now I would like to check before output to elasticsearch, if f...
Edouard HINVI
1

votes
1

answer
62

Views

Wso2carbon logs 2 logstash using tcp input plugin, without filebeat

I'm trying to transfer wso2carbon logs to elk using tcp input plugin my config for wso2 log4jproperties file. #TCP logger pattern log4j.appender.tcp=org.apache.log4j.net.SocketAppender log4j.appender.tcp.layout=org.wso2.carbon.utils.logging.TenantAwarePatternLayout # ConversionPattern will be over...
Sandeep
1

votes
1

answer
57

Views

Grok patterns format in logstash

Mostly my patterns working but after Framework Error in :] part does not see in the logs. How can print rest of it on logs. I tried at the last in patterns, but still does not work %{GREEDYDATA:restofthem} My grok Pattern: \[%{TIMESTAMP_ISO8601:ServerTimestamp}\|%{WORD:Log4netHostname}\|%{DATA:Proj...
Dylan_
1

votes
1

answer
404

Views

Grok pattern for data separated by pipe with whitespaces and optional values in it

I have a textfile/logfile in which the values are separated by a pipe symbol. '|' with multiple whitespaces. Also I just wanted to try it without gsub. An example is below, Does anyone know how to write a GROK pattern to extract it for logstash? as I am very new to it. Thanks in advance 5000|...
up6616
1

votes
0

answer
67

Views

method admin() is undefined for the type RestHighLevelClient

I'm using Ingest Attachment Processor Plugin on elasticsearch. I need to set attachment optionswith Java API. How can I do that? I am creating index and setting pipeline like below: RestHighLevelClient restHighLevelClient = null; File file = new File(filePath); try { FileInputStream fileInputStreamR...
Karthikeyan
1

votes
2

answer
178

Views

Kubernetes - Daemonset Error Reason

Have a k8s daemonset which is simply supposed to set sysctl -w vm.max_map_count=262144 on host nodes where the pod is deployed. The daemonset works as expected when the reosurce is applied the first time but, if the k8s nodes the daemonsets are running on are later restarted, the daemonset pods do n...
Going Bananas
1

votes
0

answer
133

Views

How to sync InfluxDB with Elasticsrearch?

I'm trying to synchronize an influxDB database with elasticsearch, because I want to use Kibana instead of Grafana, but I can't find any plugins to do it. Is there a plugin I can use? Is there any way to synchronize only the data I need?
Nono
1

votes
1

answer
665

Views

Elasticsearch connection refused while kibana is trying to connect

I am trying to run ELK stack using docker container. But I am getting error that kibana is unable to make connection with elasticsearch. kibana_1 | {'type':'log','@timestamp':'2018-06-22T19:31:38Z','tags':['error','elasticsearch','admin'],'pid':12,'message':'Request error, retrying\nHEAD ht...
LifeIsButifool
1

votes
0

answer
31

Views

LogStash specific fields as index

I'm trying to sync data between mongo and Elasticsearch with Logstash. I am using json files as input and single outputs to different elasticsearch indexes. I am still seeing the index generated by elastic search as random UUID. Instead I want the status field to be indexed. Here is my config: input...
jojo
1

votes
0

answer
48

Views

viewing syslogs on kibana using logstash

I have followed the steps as shown in the video. I am trying to run the below logstash3.conf file: input{ file{ path=>'/var/log/syslog' start_position=>'beginning' sincedb_path => '/dev/null' } } output{ elasticsearch{ hosts=>['elasticsearch:9200'] index=>'pop' } stdout{} } I gave the following c...
learner
1

votes
1

answer
235

Views

ELK in Docker and Beats in a machine in a different network

I was trying to configure ELK in docker containers in my private network and A Linux box with beats in another private network. I was trying to generate SSL cert for Filebeat to verify the identity of ELK Server. I tried using the public IP of the container host by forwarding the container port, but...
krishna chandu
1

votes
0

answer
83

Views

Elasticsearch query on top of aggregation that fetches the last event of each session

I have an Elasticsearch index where each document corresponds to an event. Each event has its own id (used as the document id) and a session id. Multiple events can have the same session id. Question 1: In some use cases I want to perform searches only on top of the last event of each session. This...
Luis Alves
1

votes
1

answer
506

Views

How to use custom Logstash grok patterns?

I'm using Logstash on Debian 9 and I want to use custom grok patterns. So I've added them to directory /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns - created new files and also modified existing (grok-patterns, auth, etc.). Problem is that my changes in or...
Xdg
1

votes
0

answer
59

Views

Elastic Search missing some documents while creating index from another index

I have created one index in elastic search name as documents_local and load data from Oracle Database via logstash. This index containing following values. 'FilePath' : 'Path of the file', 'FileName' : 'filename.pdf', 'Language' : 'Language_name' Then, i want to index those file contents also, s...
Karthikeyan
1

votes
0

answer
25

Views

How to get the unique count number of multi field in elasticsearch

I know single field. { 'aggs': { 'uniq_attr': { 'cardinality': { 'field': 'name.keyword' } } } } I know a clumsy approach { 'aggs' : { 'multi_field_cardinality' : { 'cardinality' : { 'script': 'doc['name.keyword'].value + ' _my_custom_separator_ ' + doc['id_number.keyword'].value' } } } } BU...
CKLogic
1

votes
0

answer
30

Views

Elastic search - Java api to search on multiple fields not giving exact results

Am search my input keyword in multiple field using java api QueryBuilders.multiMatchQuery its not giving me exact results. Its giving some random results which is not relavant to my search query. Please find my code below. SearchRequest contentSearchRequest = new SearchRequest(ATTACHMENT); SearchSo...
Karthikeyan
1

votes
0

answer
162

Views

filebeat/libbeat compiler on gccgo for solaris/sparc

I am driving to compile the project filebeat going for solar / SPARC environments. Officially, Elastic tools are not supported by Solaris. Neamoins, there is circumventenement to this problem / compile via gccgo. I actually prepare my gccgo compiler. I can compile a small project (with a single file...
1

votes
0

answer
91

Views

filebeat.yml.rpmnew or filebeat.yml

I installed recently filebeat, and I would like to edit the yml file to specify this input: filebeat.prospectors: - input_type: log paths: - /path/*.xml # scan_frequency: 60s document_type: message multiline.pattern: '^
C.med
1

votes
0

answer
44

Views

elasticsearch mixed mapping dynamic object

I'm trying to do a mapping in elasticsearch for the attached example Json without luck so far, need your help. the most problematic attribute is the 'info' and the 'info.value'. some time it is empty some time null , int, string .... an example: { 'canArchive': true, 'canDelete': true, 'canUpdate'...
Shay.R
1

votes
1

answer
199

Views

How to log scrapy log into the Logstash

I've set up ELK stack on my server correctly and with using python-logstash I could send my logs to logstash with the following snippet and everything is working correctly. import logging import logstash import sys host = 'localhost' test_logger = logging.getLogger('python-logstash-logger') test_lo...
Yusef Mohamadi
1

votes
0

answer
498

Views

Kibana - reporting option is missing

According to Getting Started page there is an option of Reporting present in Dashboards. But it is not present in any of my dashboards. Here how my top bar looks like : What am I missing? I am using kibana 6.3 version.
sid8491
1

votes
1

answer
33

Views

How can I show specific messages in logs as bars in kibana

I would like to show each bar in Kibana bar chart from a specific message from my application. Bars should be flowing from right to left. There shouldn't be more than 15 bars in the bar chart. Specifically, each microservice in my app will have its name logged in the log file, when it's being requ...
epsan
1

votes
0

answer
31

Views

How can I setup of logstash.conf in a microservice architecture?

I have a microservice based architecture where each microservice is running in a container. I'm in the process of setting up an ELK stack (also in containers). Each microservices' logs end up on the host machine at /log//*.log I'm looking at how to have injest each microservices' logs from that path...
Jas Ahluwalia
1

votes
0

answer
44

Views

How to process the HTML content using Java, within JSON files ingested into ElasticSearch using Logstash?

I am trying to ingest files (articles with a heading, body, etc) into ElasticSearch using Logstash and Kafka. The files are in JSON (fields - title, body, etc) where the body field is in HTML. My ElasticSearch version is 5.1.2. I am using Java REST Client (TransportClient) to interact with Elasti...
Agni Phoenix
1

votes
0

answer
124

Views

Logstash - Setting “config.support_escapes” hasn't been registered

I did follow the description of the official elastic.co documentation when I was configuring the ELK stack. As per the https://www.elastic.co/guide/en/logstash/master/configuration-file-structure.html#_escape_sequences page shows how to set Logstash to enable escape characters, I just uncommented th...
sz3nt
1

votes
0

answer
21

Views

Elasticsearch template unable to read underscore in ids

Here is my query template which is mustache- {\'query\':{\'ids\':{\'type\':\'component\',\'values\':[{{#id_list1}}{{value}}{{#comma}},{{/comma}}{{/id_list1}}]}},\'size\':{{#size}}{{size}}{{/size}}{{^size}}200{{/size}}}{{/id_list}} In values when I passing values such as '1110,1111,1123' or even '111...
Ravinder Baid

View additional questions