Questions tagged [logstash]

0

votes
0

answer
4

Views

Logstash.service not-found by systemctl

I'm trying to set ElasticStack but I had problems with Kibana and Logstash, I could solve Kibanas issue, but after some days, my CentOS 7 machine don't recognise logstash.service. I think that this is not a logstash issue, instead is a CentOS7 or a Systemd issue. How do you think I can make my logs...
Hikari Olsen Berg
1

votes
2

answer
920

Views

Unable to connect docker container to logstash via gelf driver

Hi guys i'm having trouble to send my server container logs to my ELK stack. No input is sent to logstash so i'm unable to set kibana index for collecting logs. I think my problem is in the port settings. Here is the docker-compose yml for the LAMP stack (only the server service): version: '3' servi...
1

votes
1

answer
234

Views

What are benefits of running ELK stack on Docker over running it on VM

I'm learning ELK stack. I wonder, why would people run it on Docker? If I understand everything correctly, it would have to have some directory of a host OS mapped to be persistent over resets of the image. Meanwhile, running just VL with installed docker would be persistent anyway. Why should I use...
Kowalski Paweł
1

votes
2

answer
673

Views

How to add newline with gsub in logstash

I have a field 'message'. When I do this : mutate { update => {'message' => '%{message} \n'} } It converts message to: '2017-12-31 Error \\n' ['2017-12-31 Error' is my message value] What I want is '2017-12-31 Error \n' Also tried to figure out using below links, but could not find a solution: Add...
S'chn T'gai Spock
1

votes
0

answer
75

Views

Moving nested json to top level in logstash

My log has structure like - { a: { b: { c: 12 } } z: 514 } I want to convert it to { z:514, b: {....}} When I try to add_field it can add only stringified JSON. I have also tried rename and copy but none seems to work for me. What is the correct way to do it ?
Krrish Raj
1

votes
1

answer
50

Views

Disabling authorisation or allow any snmp community string in snmp input trap plugin

SNMP input trap plugin assumes community string to be present or if we don't provide it takes 'public' by default. As per the docs, community string can be array also. I want to support any community string. SNMP input trap plugin doesn't support disabling authorization also. How can I make it work...
Ajinkya Dhote
1

votes
0

answer
33

Views

Snmp tran input plugin

How can I make the community to accept all the string. By default it accepts only 1 or array, I we don't provide then it take 'public'. I want it to have no restriction n the community string.
Sunil Hiray
1

votes
2

answer
293

Views

Cast or convert mongodb _id object to string in Logstash pipeline

I am working on creating a pipeline to get data from MongoDB to ElasticSearch using Logstash. I am using dbschema mongodb jdbc drivers. I am able to connect to database using driver but I am facing issue with _id . As in MongoDB its of type object So I am getting issue with converter. Here is error...
Mahajan344
1

votes
0

answer
160

Views

Logstash with elasticsearch input and output keep looping results

I would like to reindex and filter my log again. What I get the information from Internet is using the logstash to filter the data again. I tried and it can really split my data into different fields, however, the data keeps looping. That is, I have 100,000 logs but after filtering and output to ela...
Layla
1

votes
1

answer
348

Views

Logstash, how to use grok patterns coming from event data

I have an ELK stack deployed on kubernetes used to collect containers' data. Among all the rest, it is using a grok filter to parse the actual log line based on a pattern. My wish is to be able to setup this pattern by using an annotation in the kubernetes pod. I added an annotation called elk-grok-...
whites11
1

votes
1

answer
278

Views

dockerized logstash is generating huge log files

I'm running a logstash container in AWS ECS from the following image docker.elastic.co/logstash/logstash:5.5.3 Everything is at default and I am not using the stdout plugin. But logstash still outputs all the log items to stdout and the container is generating a huge log file at /var/lib/docker/con...
lingxiao
1

votes
1

answer
160

Views

LogStash Grok regex backreferences

I'm really hoping I'm doing something silly and just can't see the problem... this would be trivial in Perl or other languages. Apparently backreferences are supported in grok https://grokconstructor.appspot.com/RegularExpressionSyntax.txt, but I can't make them work. I need to match on something...
Aaron
1

votes
1

answer
286

Views

Visualizing pdf data on kibana

I have ELK(Logstash, Elasticsearch, Kibana) stack setup working. Visualizations are created using CSV files which are loaded using logstash to kibana. But, I have indexed the PDF, DOC files using elasticsearch and able to see the data on kibana and can search in them. But, I need to visualize the t...
monty
1

votes
2

answer
1.3k

Views

logstash convert string to date

I want to convert string to date in logstash. I tried to do it with 2 solutions but it doesn't work. 1) with mutate/convert grok {match => {'message' => '%{TIMESTAMP_ISO8601:log_date} - %{GREEDYDATA:key}:%INT:value}'} mutate {convert => ['log_date', 'date']} 2) with date/target : date {match => [ '...
Papiis
1

votes
0

answer
148

Views

Do I use Filebeat, Ingest or Pipelines to get rid of Logstash in my ELK stack?

I'm fairly new to filebeat, ingest, pipelines in ElasticSearch and not sure how they relate. In my old environments we had ELK with some custom grok patterns in a directory on the logstash-shipper to parse java stacktraces properly. The logstash indexer would later put the logs in ES. How do I do th...
Dennis
1

votes
1

answer
70

Views

LogStash - parsing logs

I am trying to parse my log files. It was working however there is a new requirement and I need new field from the parsed logs. Logs are more or less: |2018-02-01 13:48:00.882|[v2.8.0.0]|DEBUG|[EndpointFirst] |Session activated (documentId: 508, workflow id: 1)| |2018-02-01 13:48:00.901|[v2.8.0.0]|I...
niao
1

votes
0

answer
178

Views

Getting logstash to run on Windows

I downloaded the zip version of the logstash from LogStash Download Link I unziped it and ensured the folder path does not have any space. Also I ensured that the JDK 64bit is available in the path. But when I try to start logstash, I am getting 'could not find jruby in C:\ELK\logstash-6.1.3\vendor...
vinSan
1

votes
1

answer
301

Views

C++ : How to use SocketAppender of log4cplus to send logs to logstash server?

I'm trying to send logs of my C++ application to logstash using log4cplus library. I have read the log4cplus documentation and used below configurations to configure SocketAppender. log4cplus.rootLogger=INFO, SA log4cplus.appender.SA=log4cplus::SocketAppender log4cplus.appender.SA.port=5044 log4cp...
Durgesh
1

votes
1

answer
351

Views

logstash connection to database issue

Using logstash, what I'm trying to do is dump all tagnames from a table in a database to an index. The problem I'm facing here is, the logstash works fine if specify the IP address as 127.0.0.1 for the postgresql connection. But if I specify my actual IP or some other user's IP i'm not getting the c...
Aravind S
1

votes
3

answer
398

Views

Regular expressions for grok pattern

I was facing issues while trying to print second string (only) from below line. Input: Severity: error And the output i am expecting is, error. Can someone please help? I am new to regex and tried many options and somehow i arrived at this after trimming all other stuff from the line and stuck here.
Hazi
1

votes
0

answer
196

Views

Logstash create elasticsearch document with array as property and put new element to array

I have a document with some children inside, something like: parent : { 'id' : 1, 'name' : 'test', 'children' : [ { 'idchild' : 1, 'name' : 'c1'}, {...} ] } I want to add the children to the already existent parents. I have a logstash file to read all the parents. It's write the document in elastics...
Daniele Licitra
1

votes
0

answer
663

Views

java.lang.OutOfMemoryError: Java heap space when transferring data from jdbc to elasticsearch via logstash

I have a huge postgres database with 20 million rows and i want to transfer it to elasticsearch via logstash . I followed the advice mentioned here and I test it for a simple database with 300 rows and all things worked fine but when i tested it for my main database i allways cross with error: nar...
Fatemeh Moh
1

votes
2

answer
301

Views

Elasticsearch Mapping Template

ELK Stack version 6.2.1 I am following this tutorial and trying to send tshark captures to ELK. https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana As tshark captures all fields as text, I am trying to create a mapping in elasticsearch to ensure number field...
Bat
1

votes
0

answer
56

Views

Logstash-5.6.0 and Elastic Search-6.2.1

I have the below configuration in logstash.conf, Started my logstash with the following command ` logstash --verbose -f D:\ELK\logstash-5.6.0\logstash-5.6.0\logstash.conf` and Elastic search is running at 9200 port but logstash is not pipelining the parsed log file contents into elastic search. did...
Karthik Suresh
1

votes
0

answer
56

Views

Logstash service failure CentOS7 - Some newbie questions

I am really struggling to launch logstash as a service on CentOS 7. Since I cannot figure out what or where to set the -DJava.io.tmpdir= variable (which apparently would solve my issue), I am trying to create a little script to launch the logstash command line on boot. The following line works manua...
Quizzy Rascal
1

votes
1

answer
82

Views

Refer to objects in different index - Elasticsearch

I have a zip-code table with list of zipcodes mapped to state. In state table, I have state name, state website, key locations etc. I am planning to store these two table values in elastic search populated using logstash. My ES queries would be based on zipcode and result should return corresponding...
lpk
1

votes
1

answer
213

Views

specify elasticsearch index alias in template file

I want to create index alias in template file, I have specified index name as 'test_2017_12_02' in logstash conf file, my template is as below 'aliases' : {'test_2017_12_02' : 'test'} but not working, the index getting created without alias
arjunsv3691
1

votes
0

answer
74

Views

Logstash Persistent Queues Not Creating Tail Files

I have just started playing with logstash 5.4.0 persistent queues. I have configured logstash to use persistent queues though this always writes to head and never rolls the head over to tail. My logstash.yml is as follows queue.checkpoint.writes: 1 queue.type: persisted path.queue: /usr/share/log...
SimonDawe
1

votes
0

answer
247

Views

Extract Filebeat prospector path regexp match to field

I wonder if there's anyway to extract regexp match of Filebeat prospector path to a field, for ex. something like: filebeat.prospectors: - type: log enabled: true paths: - /var/logs/apps/[(a-z)]/*.log json.keys_under_root: true json.add_error_key: true json.message_key: log fields: log_topic: 'app-$...
Idan Gozlan
1

votes
0

answer
58

Views

Indexing with Logstash to Elasticsearch

I am working on an Elasticsearch indexing task. Currently we are indexing hundreds of thousands of documents to ES cluster (many ES instances) on daily basis. We simply read data from DB and various of data sources, collate and compile them and directly index them to ES using python elasticsearch-ds...
hevi
1

votes
1

answer
273

Views

Logstash - How to filter different types logs file coming through filebeat

I've two log files on my remote host sample.log and example.log, both of them contains different pattern logs. How can I easily use different filter for each log type in Logstash? Can someone give me an example please. I am using filebeat to ship these logs to Logstash. My Logstash.conf file is some...
Rao
1

votes
1

answer
251

Views

sentiment analysis - elastic stack

I am using the newest version of Elastic stack (Logstash, Elasticsearch, Kibana) to perform some Twitter analysis and I would like to add a sentiment analysis to it (basic one is fine, nothing too complicated) however all options I found were using libraries like tweepy to input data into elasticsea...
Angelika
1

votes
0

answer
25

Views

Get data by Popularity/Rating using Elasticsearcch

I am trying to do a get request using elasticsearch which needs to get the data with respect to its Popularity/ rating. So I followed this Link . I set the rating of my item by using the below one, #1 http://localhost:9200/cars/car/_rank_eval above is the Api which is used to create _rank_eval usi...
Beckham_Vinoth
1

votes
0

answer
112

Views

Logstash remove duplicates in array

I've got an array that can contain duplicate integers. Is there a Logstash plugin that would remove duplicates and keep just distinct values? I know you can write a Ruby script to do it, but I'm curious if there's something out of box already
1

votes
0

answer
124

Views

Kibana setup on Ubuntu 17.10 for consuming log files from Jboss Fuse

Every day I get a new log file from Jboss Fuse. Examples: fuse.log.2018-02-28 fuse.log.2018-03-01 fuse.log.2018-03-03 etc.. I want to load a log file into Kibana every day. Now this is what I have done so far: Installed Elasticsearch Installed ingest-geoip Installed Kibana on http://localhost:9200...
Sigma
1

votes
1

answer
76

Views

date format convertion in logstash elk stack

i have a date column in my table that i fetch using jdbc input in logstash.the problem is logstash gives a wrong value to elasticsearch stack. for example if i have a date start_date='2018-03-01' in elasticsearch i would get the value '2018-02-28 23:00:00.000'. What i want is to keep the format of s...
user1655410
1

votes
0

answer
85

Views

Regex in S3 input plugin

I neeed to get prefix in S3 input by using Regex, but there is no support for Regex in the doc. Folders list like this: logs/20180304/app1/app1.log logs/23180304/app2/app2.log logs/20180305/app1/app1.log logs/23180305/app2/app2.log logs/20180306/app1/app1.log logs/23180306/app2/app2.log my input plu...
khaja mohiddin
1

votes
0

answer
1.7k

Views

The final mapping would have more than 1 type Error - Intermittent

I am currently observing the below error. Strangely, it worked for the first time, later on I had to an additional field (Tag) to my data and that was the only change I have done and now i get the mapping error. I was not able to figure out the reason behind it and now when i remove the field i adde...
sdgd
1

votes
0

answer
47

Views

logstash: clones with ruby script

I want to clone some events in a logstash pipeline, several times using the corresponding plugin. So far I have been using an approach like the following: (this is from my logstash.conf) clone { clones => [1,2,3,4,5,6] } Is there a way to this in a more elegant manner to do this programatically usin...
pkaramol
1

votes
0

answer
397

Views

how to connect mongo with elasticsearch using logstash?

I want to connect mongodb with elasticsearch. I have used logstash-input-mongodb plugin, but this doesn't take my elasticsearch data updates. So when i update a document at mongodb it doesn't change at elasticsearch. I have searched through web but couldn't find a way to fix it? Is it a bug? Which...
Edit Axha

View additional questions