Questions tagged [druid]

1

votes
0

answer
350

Views

Metabase: Date Range filter to dashboards created by native query

I use Metabase for data visualization. Druid (imply-2.2.3) is data storage. Created Metabase Questions I put on the dashboard and filter them all by Date Range. When I try to add Date Range filter to question created by native query, metabase just can't find timestamp field for filtering and says: '...
Bo.
1

votes
0

answer
249

Views

SuperSet configuration with SSL DRUID

I am newbie on superset and trying to configure DRUID .. successfully start superset and saw the examples .. but when I tried to connect druid (it has only https connectionavailable for me ) .. I got error message when I tried to [[email protected]]# superset refresh_druid 2018-02-21 23:08:05...
Larry
1

votes
2

answer
227

Views

Is there a way that i can push historical data into druid over http?

I have an IOT project and want to use Druid as Time Series DBMS. Sometimes the IOT device may lose the network and will re-transfer the historical data and real-time data when reconnecting to the server. I know the Druid can ingest real-time data over http push/pull and historical data over http pul...
tz_imu
1

votes
2

answer
419

Views

how to make superset display druid data?

I have been trying to have superset display data from druid, but was unable to succeed. In my druid console I could clearly see a 'wiki-edits' data source, but, when I have specified druid cluster and druid data source in superset, it did not pick up any of that data. Have anyone been able to make t...
Eugene Goldberg
1

votes
1

answer
216

Views

what caused druid tasks failed

I had set up druid cluster(10 nodes),ingestion kafka data using indexing service.However,I found many of tasks are failed like below,but some data had been existed in segments,I am not sure if all datas are pushed in the segments. failed task lists besides that,I choose some logs of failed tasks,fou...
Zhang Xin
1

votes
1

answer
151

Views

why kafka ingestion service of druid cannot have 2 datasources

I used druid indexing service to ingest kafka data, when I created one datasource, it works well. But when I added another datasource, there are no tasks to assign for the second datasource.How should I do make all datasources can work well?there are 2 datasources below. But running tasks are all ab...
Zhang Xin
1

votes
0

answer
69

Views

Error while query druid using sql like in ()

Query druid using the sql below SELECT CASE WHEN sale_mode in (1,2,3) THEN 'a' ELSE 'b' END AS type FROM '20051001' WHERE __time>'2018-04-22 00:00:00' and return the following error message Bad Request: Unknown exception: Error while applying rule DruidQueryRule:SELECT_PROJECT, args [rel#103796215...
Adam Lyu
1

votes
2

answer
101

Views

How to write integration tests depending on Druid?

I am coding an application that generates reports from a Druid database. My integration tests need to read data from that database. My current approach involves creating synthetic data for each of the tests. However, I am unable to remove data created from the database (be it by removing entries or...
1

votes
0

answer
46

Views

Loading of Apache Avro plugin for Tranquility fails with Exception

For the Kafka Avro producer I run : ./kafka-avro-console-producer --broker-list localhost:9092 --topic pageviews --property value.schema='{'type':'record','name':'mypageviews','fields':[{'name':'time','type':'string'},{'name':'url','type':'string'},{'name':'user','type':'string'},{'name':'latencyMs'...
Saeed Mohtasham
1

votes
1

answer
207

Views

Setting up basic authentication in druid

I want to set up an authentication for Druid. I followed the example to set up druid from http://druid.io/docs/latest/tutorials/quickstart.html Once done, I followed this document to setup authentication: http://druid.io/docs/latest/development/extensions-core/druid-basic-security.html These are the...
Juvenik
1

votes
0

answer
61

Views

Why Druid cluster crashed for indexing task?

Can anyone tell me what & why cause my Druid cluster crashed during run indexing service? 2018-06-15T11:20:30,005 ERROR [forking-task-runner-1] com.google.common.util.concurrent.ExecutionList - RuntimeException while executing runnable [email protected] with execut...
Casel Chen
1

votes
2

answer
246

Views

Time-series charts for large amounts of data

I have a couple of thousand time-series covering several years at second-granularity. I'd like to store the data in a suitable DB (i.e. one that scales well and can retain all data at original granularity, e.g. Druid, openTSDB or similar). The goal is to be able to view the data in a browser (e.g....
MrFancypants
1

votes
1

answer
348

Views

Resource limit exceeded in druid groupBy Query

I am trying to run groupBy Query above the limit of 500k data. I am getting this error. { 'error': 'Resource limit exceeded', 'errorMessage': 'Not enough dictionary space to execute this query. Try increasing druid.query.groupBy.maxMergingDictionarySize or enable disk spilling by setting druid.query...
Salman S
1

votes
1

answer
35

Views

Whats does 'granularity' affects in druid's select query?

As it says on druid doc, Select queries return raw Druid rows, so what does granularity mean in select query? In my opinion, select dones't need this argument since it returns raw rows.
user9573581
1

votes
2

answer
75

Views

How do I limit the size of log file generated by druid while using imply?

I'm using imply to handle druid's cluster. But my logs files have increased to hundreds of gigabytes of storage. I'm talking about logs files present in imply/var/sv/ directory in which there are these 7 log files, broker.log, historical.log, middleManager.log zk.log, coordinator.log, imply-ui.log,...
Point Networks
1

votes
1

answer
63

Views

Druid Postgresql syntax error while Initializing lookups on coordinator for first time

I am trying to setup lookups on a cluster-wide druid. As mentioned in the doc- http://druid.io/docs/latest/querying/lookups.html, 1. Have included 'druid-lookups-cached-global' in my load list. 2. I tried to post an empty json object to initialize the configuration. Command I ran: curl -X 'POST' -...
simba
1

votes
0

answer
51

Views

Ingestion tasks failing with IllegalArgumentException druid

We have been experiencing this issue but couldn't seem to find the root cause of it. All the indexing tasks seems to be failing Logs: 2018-11-09T15:45:10,861 ERROR [task-runner-0-priority-0] io.druid.indexing.common.task.MergeTaskBase - Exception merging[test-requests]: {class=io.druid.indexing.com...
sn879
1

votes
1

answer
76

Views

Druid parquet poor ingestion performance

Are there any reasons for Druid ingestion to be slow when ingested with parquet data? We have observed that the ingestion speed is atleast 2 times better when JSON data was ingested. In general, which one is better? JSON or Parquet in the context of Druid Hadoop batch ingestion.
Vijay Muvva
0

votes
1

answer
26

Views

SQL: How to get the sum of all strings in rows of sentences of a column

| random_column | |---------------------------------------------------------------| | The magnetic quality of a sample of iron depends on the purity| | A sample is taken from each bale separately. | | I love Switzerland...
Pavitran
1

votes
2

answer
789

Views

DRUID.io vs Esper CEP

I have been reading a little about Druid last few days and wondering how is this different from Esper. I have been using Esper for realtime event processing. It feels Druid does that with its simple JSON like queries and much simpler interface. Can someone please correct me and share more light? Edi...
change
1

votes
1

answer
256

Views

OLAP Realtime querying for huge volume data - options

We have an OLAP table with 180 billion rows and 100+ columns, and the volume is close to 8TB in Hive. Most of the columns are dimensions and also we have few metrics columns also. We would like to build a real-time system support ad-hoc queries to run the dashboard applications, where the queries s...
sureshsiva
1

votes
1

answer
317

Views

Inserting 1 million+ rows/second into a database on Druid/Cassandra [closed]

We have a huge influx of data, around 8 million rows in 1 second. Each row has 12 numeric values(timestamp, integers and floats, but no text). I'm looking for Streaming inserts, but we can manage with Bulk inserts if it can achieve the speeds. Queries: Our queries are basically aggregations. I have...
user1471299
1

votes
1

answer
358

Views

How to read all data from druid datasource

I am using below json to read all data from a druid datasource. But in the request threshold field/value is mandatory. It returns number of rows specified in threshold. { 'queryType' : 'select', 'dataSource' : 'wikiticker', 'granularity' : 'day', 'intervals' : [ '1000/3000' ], 'filter' :null, 'dimen...
sidnakoppa
1

votes
2

answer
680

Views

Apache Kafka Consumer - Analytics Platform - Visualize data

I am new to Apache Kafka and also to data analytics. I am able to consume messages from Apache kafka consumer in Java. Now I want to take these real time data and display it like a dashboard. I want to visualize all these data using any open source tool. One tool which I found is Druid but the do...
Pavithra K C
1

votes
1

answer
265

Views

Connect datasource with proxy

For some reason, I have to connect mysql with socks proxy during the development, and the program uses druid as datasource. I've tried to add vm option -Dhttp.proxyHost=host -Dhttp.proxyPort=port in idea's run/debug configurations, and set JAVA_OPTS='$JAVA_OPTS -Dhttp.proxyHost=host -Dhttp.proxyPor...
NeB Nep
1

votes
1

answer
721

Views

Cannot write to Druid through SparkStreaming and Tranquility

I am trying to write results from Spark Streaming job to Druid datasource. Spark successfully completes its jobs and hands to Druid. Druid starts indexing but does not write anything. My code and logs are as follows: import org.apache.spark._ import org.apache.spark._ import org.apache.spark.strea...
Gofrette
1

votes
2

answer
1.8k

Views

Error while extracting DatabaseMetaData; nested exception is com.mysql.jdbc.exceptions.jdbc4. No operations allowed after connection closed

I have a scheduled task running every 20 minutes which was configured through '@Scheduled(cron = '0 0/20 * * * ?')' based on spring boot. During this task, I need to search some data from mysql and then save them into redis. The following are my configure of database: mybatis.configuration.map-under...
taichi_tiger
1

votes
1

answer
490

Views

alter table/add columns in non native table in hive

I created a hive table with a storage handler and now I want to add a column to that table but it gives me below error: [Code: 10134, SQL State: 42000] Error while compiling statement: FAILED: SemanticException [Error 10134]: ALTER TABLE can only be used for [ADDPROPS, DROPPROPS] to a non-native...
Jainik
1

votes
1

answer
187

Views

How to Handle Rows that Change over Time in Druid

I'm wondering how we could handle data that changes over time in Druid. I realize that Druid is built for streaming data where we wouldn't expect a particular row to have data elements change. However, I'm working on a project where we want to stream transactional data from a logistics management sy...
Clay Crosby
1

votes
1

answer
84

Views

Can I create a Druid lookup using time as a dimension?

I am able to build lookups like { 'type':'lookup', 'dimension':'type', 'outputName':'type_name', 'outputType': 'STRING', 'retainMissingValue':true, 'lookup':{'type': 'map', 'map': {'0': 'Unknown', '1': 'Mobile(Other)', '2': 'Desktop/Notebook', '3': 'Connected/Smart TV', '4': 'Mobile Phone'}, 'is...
1

votes
1

answer
127

Views

Druid Kafka ingestion with read-your-writes

I'm learning Druid now. I read that ingestion via Kafka Indexing Service guarantees exactly-once semantics. However, I have a problem with determining consistency model of Druid. Typically streams are asynchronous, but I want to have read-your-writes semantics in application. Is there any possibilit...
T. Gawęda
1

votes
1

answer
173

Views

Druid count differ when we run same query on daliy and row data

When run query to ABS Data Source in Druid.I got some count but that differ when same query run with ABS_DAILY data source. And we make ABS_DAILY from ABS. { 'queryType' : 'groupBy', 'dataSource' : 'ABS', 'granularity' : 'all', 'intervals' : [ '2018-07-12T00:00:00.000Z/2018-07-13T00:00:00.000Z' ],...
Sudhanshu Lenka
1

votes
1

answer
122

Views

Percentage calculation in Druid

I have a data in the following format. I am having a hard time doing profit percentage calculation of a particular item or brand w.r.t to all products on a particular day. Date Item Brand Profit 15-08-2019 A Nike 5 15-08-2019 B Nike 10 15-08-2019 C Nike 12 15-08-2019 D N...
Abhishek Kumar
1

votes
1

answer
999

Views

java.sql.SQLException: sql injection violation, deny object : dbms_random

As the title, I get a SQLException when I use dbms_random function like this: SELECT DBMS_RANDOM.value(1, 100) AS datas FROM dual but the SQL is correctly executed in PL/SQL. Besides, the connection pool which I use is Druid. Exception Stack as follow: 15-Sep-2014 11:02:04.397 SEVERE [http-apr-8080-...
ArenaGod
1

votes
2

answer
280

Views

Druid with Kafka Ingestion: filtering data

is it possible to filter data by dimension value during ingestion from Kafka to Druid? e.g. Considering dimension: version, which might have values: v1, v2, v3 I would like to have only v2 loaded. I realize it can be done using Spark/Flink/Kafka Streams, but maybe there is an out-of-the-box solutio...
pcejrowski
0

votes
0

answer
3

Views

Writing Custom Extensions in Druid

I am new to Druid. Problem Statement We do currently push raw event data to Druid. I have a requirement to apply certain calculations on the data (say like certain stat techniques) which are not supported by Druid or the extensions it provides out of the box. There are two questions I have - What w...
Uno
1

votes
1

answer
820

Views

Java client with Apache HttpClient to connect to Druid

I am working on ingesting and query data on Druid Server. But, when I query I just using the command line as below: curl -X 'POST' -H 'Content-Type:application/json' -d @quickstart/ingest_statistic_hourly_generate.json localhost:8090/druid/indexer/v1/task Can anyone tell me the way of utilizing Jav...
VanThaoNguyen
1

votes
3

answer
637

Views

Registered lookup not working in Druid

I'm working with druid for a short time now and I'm testing the registered lookup functionality. I've already created the lookup under the http://:/druid/coordinator/v1/lookups, as following: { '__default': { 'home_post_code': { 'type': 'map', 'map': {'13210': 'Syracuse, NY'} } } } As far as I under...
Noel Ferreira
1

votes
2

answer
773

Views

Druid - Order data by timestamp column

I've set up a Druid cluster to ingest real-time data from Kafka. Question Does Druid support fetching data that's sorted by timestamp? For example, let's say I need to retrieve the latest 10 entries from a Datasource X. Can I do this by using a LimitSpec (in the Query JSON) that includes the timesta...
jithinpt
1

votes
1

answer
58

Views

Added io.druid dependency breaks Glassfish deployment

My problem looks similar to this one but I already use Glassfish 4.1.13 I try to add druid-client to my Glassfish project. I added druid-client as separate module. pom.xml of druid-client includes following Druid dependency: io.druid druid-server 0.9.1.1 Version of Glassfish: 4.1.13 Also I use maven...
Bo.

View additional questions