Questions tagged [confluent]

1

votes
1

answer
918

Views

Kafka producer difference between flush and poll

We have a Kafka consumer which will read messages and do so stuff and again publish to Kafka topic using below script producer config : { 'bootstrap.servers': 'localhost:9092' } I haven't configured any other configuration like queue.buffering.max.messages queue.buffering.max.ms batch.num.messages I...
shakeel
1

votes
1

answer
218

Views

Kafka Connect JDBC Sink Connector

I am trying to write data from a topic (json data) into a MySql Database. I believe I want a JDBC Sink Connector. How do I configure the connector to map the json data in the topic to how to insert data into the database. The only documentation I can find is this. 'The sink connector requires knowl...
Chris
1

votes
1

answer
544

Views

How do I install mssql jdbc driver to confluent / kafka

I'm trying to create a connector through the rest and it won't work stating that 'Failed to find any class that implements Connector and which name matches com.microsoft.sqlserver.jdbc.SQLServerDriver I'm on ubuntu 16.04 and I've tried the following and each time i try something new i stop then sta...
matthewdaniel
1

votes
0

answer
497

Views

Kafka spends too much time to start reading

Currently I am working with the Confluent kafka nuget package, but do not use the Confluent platform itself (https://github.com/confluentinc/confluent-kafka-dotnet). According to the application needs, it is required to create a consumer, get the first message, and shutdown the consumer. The whole c...
Konstantin
1

votes
2

answer
920

Views

Kafka - ERROR Stopping after connector error java.lang.IllegalArgumentException: Number of groups must be positive

Working on setting up Kafka running from our RDS Postgres 9.6 to Redhift. Using the guidelines at https://blog.insightdatascience.com/from-postgresql-to-redshift-with-kafka-connect-111c44954a6a and we have the all of the infrastructure set up, and am working on fully setting up Confluent. I'm gett...
dragos_kai
1

votes
0

answer
410

Views

Conlfluent 4.0 - Error starting kafka-connect with avro support

I am getting lots of exceptions when trying to start kafka-connect with the command: ./bin/connect-distributed ./etc/schema-registry/connect-avro-distributed.properties With every single jar in the share/java dir (at least), I get exceptions like this, complaining about bad magic number: [2018-01-30...
Randomizer
1

votes
3

answer
272

Views

Kafka JDBC Connect query causes ORA-00933: SQL command not properly ended

I have this Oracle SQL query: SELECT * FROM (SELECT SO_ORDER_KEY,QUEUE_TYPE,SYS_NO, DENSE_RANK() OVER (PARTITION BY SO_ORDER_KEY ORDER BY SYS_NO DESC) ORDER_RANK FROM TSY940) WHERE ORDER_RANK=1; When running in SQL developer, it returns the desired result. For some reason when I use this query in...
Alex
1

votes
1

answer
1k

Views

Confluent Start -> Schema Registry Failed to Start

When I start Confluent, Schema-registry fails, preventing the process from completing successfully. This is the response I get: Starting zookeeper zookeeper is [UP] Starting kafka kafka is [UP] Starting schema-registry Schema Registry failed to start schema-registry is [DOWN] Starting kafka-rest Kaf...
Scicrazed
1

votes
0

answer
453

Views

Confluent's Kafka and Zookeeper Docker Images don't play well on Mac OS X

Trying to run the Docker Images for Kafka & Zookeeper from confluentinc on my Mac. Unfortunately, the commands that the Confluent Docker Quickstart documentation provides utilize net=host which does not work on Mac.. Here are the original docker start-up scripts: Zookeeper: docker run -d \ --net=ho...
Jared Hooper
1

votes
0

answer
448

Views

How to use Schema Registry and AvroConverter in Kafka Source connector?

I'm trying to write custom Source and Sink Kafka connectors for MongoDB with schema registry and Avro. I have configured the key and value converters as AvroConverter and the schema registry url in the properties. What code do i have to add in my connector so that it converts the data to Avro and v...
Chitchat16
1

votes
0

answer
824

Views

Kafka to postgresql using kafka connect jdbc sink connector

I am trying to connect kafka with postgresql. I am using JDBC sink connector. This is my sink.properties file: name=test-sink connector.class=io.confluent.connect.jdbc.JdbcSinkConnector tasks.max=1 topics=orders connection.url=jdbc:postgresql://localhost:5432/casb auto.create=true I have created a t...
el323
1

votes
1

answer
966

Views

Kafka Conenct HDFS sink saving data in parquet format

Using Kafka Connect HDFS Sink I am able to write avro data to Kafka topic and save data in hive/hdfs. I am trying to save data in parquet file format using format class format.class=io.confluent.connect.hdfs.parquet.ParquetFormat quickstart-hdfs.properties are as follows name=hdfs-sink connector.cl...
Madhu
1

votes
0

answer
445

Views

Kafka Message Corruption

We are on confluent platform 3.2.2 (kafka 0.10.2.1-2). We are seeing a large number of Corrupt messages in the logs very frequently. We recently upgraded from 0.8.2 and the frequency of corrupt messages have shot up. We have tried with two different consumers so can confirm that this is indeed a cor...
Vineet Goel
1

votes
0

answer
49

Views

How does Schema Registry integrate with Kafka Source Connector?

I have added Topic-Key and Topic-Value schemas for a given topic using REST APIs. In my custom connector, do I need to create a schema again using SchemaBuilder? How do I access the registered schemas inside my connector ?
Chitchat16
1

votes
0

answer
66

Views

How to reconfigure an existing confluent kafka producer instance in C#?

I have a kafka producer initialized as follows: var config = new Dictionary { { 'bootstrap.servers', BOOTSTRAP_SERVERS } }; Producer _producer = new Producer(config); After the initialization, when I am about to produce a message on a particular topic I want to add compression to the producer (Only...
Midhun Madhavan
1

votes
2

answer
463

Views

How to use send() method with Callback to return metadata?

I want to use callback method for Kafka Producer send() like this: RecordMetadata recordmetadata = kafkaProducer.send(new ProducerRecord(topic, null, timestamp, key, message), this::onCompletion); private RecordMetadata onCompletion(RecordMetadata metadata, Exception exception) { if (exception == nu...
NoName
1

votes
0

answer
67

Views

Segmentation fault while commiting consumer using python confluent kafka

Not able to commit the consumer using confluent kafka. Facing an error 'Segmentation fault', while commiting the consumer. defined consumer with 'enable.auto.commit' = False; default.topic.config {auto.offset.reset:'earliest'} polling the messages with timeout = 50 creating a TopicPartition object u...
Venkat
1

votes
1

answer
438

Views

Root cause for Connection broken for id 1, my id = 3, error =

I am using Confluent 4 for kafka and zookeeper installation. On our Kafka Cluster environment (of 3 brokers and 3 zookeeper nodes running on 3 aws instances) we are seeing a set of below warnings, repeatedly getting recorded in the broker's server.log file. We have not observed any functionality is...
1

votes
1

answer
236

Views

kafka confluent error java.lang.IllegalArgumentException: /tmp/confluent.PVghAKRg/zookeeper/data/myid file is missing

I am running Kafka via a Confluent platform on 3 nodes but when i running confluent start get this error : [2018-04-09 10:54:25,995] INFO Reading configuration from: /tmp/confluent.SVNfiLFU/zookeeper/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2018-04-09 10:54:26,011]...
1

votes
1

answer
216

Views

Kafka failover across regions

If we have a Kafka cluster with replication factor turned on but all brokers in the same region - how do we prevent from a case if a whole region goes down? Is it possible to have some of the brokers in e.g. East Region and some of them in West Region? How much latency affects this? If that is not...
Joe
1

votes
0

answer
353

Views

Read avro messages from kafka to PySpark 2.2

I am using kafka Confluent 4.0.0 to fetch data from SQL Server into kafka topics. I want to read topics data stored in kafka from spark streaming program using below program: from pyspark import SparkContext from pyspark.streaming import StreamingContext from pyspark.streaming.kafka import KafkaUtil...
ashok kumar
1

votes
2

answer
2.3k

Views

Unable to generate avro generic record from object

I am trying to send avro records to A Kafka topic using Kafka producer. I have a User class and I am sending object of that class. Below code works fine if i use avroRecord.put(); to set each attribute. But what i want is to create A Generic Record from an object without using avroRecord.put(); for...
blasteralfred Ψ
1

votes
2

answer
1.8k

Views

How to create a Kafka Topic using Confluent.Kafka .Net Client

It seems like most popular .net client for Kafka (https://github.com/confluentinc/confluent-kafka-dotnet) is missing methods to setup and create Topics. When calling Producer.ProduceAsync() the topic is created automatically but I can't find a way to setup partitions, retention policy and other set...
Michael D.
1

votes
1

answer
269

Views

How to specify Kafka AdminClientConfig (Kafka Admin api) With Trust store and Password

How to Create Kafka AdminClientConfig (Kafka JAVA Admin api) With Trust store and Password. AdminClientConfig has method to specify AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG but How can we specify 'ssl.truststore.location' and 'password' property ? If we use property file how to create AdminClient...
Bharat
1

votes
2

answer
1.1k

Views

Does Kafka broker store metadata?

Does Kafka broker store metadata which producer API uses (e.g. which partitions are leader for a topic etc.)? As per my understanding this metadata is stored in Zookeeper , is it correct? If it is true then how Brokers are updated by Zookeeper with latest information?
Aman
1

votes
1

answer
206

Views

Can we run embedded Kafka broker like ActiveMQ?

When I say embedded this what I mean please refer to following thread - https://stackoverflow.com/a/28858630/5375223 Let me give some context, in our project we are currently using ActiveMQ as messaging broker and now we are planning to migrate to Kafka. Please see the attached picture. Like in abo...
krishna_5c3
1

votes
0

answer
191

Views

Kafka state store rollback on topology exception

I am trying to use a state store to deduplicate messages received by a kafka-streams topology (i.e. duplicate based upon some business-derived deduplication key in case a producer duplicates messages over some long period of time). I notice that if I add a value to the keystore during the transform...
Jon Bates
1

votes
0

answer
146

Views

Can't consume avro data from Confluent Kafka topics to Hbase

I use below sample code to load some test data from Confluent Kafka to Hbase, but somehow I do not see data comes in Hbase. I tried everything but did not work at all. Anyone can help this? How to duplicate the issue: Download the Sample code: https://github.com/mravi/kafka-connect-hbase Compile th...
Cauchy Wu
1

votes
0

answer
458

Views

Sending Byte Array in kafka .Net

I am currently facing a problem with my code below which I have written for Kafka. The code is quite simple, the Producer produces a byte array and the consumer wants to consume the array. The problem with it is that it neither displays any error, nor I am able to get the required output. I am sure...
Kenil44
1

votes
1

answer
61

Views

Kafka (Confluent Platform) input for Logstash - broken message encoding

I have a Confluent Platform (version 4.1.1). It is configured to read data from the database. The configuration for this is: name = source-mysql-requests connection.url = jdbc:mysql://localhost:3306/Requests connector.class = io.confluent.connect.jdbc.JdbcSourceConnector connection.user = *** connec...
Nikita Lipatov
1

votes
2

answer
60

Views

How is schema from Schema-Registry is propagated over Replicator

How do schemas from Confluent Schema-Registry get propagated by Confluent-Replicator to destination Kafka-Cluster and Schema-Registry? Is each replicated message schema contained in it or are schemas replicated somehow separately through a separate topic? I didn't see any configuration possibilities...
Neven
1

votes
1

answer
1.2k

Views

JDBC Kafka Connect with DB2

I'm struggling to get Confluent's kafka connector to connect to DB2. I am running an ubuntu instance inside docker for testing pruposes. The solution needs to be deployed to kubernetes, so docker it is. I have installed the Confluent platform using apt-get and adding their repos. All services are ru...
OzzMaster
1

votes
0

answer
282

Views

WARNING : createConnector contains empty path annotation

Using documentation of confluent for kafka_cluster I got this error : Jul 03, 2018 4:20:41 PM org.glassfish.jersey.internal.Errors logErrors WARNING: The following warnings have been detected: WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.Connec...
Wiem Bel Hadj
1

votes
0

answer
180

Views

How to avoid WARN messages about ReflectionsException

how can I avoid WARN messages to be displayed in logs (without putting the log4j level to ERROR) when I launch Confluent ? I have set up my plugin.path variable in the properties file with value ${CONFLUENT_HOME}/share/java/kafka-connect-jdbc (with final comma). I tried to put in the classpath the k...
Diego T
1

votes
0

answer
139

Views

Confluent go kafka library starting from earliest offsets on restart

We currently use Kafka 0.8.2 for our brokers. We use the .Poll() method to grab messages and commit after we collect 1000 messages. We are able to consume from the cluster fine using this library, and I can see that the offsets are being stored in the __consumer_offsets topic from a manual commit ca...
Conor D'Arcy
1

votes
0

answer
285

Views

Is it possible to use multiple left join in Confluent KSQL query? tried to join stream with more than 1 tables , if not then whats the solution?

Stream : describe ammas; Field | Type ------------------------------------- ROWTIME | BIGINT (system) ROWKEY | VARCHAR(STRING) (system) ID | INTEGER ------------------------------------- For runtime statistics and query details...
1

votes
1

answer
224

Views

ERROR: using kafka-connect-cdc-mssql:1.0.0-preview

I try read the log transaction with Kafka Connector, with connector kafka-connect-cdc-mssql:1.0.0-preview I'm using the Confluent CLI and I have already modified the configuration file: etc/schema-registry/connect-avro-standalone.properties I add : plugin.path=/usr/share/java,/tmp/Softwares/conflue...
1

votes
0

answer
329

Views

Kafka JDBC-Sink connector not working as intended

I a trying to use the JDBC sink connector to put data into Postgres, however, I am not seeing any data created in my database. The connector config I am using it { 'name': 'Test-Insert', 'config': { 'connector.class': 'io.confluent.connect.jdbc.JdbcSinkConnector', 'tasks.max': '1', 'connection.url':...
Tom Halson
1

votes
2

answer
155

Views

Dynamic topic in kafka connector

kafka added new feature to use regex in connectors, however it seems that the topic data from the newly added topics after the connector has been started is not consumed until the connector is restarted. We have a need to dynamically added new topic and have connector consume the topic based on rege...
Pratik Gaglani
1

votes
0

answer
260

Views

Unable to sink messages from a Kafka consumer to Postgres

i am able to read messages from a kafka topic using kafka-console-consumer /bin/kafka-console-consumer --topic Test --consumer.config /etc/kafka/consumer.properties --from-beginning --bootstrap-server ip1:9095,ip2:9095,ip3:9095 but unable to sink these messages to a table in postgres using jdbc-s...
fangorn

View additional questions