Questions tagged [apache-kafka-connect]

1

votes
1

answer
218

Views

Kafka Connect JDBC Sink Connector

I am trying to write data from a topic (json data) into a MySql Database. I believe I want a JDBC Sink Connector. How do I configure the connector to map the json data in the topic to how to insert data into the database. The only documentation I can find is this. 'The sink connector requires knowl...
Chris
1

votes
1

answer
491

Views

Kafka Connect HDFS Sink with Azure Blob Storage

I want to connect to Azure Blob Storage with Kafka HDFS Sink Connector. So far I have done: Set kafka-connect properties: hdfs.url=wasbs:// hadoop.conf.dir={hadoop_3_home}/etc/hadoop/ hadoop.home={hadoop_3_home} And in core-site.xml added support for wasbs: fs.wasbs.impl org.apache.hadoop.fs.azure.N...
zhandosso
1

votes
0

answer
310

Views

How to read kafka statestore in processor which was updated from a different processor?

I have created 2 kafka statestores corresponding to 2 different topics as below. StateStoreSupplier tempStore1 = Stores.create('tempStore1').withKeys(Serdes.String()).withValues(valueSerde).persistent().build(); StateStoreSupplier tempStore2 = Stores.create('tempStore2').withKeys(Serdes.String()).wi...
Nar123
1

votes
2

answer
920

Views

Kafka - ERROR Stopping after connector error java.lang.IllegalArgumentException: Number of groups must be positive

Working on setting up Kafka running from our RDS Postgres 9.6 to Redhift. Using the guidelines at https://blog.insightdatascience.com/from-postgresql-to-redshift-with-kafka-connect-111c44954a6a and we have the all of the infrastructure set up, and am working on fully setting up Confluent. I'm gett...
dragos_kai
1

votes
0

answer
410

Views

Conlfluent 4.0 - Error starting kafka-connect with avro support

I am getting lots of exceptions when trying to start kafka-connect with the command: ./bin/connect-distributed ./etc/schema-registry/connect-avro-distributed.properties With every single jar in the share/java dir (at least), I get exceptions like this, complaining about bad magic number: [2018-01-30...
Randomizer
1

votes
1

answer
895

Views

Kafka connection transformations. Parse input string and get a record key

I use a simple file source reader connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector tasks.max=1 File content is a simple JSON object in each line. I found that there is a way to replace a record key and use transformations to do this, like # Add the `id` field as the key using...
Sergei Grigorev
1

votes
3

answer
272

Views

Kafka JDBC Connect query causes ORA-00933: SQL command not properly ended

I have this Oracle SQL query: SELECT * FROM (SELECT SO_ORDER_KEY,QUEUE_TYPE,SYS_NO, DENSE_RANK() OVER (PARTITION BY SO_ORDER_KEY ORDER BY SYS_NO DESC) ORDER_RANK FROM TSY940) WHERE ORDER_RANK=1; When running in SQL developer, it returns the desired result. For some reason when I use this query in...
Alex
1

votes
2

answer
264

Views

Does scaling Kafka Connect is same as scaling Kafka Consumer?

We need to pull data from Kafka and write into AWS s3. The Kafka is managed by separate department and we have access to only specific topic. Based on Kafka documentation it looks like Kafka Connect is easy solution for me because I don't have any custom message processing logic. Normally when we ru...
1

votes
0

answer
448

Views

How to use Schema Registry and AvroConverter in Kafka Source connector?

I'm trying to write custom Source and Sink Kafka connectors for MongoDB with schema registry and Avro. I have configured the key and value converters as AvroConverter and the schema registry url in the properties. What code do i have to add in my connector so that it converts the data to Avro and v...
Chitchat16
1

votes
0

answer
824

Views

Kafka to postgresql using kafka connect jdbc sink connector

I am trying to connect kafka with postgresql. I am using JDBC sink connector. This is my sink.properties file: name=test-sink connector.class=io.confluent.connect.jdbc.JdbcSinkConnector tasks.max=1 topics=orders connection.url=jdbc:postgresql://localhost:5432/casb auto.create=true I have created a t...
el323
1

votes
1

answer
966

Views

Kafka Conenct HDFS sink saving data in parquet format

Using Kafka Connect HDFS Sink I am able to write avro data to Kafka topic and save data in hive/hdfs. I am trying to save data in parquet file format using format class format.class=io.confluent.connect.hdfs.parquet.ParquetFormat quickstart-hdfs.properties are as follows name=hdfs-sink connector.cl...
Madhu
1

votes
0

answer
49

Views

How does Schema Registry integrate with Kafka Source Connector?

I have added Topic-Key and Topic-Value schemas for a given topic using REST APIs. In my custom connector, do I need to create a schema again using SchemaBuilder? How do I access the registered schemas inside my connector ?
Chitchat16
1

votes
0

answer
182

Views

Kafka Connect from RDS to RedShift not starting

I was able to implement Kafka Connect on a much smaller table but am trying to implement it on a larger database. My source and sink configuration are as followed source: name=rds-source connector.class=io.confluent.connect.jdbc.JdbcSourceConnector table.whitelist=users,places,sales tasks.max=1 conn...
Minh Mai
1

votes
1

answer
402

Views

Maximum retries and retry interval for Kafka JDBC Sink Connector when a database is down

I am trying to test and evaluate the behavior of a Kafka JDBC Sink Connector when the database is down. When a new message is received in Kafka while the database is down, the following error is reported: INFO Unable to connect to database on attempt 1/3. Will retry in 10000 ms. (io.confluent.conn...
1

votes
0

answer
373

Views

Does a Kafka Sink Connector for CSV file currently exists?

I need a kafka sink connector allowing the user to persist topic content as .CSV files. I've been investigating a while. Confluent provides FileSink connector which - as far as I understood - doesn't support CSV format. I've been playing a bit with this project but the sink task seems not impleme...
spi-x-i
1

votes
0

answer
229

Views

Kafka jdbc sink connector does not correct convert timestamp

i have a schema { 'type' : 'record', 'name' : 'test', 'namespace' : 'test', 'fields' : [ { 'name' : 'time', 'type': [ 'null', { 'type': 'long', 'logicalType': 'timestamp-millis' }] }, .... { 'name' : 'time2', 'type': ['null', { 'type' : 'long', 'logicaltype': 'timestamp-millis' }] } } But when kaf...
Mikhail
1

votes
0

answer
327

Views

jmx_prometheus_javaagent configuration for Kafka Connect?

The Prometheus jmx_exporter project gives an official example configuration for Kafka here: https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-0-8-2.yml Is there any similar configuration for Kafka Connect to export Kafka Connect JMX metrics to Prometheus?
clay
1

votes
0

answer
48

Views

Is there a way consume Kafka Messages from windows to a file on the server through Kafka Connect?

I'm able to write messages to a Kafka Topic, but I can't find the windows command to consume a message to a file through Kafka connect. Does anyone know the command? This commang works fine to write messages, but looking for the command to consume messages and write to a file on the server. ./bin/c...
JordanBlairDI
1

votes
0

answer
146

Views

Can't consume avro data from Confluent Kafka topics to Hbase

I use below sample code to load some test data from Confluent Kafka to Hbase, but somehow I do not see data comes in Hbase. I tried everything but did not work at all. Anyone can help this? How to duplicate the issue: Download the Sample code: https://github.com/mravi/kafka-connect-hbase Compile th...
Cauchy Wu
1

votes
0

answer
316

Views

Cannot sink Kafka stream to JDBC: unrecoverable exception

I'm trying to sink a stream of data from a MQTT Source with schema enabled to a Microsoft SQL Server database. I've followed many posts on the matter, regardless I receive the following error: org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception. at...
ilcorvo
1

votes
1

answer
61

Views

Kafka (Confluent Platform) input for Logstash - broken message encoding

I have a Confluent Platform (version 4.1.1). It is configured to read data from the database. The configuration for this is: name = source-mysql-requests connection.url = jdbc:mysql://localhost:3306/Requests connector.class = io.confluent.connect.jdbc.JdbcSourceConnector connection.user = *** connec...
Nikita Lipatov
1

votes
0

answer
1.6k

Views

Kafka Consumer Group Rebalance and Group Coordinator dead

I have been playing around with Kafka (1.0.0) for a couple of months and trying to understand how consumer group works. I have a single broker Kafka and I am using Kafka-Connect-Cassandra to consume messages from topics to database tables. I have 10 topics, all have just one partition and I have a S...
el323
1

votes
1

answer
460

Views

replication slot already exists

Whenever I restart the debezium kafka-connect container, or deploy another instance, I get the following error: io.debezium.jdbc.JdbcConnectionException: ERROR: replication slot 'debezium' already exists at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.initReplicationSlot...
Zach Mays
1

votes
0

answer
46

Views

Loading of Apache Avro plugin for Tranquility fails with Exception

For the Kafka Avro producer I run : ./kafka-avro-console-producer --broker-list localhost:9092 --topic pageviews --property value.schema='{'type':'record','name':'mypageviews','fields':[{'name':'time','type':'string'},{'name':'url','type':'string'},{'name':'user','type':'string'},{'name':'latencyMs'...
Saeed Mohtasham
1

votes
1

answer
58

Views

Am i using Kafka right?

I have installed Kafka 'kafka_2.12-0.11.0.0'. Started services of both zookeper and Kafka-server. Then i created a java project (kafka lib included) on IntelliJ. I also included an SQLite jar needed for the JDBC connect. I initially thought of using confluent.kafka-connect, but since mine is a windo...
Shashank
1

votes
0

answer
323

Views

Kafka connect to HBase

I am trying to to use Kafka Connect to HBase and there are no Confluent supported connectors available for HBase, though there are some community connectors available. We are not really ready to take risk in production with out support to the connectors: Is there any other work around for HBase conn...
BARATH
1

votes
1

answer
1.2k

Views

JDBC Kafka Connect with DB2

I'm struggling to get Confluent's kafka connector to connect to DB2. I am running an ubuntu instance inside docker for testing pruposes. The solution needs to be deployed to kubernetes, so docker it is. I have installed the Confluent platform using apt-get and adding their repos. All services are ru...
OzzMaster
1

votes
1

answer
236

Views

get task id's from kafka connect API to print in logs

I have a kafka connect sink code for which below json is passed as curl command to register tasks. Please let me know if anyone has any idea on how to get the task id's of my connect. For example in below example, we have defined max tasks is 3, so I need to know the name of 3 tasks for logs i.e. I...
Arun Shankar
1

votes
0

answer
282

Views

WARNING : createConnector contains empty path annotation

Using documentation of confluent for kafka_cluster I got this error : Jul 03, 2018 4:20:41 PM org.glassfish.jersey.internal.Errors logErrors WARNING: The following warnings have been detected: WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.Connec...
Wiem Bel Hadj
1

votes
1

answer
41

Views

where can I find the data sent from kafka to elasticSearch ? knowing that I using a ElasticSearch sink?

This my output. I think that everything works fine: [2018-07-06 22:27:51,458] INFO [Consumer clientId=consumer-4, groupId=connect-elk-sink] Setting newly assigned partitions [elk-test-0] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [2018-07-06 22:27:51,471] INFO [Consumer client...
Wiem Bel Hadj
1

votes
0

answer
86

Views

KCQL query to push Influx line protocol points from kafka topic to InfluxDB

I send influx line protocol points to a kafka-topic. I am able to read those points from kafka-cli . Now I want to insert all the points in the kafka-topic in InfluxDB. I configured kafka influxdb sink connector to push data into influxdb. I am not able to figure out the KCQL query to do that. I tr...
kid
1

votes
1

answer
224

Views

ERROR: using kafka-connect-cdc-mssql:1.0.0-preview

I try read the log transaction with Kafka Connector, with connector kafka-connect-cdc-mssql:1.0.0-preview I'm using the Confluent CLI and I have already modified the configuration file: etc/schema-registry/connect-avro-standalone.properties I add : plugin.path=/usr/share/java,/tmp/Softwares/conflue...
1

votes
0

answer
329

Views

Kafka JDBC-Sink connector not working as intended

I a trying to use the JDBC sink connector to put data into Postgres, however, I am not seeing any data created in my database. The connector config I am using it { 'name': 'Test-Insert', 'config': { 'connector.class': 'io.confluent.connect.jdbc.JdbcSinkConnector', 'tasks.max': '1', 'connection.url':...
Tom Halson
1

votes
0

answer
170

Views

Confluent MSSQL CDC Connector not fetching changes

If anyone had played with Confluent MSSQL CDC Connector (https://docs.confluent.io/current/connect/kafka-connect-cdc-mssql/index.html) I tried setting up this connector, downloading the jar and setting up config files as mentioned in docs. Running it is actually not throwing any error but it NOT abl...
dbansal
1

votes
2

answer
155

Views

Dynamic topic in kafka connector

kafka added new feature to use regex in connectors, however it seems that the topic data from the newly added topics after the connector has been started is not consumed until the connector is restarted. We have a need to dynamically added new topic and have connector consume the topic based on rege...
Pratik Gaglani
1

votes
0

answer
260

Views

Unable to sink messages from a Kafka consumer to Postgres

i am able to read messages from a kafka topic using kafka-console-consumer /bin/kafka-console-consumer --topic Test --consumer.config /etc/kafka/consumer.properties --from-beginning --bootstrap-server ip1:9095,ip2:9095,ip3:9095 but unable to sink these messages to a table in postgres using jdbc-s...
fangorn
1

votes
0

answer
104

Views

Kafka Connect MySQL to ES

Lets say I want to move data from 3 mysql tables to 1 ES index using Kafka connectors. This 3 tables are the result of database normalization, in ES there is no need to have 3 indices for this, ideally I would have one index and each document on that index would have properties. For example in MySQ...
lloiacono
1

votes
0

answer
51

Views

Kafka Connector - From Kafka to Hadoop

I want to use Kafka-HDFS-Connector Sending Data From Kafka to Hadoop. I have around 150B JSON records per day, Is it possible to do it with Kafka-HDFS-Connector (performance issue)? Thanks.
Ya Ko
1

votes
0

answer
59

Views

Kafka HBase Sink Connector unable to deliver its messages to HBase

I have particular Kafka HBase Sink Connector problem for which I will appreciate any advise or suggestions. It is a 3-node Kafka cluster - 2 nodes for connect-distributed and 1 node for schema registry + kafka streaming. The Kafka version is 0.10.1 and is part of the Hortonworks platform 2.6.3. Ther...
kratunko
1

votes
0

answer
94

Views

Kafka connect HDFS sink ERROR failed creating a WAL

I'm using Kafka connect HDFS. When I'm trying to run my connector I'm got the following exception: ERROR Failed creating a WAL Writer: Failed to create file[/path/log] for [DFSClient_NONMAPREDUCE_208312334_41] for client [IP] because this file is already beign create by [DFSClient_NONMAPREDUCE_16532...
Ya Ko

View additional questions