Questions tagged [apache-kafka-connect]
309 questions
1
votes
1
answer
218
Views
Kafka Connect JDBC Sink Connector
I am trying to write data from a topic (json data) into a MySql Database. I believe I want a JDBC Sink Connector.
How do I configure the connector to map the json data in the topic to how to insert data into the database.
The only documentation I can find is this.
'The sink connector requires knowl...
1
votes
1
answer
491
Views
Kafka Connect HDFS Sink with Azure Blob Storage
I want to connect to Azure Blob Storage with Kafka HDFS Sink Connector.
So far I have done:
Set kafka-connect properties:
hdfs.url=wasbs://
hadoop.conf.dir={hadoop_3_home}/etc/hadoop/
hadoop.home={hadoop_3_home}
And in core-site.xml added support for wasbs:
fs.wasbs.impl
org.apache.hadoop.fs.azure.N...
1
votes
0
answer
310
Views
How to read kafka statestore in processor which was updated from a different processor?
I have created 2 kafka statestores corresponding to 2 different topics as below.
StateStoreSupplier tempStore1 = Stores.create('tempStore1').withKeys(Serdes.String()).withValues(valueSerde).persistent().build();
StateStoreSupplier tempStore2 = Stores.create('tempStore2').withKeys(Serdes.String()).wi...
1
votes
2
answer
920
Views
Kafka - ERROR Stopping after connector error java.lang.IllegalArgumentException: Number of groups must be positive
Working on setting up Kafka running from our RDS Postgres 9.6 to Redhift. Using the guidelines at https://blog.insightdatascience.com/from-postgresql-to-redshift-with-kafka-connect-111c44954a6a and we have the all of the infrastructure set up, and am working on fully setting up Confluent. I'm gett...
1
votes
0
answer
410
Views
Conlfluent 4.0 - Error starting kafka-connect with avro support
I am getting lots of exceptions when trying to start kafka-connect with the command:
./bin/connect-distributed ./etc/schema-registry/connect-avro-distributed.properties
With every single jar in the share/java dir (at least), I get exceptions like this, complaining about bad magic number:
[2018-01-30...
1
votes
1
answer
895
Views
Kafka connection transformations. Parse input string and get a record key
I use a simple file source reader
connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector
tasks.max=1
File content is a simple JSON object in each line. I found that there is a way to replace a record key and use transformations to do this, like
# Add the `id` field as the key using...
1
votes
3
answer
272
Views
Kafka JDBC Connect query causes ORA-00933: SQL command not properly ended
I have this Oracle SQL query:
SELECT * FROM
(SELECT SO_ORDER_KEY,QUEUE_TYPE,SYS_NO,
DENSE_RANK() OVER (PARTITION BY SO_ORDER_KEY ORDER BY SYS_NO DESC) ORDER_RANK
FROM TSY940)
WHERE ORDER_RANK=1;
When running in SQL developer, it returns the desired result.
For some reason when I use this query in...
1
votes
2
answer
264
Views
Does scaling Kafka Connect is same as scaling Kafka Consumer?
We need to pull data from Kafka and write into AWS s3. The Kafka is managed by separate department and we have access to only specific topic.
Based on Kafka documentation it looks like Kafka Connect is easy solution for me because I don't have any custom message processing logic.
Normally when we ru...
1
votes
0
answer
448
Views
How to use Schema Registry and AvroConverter in Kafka Source connector?
I'm trying to write custom Source and Sink Kafka connectors for MongoDB with schema registry and Avro. I have configured the key and value converters as AvroConverter and the schema registry url in the properties. What code do i have to add in my connector so that it converts the data to Avro and v...
1
votes
0
answer
824
Views
Kafka to postgresql using kafka connect jdbc sink connector
I am trying to connect kafka with postgresql. I am using JDBC sink connector.
This is my sink.properties file:
name=test-sink
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
topics=orders
connection.url=jdbc:postgresql://localhost:5432/casb
auto.create=true
I have created a t...
1
votes
1
answer
966
Views
Kafka Conenct HDFS sink saving data in parquet format
Using Kafka Connect HDFS Sink I am able to write avro data to Kafka topic and save data in hive/hdfs.
I am trying to save data in parquet file format using format class
format.class=io.confluent.connect.hdfs.parquet.ParquetFormat
quickstart-hdfs.properties are as follows
name=hdfs-sink
connector.cl...
1
votes
0
answer
49
Views
How does Schema Registry integrate with Kafka Source Connector?
I have added Topic-Key and Topic-Value schemas for a given topic using REST APIs. In my custom connector, do I need to create a schema again using SchemaBuilder? How do I access the registered schemas inside my connector ?
1
votes
0
answer
182
Views
Kafka Connect from RDS to RedShift not starting
I was able to implement Kafka Connect on a much smaller table but am trying to implement it on a larger database. My source and sink configuration are as followed
source:
name=rds-source
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
table.whitelist=users,places,sales
tasks.max=1
conn...
1
votes
1
answer
402
Views
Maximum retries and retry interval for Kafka JDBC Sink Connector when a database is down
I am trying to test and evaluate the behavior of a Kafka JDBC Sink Connector when the database is down.
When a new message is received in Kafka while the database is down, the following error is reported:
INFO Unable to connect to database on attempt 1/3. Will retry in 10000 ms. (io.confluent.conn...
1
votes
0
answer
373
Views
Does a Kafka Sink Connector for CSV file currently exists?
I need a kafka sink connector allowing the user to persist topic content as .CSV files. I've been investigating a while.
Confluent provides FileSink connector which - as far as I understood - doesn't support CSV format.
I've been playing a bit with
this project but the sink task seems not impleme...
1
votes
0
answer
229
Views
Kafka jdbc sink connector does not correct convert timestamp
i have a schema
{
'type' : 'record',
'name' : 'test',
'namespace' : 'test',
'fields' : [ {
'name' : 'time',
'type': [ 'null', {
'type': 'long',
'logicalType': 'timestamp-millis'
}]
},
....
{
'name' : 'time2',
'type': ['null', {
'type' : 'long',
'logicaltype': 'timestamp-millis'
}]
}
}
But when kaf...
1
votes
0
answer
327
Views
jmx_prometheus_javaagent configuration for Kafka Connect?
The Prometheus jmx_exporter project gives an official example configuration for Kafka here:
https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-0-8-2.yml
Is there any similar configuration for Kafka Connect to export Kafka Connect JMX metrics to Prometheus?
1
votes
0
answer
48
Views
Is there a way consume Kafka Messages from windows to a file on the server through Kafka Connect?
I'm able to write messages to a Kafka Topic, but I can't find the windows command to consume a message to a file through Kafka connect. Does anyone know the command?
This commang works fine to write messages, but looking for the command to consume messages and write to a file on the server.
./bin/c...
1
votes
0
answer
146
Views
Can't consume avro data from Confluent Kafka topics to Hbase
I use below sample code to load some test data from Confluent Kafka to Hbase, but somehow I do not see data comes in Hbase. I tried everything but did not work at all. Anyone can help this?
How to duplicate the issue:
Download the Sample code:
https://github.com/mravi/kafka-connect-hbase
Compile th...
1
votes
0
answer
316
Views
Cannot sink Kafka stream to JDBC: unrecoverable exception
I'm trying to sink a stream of data from a MQTT Source with schema enabled to a Microsoft SQL Server database.
I've followed many posts on the matter, regardless I receive the following error:
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at...
1
votes
1
answer
61
Views
Kafka (Confluent Platform) input for Logstash - broken message encoding
I have a Confluent Platform (version 4.1.1). It is configured to read data from the database. The configuration for this is:
name = source-mysql-requests
connection.url = jdbc:mysql://localhost:3306/Requests
connector.class = io.confluent.connect.jdbc.JdbcSourceConnector
connection.user = ***
connec...
1
votes
0
answer
1.6k
Views
Kafka Consumer Group Rebalance and Group Coordinator dead
I have been playing around with Kafka (1.0.0) for a couple of months and trying to understand how consumer group works. I have a single broker Kafka and I am using Kafka-Connect-Cassandra to consume messages from topics to database tables. I have 10 topics, all have just one partition and I have a S...
1
votes
1
answer
460
Views
replication slot already exists
Whenever I restart the debezium kafka-connect container, or deploy another instance, I get the following error:
io.debezium.jdbc.JdbcConnectionException: ERROR: replication slot 'debezium' already exists
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.initReplicationSlot...
1
votes
0
answer
46
Views
Loading of Apache Avro plugin for Tranquility fails with Exception
For the Kafka Avro producer I run :
./kafka-avro-console-producer --broker-list localhost:9092 --topic pageviews --property value.schema='{'type':'record','name':'mypageviews','fields':[{'name':'time','type':'string'},{'name':'url','type':'string'},{'name':'user','type':'string'},{'name':'latencyMs'...
1
votes
1
answer
58
Views
Am i using Kafka right?
I have installed Kafka 'kafka_2.12-0.11.0.0'.
Started services of both zookeper and Kafka-server.
Then i created a java project (kafka lib included) on IntelliJ.
I also included an SQLite jar needed for the JDBC connect.
I initially thought of using confluent.kafka-connect, but since mine is a windo...
1
votes
0
answer
323
Views
Kafka connect to HBase
I am trying to to use Kafka Connect to HBase and there are no Confluent supported connectors available for HBase, though there are some community connectors available. We are not really ready to take risk in production with out support to the connectors: Is there any other work around for HBase conn...
1
votes
1
answer
1.2k
Views
JDBC Kafka Connect with DB2
I'm struggling to get Confluent's kafka connector to connect to DB2.
I am running an ubuntu instance inside docker for testing pruposes. The solution needs to be deployed to kubernetes, so docker it is.
I have installed the Confluent platform using apt-get and adding their repos. All services are ru...
1
votes
1
answer
236
Views
get task id's from kafka connect API to print in logs
I have a kafka connect sink code for which below json is passed as curl command to register tasks.
Please let me know if anyone has any idea on how to get the task id's of my connect. For example in below example, we have defined max tasks is 3, so I need to know
the name of 3 tasks for logs i.e. I...
1
votes
0
answer
282
Views
WARNING : createConnector contains empty path annotation
Using documentation of confluent for kafka_cluster
I got this error :
Jul 03, 2018 4:20:41 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.Connec...
1
votes
1
answer
41
Views
where can I find the data sent from kafka to elasticSearch ? knowing that I using a ElasticSearch sink?
This my output. I think that everything works fine:
[2018-07-06 22:27:51,458] INFO [Consumer clientId=consumer-4, groupId=connect-elk-sink] Setting newly assigned partitions [elk-test-0] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2018-07-06 22:27:51,471] INFO [Consumer client...
1
votes
0
answer
86
Views
KCQL query to push Influx line protocol points from kafka topic to InfluxDB
I send influx line protocol points to a kafka-topic. I am able to read those points from kafka-cli . Now I want to insert all the points in the kafka-topic in InfluxDB. I configured kafka influxdb sink connector to push data into influxdb. I am not able to figure out the KCQL query to do that.
I tr...
1
votes
1
answer
224
Views
ERROR: using kafka-connect-cdc-mssql:1.0.0-preview
I try read the log transaction with Kafka Connector, with connector kafka-connect-cdc-mssql:1.0.0-preview
I'm using the Confluent CLI and I have already modified the configuration file:
etc/schema-registry/connect-avro-standalone.properties
I add :
plugin.path=/usr/share/java,/tmp/Softwares/conflue...
1
votes
0
answer
329
Views
Kafka JDBC-Sink connector not working as intended
I a trying to use the JDBC sink connector to put data into Postgres, however, I am not seeing any data created in my database. The connector config I am using it
{
'name': 'Test-Insert',
'config': {
'connector.class': 'io.confluent.connect.jdbc.JdbcSinkConnector',
'tasks.max': '1',
'connection.url':...
1
votes
0
answer
170
Views
Confluent MSSQL CDC Connector not fetching changes
If anyone had played with Confluent MSSQL CDC Connector (https://docs.confluent.io/current/connect/kafka-connect-cdc-mssql/index.html)
I tried setting up this connector, downloading the jar and setting up config files as mentioned in docs. Running it is actually not throwing any error but it NOT abl...
1
votes
2
answer
155
Views
Dynamic topic in kafka connector
kafka added new feature to use regex in connectors, however it seems that the topic data from the newly added topics after the connector has been started is not consumed until the connector is restarted. We have a need to dynamically added new topic and have connector consume the topic based on rege...
1
votes
0
answer
260
Views
Unable to sink messages from a Kafka consumer to Postgres
i am able to read messages from a kafka topic using kafka-console-consumer
/bin/kafka-console-consumer --topic Test --consumer.config
/etc/kafka/consumer.properties --from-beginning --bootstrap-server
ip1:9095,ip2:9095,ip3:9095
but unable to sink these messages to a table in postgres using jdbc-s...
1
votes
0
answer
104
Views
Kafka Connect MySQL to ES
Lets say I want to move data from 3 mysql tables to 1 ES index using Kafka connectors.
This 3 tables are the result of database normalization, in ES there is no need to have 3 indices for this, ideally I would have one index and each document on that index would have properties. For example in MySQ...
1
votes
0
answer
51
Views
Kafka Connector - From Kafka to Hadoop
I want to use Kafka-HDFS-Connector Sending Data From Kafka to Hadoop.
I have around 150B JSON records per day, Is it possible to do it with Kafka-HDFS-Connector (performance issue)?
Thanks.
1
votes
0
answer
59
Views
Kafka HBase Sink Connector unable to deliver its messages to HBase
I have particular Kafka HBase Sink Connector problem for which I will appreciate any advise or suggestions.
It is a 3-node Kafka cluster - 2 nodes for connect-distributed and 1 node for schema registry + kafka streaming. The Kafka version is 0.10.1 and is part of the Hortonworks platform 2.6.3. Ther...
1
votes
0
answer
94
Views
Kafka connect HDFS sink ERROR failed creating a WAL
I'm using Kafka connect HDFS.
When I'm trying to run my connector I'm got the following exception:
ERROR Failed creating a WAL Writer: Failed to create file[/path/log] for [DFSClient_NONMAPREDUCE_208312334_41] for client [IP] because this file is already beign create by [DFSClient_NONMAPREDUCE_16532...