Creating a connector with Kafka Connect Distributed returning 500 error


March 2019


1.4k time


I am working on migrating from Camus to Kafka Connect. I am working on the implementation of Kafka Connect and specifically focused on distributed mode. I am able to start a worker successfully on my local machine which I assume communicates with my Kafka cluster. I am further able to run two GET commands such as / and /connector-plugins which return the correct JSON. However, when I try to POST a command to create a connector, I receive a 500 error and a time out. Specifically, I use this command to POST for testing:

curl -X POST -H "Content-Type: application/json" --data '{"name": "local-file-sink", "config": {"connector.class":"FileStreamSinkConnector", "tasks.max":"1", "file":"test.sink.txt", "topics":"myTopic" }}' localhost:8083/connectors

and eventually I get this response:

{"error_code": 500, "message": "Request timed out"}

I am lost as to what is going on. The logs from my Kafka Connect distributed worker show this:

[2016-12-05 14:34:32,436] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:22:34:32 +0000] "GET /connector-plugins HTTP/1.1" 200 315  2 (
[2016-12-05 15:05:25,422] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:25 +0000] "GET /connector-plugins HTTP/1.1" 200 315  3 (
[2016-12-05 15:05:28,389] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:05:28 +0000] "GET /connector-plugins HTTP/1.1" 200 315  2 (
[2016-12-05 15:07:38,644] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:06:08 +0000] "GET /connectors HTTP/1.1" 500 48  90003 (
[2016-12-05 15:07:44,450] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:07:44 +0000] "GET /connector-plugins HTTP/1.1" 200 315  1 (
[2016-12-05 15:13:06,703] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:11:36 +0000] "POST /connectors HTTP/1.1" 500 48  90003 (
[2016-12-05 15:15:38,506] INFO 0:0:0:0:0:0:0:1 - - [05/Dec/2016:23:14:08 +0000] "POST /connectors HTTP/1.1" 500 48  90005 (

where you can see the error codes and the commands.

I guess my main questions and issues are:

  1. How can I better debug Kafka Connect so I can try and fix this?
  2. Is there anything that I'm doing that is glaringly wrong?
  3. Is there any step-by-step documentation or blog posts on getting a Kafka Connect distributed worker and connector to run? I have not really seen anything or even best practices kinds of documentation? Maybe I am just way too early of an adopter.

I look forward to hearing back from the community and thank you for your help!

1 answers


I was able to fix this issue. I was running Confluent Platform 3.0.1 on both my client and my cluster. I upgraded my cluster to Confluent Platform 3.1.1 and my client to the new version. I was able to start the cluster. Additionally, you can learn more about debugging Kafka Connect by reading this thread on the Kafka Users Group regarding this same question:[email protected]%3e

Hopefully that helps others.