Kafka-Connect-MongoDB Sink Connector Update Issue on MongoDB

Refresh

April 2019

Views

132 time

1

I am using JDBC source and kafka-connect-mongodb sink connector to stream data from JDBC to MongoDB.

So far, I am able to get all inserts and updates as a new document (Yes, updates are inserted as new documents) on MongoDB side with following configs;

repo: https://github.com/hpgrahsl/kafka-connect-mongodb

configs:

"mongodb.document.id.strategy":"at.grahsl.kafka.connect.mongodb.processor.id.strategy.BsonOidStrategy",
"mongodb.post.processor.chain":"at.grahsl.kafka.connect.mongodb.processor.DocumentIdAdder",
"mongodb.delete.on.null.values":false,
"mongodb.writemodel.strategy":"at.grahsl.kafka.connect.mongodb.writemodel.strategy.ReplaceOneDefaultStrategy"

What I would like to achieve now is to update a document in case a row is updated on Oracle side.

With below configurations, I am able to get only the last insert or update done in Oracle side into MongoDB collection as a single document. I mean what I found in MongoDB is a collection with one document which is the last operation.

"mongodb.document.id.strategy":"at.grahsl.kafka.connect.mongodb.processor.id.strategy.PartialValueStrategy",
"mongodb.key.projection.list": "name,surname",
"mongodb.key.projection.type":"whitelist",
"mongodb.post.processor.chain":"at.grahsl.kafka.connect.mongodb.processor.DocumentIdAdder",
"mongodb.writemodel.strategy":"at.grahsl.kafka.connect.mongodb.writemodel.strategy.ReplaceOneBusinessKeyStrategy"

Appriciate any help in advance. Many thanks, Soner

0 answers