Overview
KafkaTools provides the following functionality that allows publishing to and consuming from topics on Kafka Servers.
Smart Services
In order to process the messages consumed from Kafka, it is recommended to use the Transaction Manager application. The models designed to process messages will be configured an assigned through the transaction manager job types. See documentation for Transaction Manager
Key Features & Functionality
Please refer to the README for additional details.
Our client uses the Confluent tool, which encompasses Kafka.From Appian, with this plugin, we can publish & consume messages in a topic, but, as is being done through Confluent, the messages negotiate with a scheme, where the message format is AVRO (Json with headers) and each message published and consumed have to go through serialization / deserialization. Therefore, consumed messages arrive at Appian in an unreadable format.
This is a message to the author of the "Kafka Tools" plugin: Is there an update in progress for this Appian plugin that would allow for the necessary Java libraries for Confluent, as well as the necessary classes to allow the message to be serialized/deserialized?
And please, if anyone has had a situation similar to what I am talking about, please contact me, I will greatly appreciate it.
Thank you so much,I am looking forward to your response
Me and my client is phasing exactly the same problem. Except byte array deserializing (with ByteArrayDeserializer) we also need to deserialize the Avro to be able to store a readable string into TM database.
Have you got any input or coming any further into a solution to the problem?
The solution that my client has adopted regarding this is to develop an external microservice and put it as an intermediate layer between Appian and Confluent. The microservice is able to read the message in AVRO format and transform it to String type, and will publish it to a Kafka topic in Confluent in string format. So, the reading from Appian is correct, because we are finally reading a string type. I hope this information helps you :)