Hello everyone.
I would like to know if it's possible to use Appian as a consumer for a specific Kafka topic.
In our very specific case, let's suppose a user updates an address and the message is replicated to multiple systems, through Kafka.
Our idea was to enable Appian to consume from this topic and these messages.
I tried to configure the Kafka Tools plugin available in the AppMarket, but I can't seem to configure the nodes correctly.
I have a demo instance in Confluent Cloud and I was able to do some producer/consumer tests locally and also via Python Client, but I'm not able to consume anything when it concerns Appian.
Do we have a better way of integrating with a Kafka topic?
Kind regards
Discussion posts and replies are publicly visible
I am trying to use this plugin as well. But I can't configure it properly to work. I can't find any documentations on how to use it in appian or how it works
Hi Mr. Agda, if you download the .zip file for the plugin, you'll find a guide inside.
I was able to connect to Confluent Cloud to produce messages, but I wasn't able to consume (you need to install the Transaction Manager in order to properly configure it).
Where did you get the private key? I only recieve a username and password for credentials
Has anyone ever figured out the solution to capture Kafka message inside Appian ? Any help is greatly appreciated as there is no documentation around this technical feature. Thanks!
We are able to use this kafka plugin in appian. We are able to consume topics directly to the database though this also relies on the transaction manager.
Hello Agda,
We are trying to consume data from Kafka topic but getting below error, can you please help step by step procedure to configure the keystore and truststore. We struck with the below issue and your guidance would be highly appreciated.
2023-05-25 11:44:18,210 [Thread-6289] ERROR com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumerThread - Received an exception from consumer: 13640642 - Failed to construct kafka consumerorg.apache.kafka.common.KafkaException: Failed to construct kafka consumer at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:830) at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:665) at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646) at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:626) at com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumer.run(AppianKafkaConsumer.java:94) at com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumerThread.run(AppianKafkaConsumerThread.java:30)Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /usr/local/appian/ae/_admin/accdocs2/8355/10010502.jks of type JKS at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:375) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:347) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createTruststore(DefaultSslEngineFactory.java:320) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:168) at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:140) at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:97) at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:737) ... 5 moreCaused by: java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:792) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:57) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:71) at java.security.KeyStore.load(KeyStore.java:1445) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:372) ... 15 moreCaused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:790) ... 20 more
in our case we left the truststore and keystore as blank on the consume smart service. From what I see on your logs, you are getting a password verification issue. Did you create a third party credentials on the admin console? and use it as a 'secure credentials store key' value on the smart service?
Thanks for your reply!!
Yes, we have created a third party credentials on the admin console and used it as a 'secure credentials store key' value on the smart service.
Have you used self signed certificate (keystore) or signed by any CA authority ?
Mr. Agda said:Did you create a third party credentials on the admin console? and use it as a 'secure credentials store key' value on the smart service?
We have confusions in creating keystore and adding them to the third party credentials.
Can you please explain how did you generate the keystore, what are the parameter that used from kafka and appian side?
Hi Sirb,
Just quick question where did you find these error listed? Cannot find any logs related to Kafka Consumer errors. And there is no error in PM as well.
Thank you so much!
I am trying to produce a message on my topic. I am getting an exception which is a timeout exception. Topic not present in metadata after 60000ms.
But the same topic I am using through the spring boot application and it is working as expected. how to solve this issue.
Did you ever resolve this issue? Currently encountering the same problem on my end.