Appian integrated with Kafka

A Score Level 1

Hello everyone.

I would like to know if it's possible to use Appian as a consumer for a specific Kafka topic.

In our very specific case, let's suppose a user updates an address and the message is replicated to multiple systems, through Kafka.

Our idea was to enable Appian to consume from this topic and these messages.

I tried to configure the Kafka Tools plugin available in the AppMarket, but I can't seem to configure the nodes correctly.

I have a demo instance in Confluent Cloud and I was able to do some producer/consumer tests locally and also via Python Client, but I'm not able to consume anything when it concerns Appian.

Do we have a better way of integrating with a Kafka topic?

Kind regards

  Discussion posts and replies are publicly visible

Parents Reply Children
  • 0
    A Score Level 1
    in reply to Mr. Agda

    Hi Mr. Agda, if you download the .zip file for the plugin, you'll find a guide inside.

    I was able to connect to Confluent Cloud to produce messages, but I wasn't able to consume (you need to install the Transaction Manager in order to properly configure it).

  • 0
    Certified Associate Developer
    in reply to Marta

    Where did you get the private key? 
    I only recieve a username and password for credentials

  • Has anyone ever figured out the solution to capture Kafka message inside Appian ? Any help is greatly appreciated as there is no documentation around this technical feature. Thanks!

  • 0
    Certified Associate Developer
    in reply to Harsha Pasunuri

    We are able to use this kafka plugin in appian. We are able to consume topics directly to the database though this also relies on the transaction manager. 

  • Hello Agda, 

    We are trying to consume data from Kafka topic but getting below error, can you please help step by step procedure to configure the keystore and truststore. We struck with the below issue and your guidance would be highly appreciated.

    2023-05-25 11:44:18,210 [Thread-6289] ERROR com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumerThread - Received an exception from consumer: 13640642 - Failed to construct kafka consumer
    org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:830)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:665)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:626)
    at com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumer.run(AppianKafkaConsumer.java:94)
    at com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumerThread.run(AppianKafkaConsumerThread.java:30)
    Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /usr/local/appian/ae/_admin/accdocs2/8355/10010502.jks of type JKS
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:375)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:347)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createTruststore(DefaultSslEngineFactory.java:320)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:168)
    at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:140)
    at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:97)
    at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73)
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:737)
    ... 5 more
    Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
    at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:792)
    at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:57)
    at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
    at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:71)
    at java.security.KeyStore.load(KeyStore.java:1445)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:372)
    ... 15 more
    Caused by: java.security.UnrecoverableKeyException: Password verification failed
    at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:790)
    ... 20 more

  • 0
    Certified Associate Developer
    in reply to srib0003

    in our case we left the truststore and keystore as blank on the consume smart service. From what I see on your logs, you are getting a password verification issue. 
    Did you create a third party credentials on the admin console? and use it as a 'secure credentials store key' value on the smart service? 

  • Thanks for your reply!!

    Yes, we have created a third party credentials on the admin console and used it as a 'secure credentials store key' value on the smart service.

    Have you used self signed certificate (keystore) or signed by any CA authority ? 

  • Did you create a third party credentials on the admin console? and use it as a 'secure credentials store key' value on the smart service?

    We have confusions in creating keystore and adding them to the third party credentials.

    Can you please explain how did you generate the keystore, what are the parameter that used from kafka and appian side?

  • 0
    Certified Senior Developer
    in reply to srib0003

    Hi Sirb,

    Just quick question where did you find these error listed? Cannot find any logs related to Kafka Consumer errors. And there is no error in PM as well.

    Thank you so much!

  • I am trying to produce a message on my topic. I am getting an exception which is a timeout exception. Topic not present in metadata after 60000ms.

    But the same topic I am using through the spring boot application and it is working as expected. how to solve this issue.