Appian integrated with Kafka

A Score Level 1

Hello everyone.

I would like to know if it's possible to use Appian as a consumer for a specific Kafka topic.

In our very specific case, let's suppose a user updates an address and the message is replicated to multiple systems, through Kafka.

Our idea was to enable Appian to consume from this topic and these messages.

I tried to configure the Kafka Tools plugin available in the AppMarket, but I can't seem to configure the nodes correctly.

I have a demo instance in Confluent Cloud and I was able to do some producer/consumer tests locally and also via Python Client, but I'm not able to consume anything when it concerns Appian.

Do we have a better way of integrating with a Kafka topic?

Kind regards

  Discussion posts and replies are publicly visible

Parents
  • 0
    Certified Associate Developer

    I am trying to use this plugin as well. But I can't configure it properly to work. I can't find any documentations on how to use it in appian or how it works

  • 0
    A Score Level 1
    in reply to Mr. Agda

    Hi Mr. Agda, if you download the .zip file for the plugin, you'll find a guide inside.

    I was able to connect to Confluent Cloud to produce messages, but I wasn't able to consume (you need to install the Transaction Manager in order to properly configure it).

  • 0
    Certified Associate Developer
    in reply to Marta

    Where did you get the private key? 
    I only recieve a username and password for credentials

  • Has anyone ever figured out the solution to capture Kafka message inside Appian ? Any help is greatly appreciated as there is no documentation around this technical feature. Thanks!

  • 0
    Certified Associate Developer
    in reply to Harsha Pasunuri

    We are able to use this kafka plugin in appian. We are able to consume topics directly to the database though this also relies on the transaction manager. 

  • Hello Agda, 

    We are trying to consume data from Kafka topic but getting below error, can you please help step by step procedure to configure the keystore and truststore. We struck with the below issue and your guidance would be highly appreciated.

    2023-05-25 11:44:18,210 [Thread-6289] ERROR com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumerThread - Received an exception from consumer: 13640642 - Failed to construct kafka consumer
    org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:830)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:665)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:626)
    at com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumer.run(AppianKafkaConsumer.java:94)
    at com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumerThread.run(AppianKafkaConsumerThread.java:30)
    Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /usr/local/appian/ae/_admin/accdocs2/8355/10010502.jks of type JKS
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:375)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:347)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createTruststore(DefaultSslEngineFactory.java:320)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:168)
    at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:140)
    at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:97)
    at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73)
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:737)
    ... 5 more
    Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
    at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:792)
    at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:57)
    at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
    at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:71)
    at java.security.KeyStore.load(KeyStore.java:1445)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:372)
    ... 15 more
    Caused by: java.security.UnrecoverableKeyException: Password verification failed
    at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:790)
    ... 20 more

Reply
  • Hello Agda, 

    We are trying to consume data from Kafka topic but getting below error, can you please help step by step procedure to configure the keystore and truststore. We struck with the below issue and your guidance would be highly appreciated.

    2023-05-25 11:44:18,210 [Thread-6289] ERROR com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumerThread - Received an exception from consumer: 13640642 - Failed to construct kafka consumer
    org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:830)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:665)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:626)
    at com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumer.run(AppianKafkaConsumer.java:94)
    at com.appiancorp.cs.plugin.kafka.consumer.AppianKafkaConsumerThread.run(AppianKafkaConsumerThread.java:30)
    Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /usr/local/appian/ae/_admin/accdocs2/8355/10010502.jks of type JKS
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:375)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:347)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createTruststore(DefaultSslEngineFactory.java:320)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:168)
    at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:140)
    at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:97)
    at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73)
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:737)
    ... 5 more
    Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
    at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:792)
    at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:57)
    at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
    at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:71)
    at java.security.KeyStore.load(KeyStore.java:1445)
    at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:372)
    ... 15 more
    Caused by: java.security.UnrecoverableKeyException: Password verification failed
    at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:790)
    ... 20 more

Children