Kafka Direct to Autoscale Plugin

Overview

The Kafka Direct To Autoscale Plugin enables users to securely connect to and poll messages from their Kafka brokers, successfully initiating Autoscale processes for each message based on their configurations. This is achieved by utilizing the custom Kafka Autoscale Batching smart service node and setting up auxiliary process models to support the batching of Kafka records and initiation of Autoscale processes. As this plugin consumes Kafka records and triggers Autoscale processes, Appian’s Autoscale feature will dynamically adjust process execution capacity, ensuring high process throughput in heavy-load scenarios.

This plugin is particularly useful for:

Leveraging the capabilities of the Autoscale feature
Establishing a seamless and secure connection to a Kafka broker, and subscribing to Kafka topics for real-time data consumption
Taking action on each consumed Kafka record by initiating an Autoscale process

For more inquiries about the plugin, please contact techpartners@appian.com.

Key Features & Functionality

Smart Services: Kafka Autoscale Batching
The Kafka Autoscale Batching node acts as a Kafka consumer that performs the following tasks:

Subscription: Subscribes to one or more Kafka topics using a specified consumer group ID
Message Polling: Continuously polls a batch of messages from the subscribed topic(s)
Batch Processing: Triggers an intermediary Autoscale process for each batch, passing the polled messages as a process parameter for downstream processing

Along with the process model that includes the Kafka Autoscale Batching node, additional processes will need to be configured to provide an optimal setup. Please review the official documentation for more details.

Compatible Site Versions: 24.4+

Please note that the site the plugin is deployed to must have Autoscale enabled. While the plugin can be deployed to a site that doesn’t have Autoscale, this site will not let users create Autoscale processes and will throw an error if an attempt to run an Autoscale Process from the Kafka Autoscale Batching node is made.