Open Systems Connector: Enhanced Kafka Consumer Thread Configuration for Scalability

What Changed

Open Systems Data Connector ARM template (azureDeploy-OpenSystems-DataConnector.json) adds configurable consumer thread parameters for all six log stream types: Secure Web Gateway, Identity, Firewall, Zero Trust Network Access, Secure Email Gateway, and Intrusion Detection System.

Connector Configuration Enhancement

The ARM template now exposes consumer_threads parameters for each Kafka consumer, replacing hardcoded single-threaded processing. Key changes:

• New parameters: proxyConsumerThreads, identityConsumerThreads, firewallConsumerThreads, ztnaConsumerThreads, secureE-MailGatewayConsumerThreads, intrusionDetectionSystemConsumerThreads (all default to 1)
• Logstash configuration: Each Kafka input block gains consumer_threads and enable_auto_commit => false
• Parameter reorganization: Moved authentication and location parameters to logical grouping at template top

Data Ingestion Impact

Previously, all Open Systems log streams used single-threaded Kafka consumers, creating potential throughput bottlenecks in high-volume environments. This change enables administrators to scale consumer threads per log type based on actual data volumes — critical for deployments with heavy proxy traffic or high authentication event rates.

The change maintains backward compatibility with default single-threaded operation while allowing performance tuning for production deployments experiencing ingestion lag.

Affected Files

Solutions/Open Systems/DataConnectors/azureDeploy-OpenSystems-DataConnector.json