Since we will exposing our application to the public Internet, we need. In this post, we will do a step-by-step configuration of the strimzi-operator & use Openshift routes as an external listener with security: SASL_SSL. To configure an external listener that uses the NodePort access method, complete the following steps. For more complex networking this might be an IP address associated with a given network interface on a machine. To use the protocol, you must specify one of the four authentication methods supported by Apache Kafka: GSSAPI, Plain, SCRAM-SHA-256/512, or OAUTHBEARER. Using LoadBalancer services or using NodePort services. This allows users to switch on and off the nodeports access for an already running KUDO Kafka cluster. Using the NodePort access method, external listeners make Kafka brokers accessible through either the external IP of a Kubernetes cluster's node, or on an external IP that routes into the cluster. The broker currently looks for an entry named KafkaServer. It changes only the port number used in the advertised.listeners Kafka broker configuration parameter.. Internal load balancers. To enable the external access via node ports: kubectl kudo update --instance=kafka-instance \ -p EXTERNAL_ADVERTISED_LISTENER=true \ -p EXTERNAL_ADVERTISED_LISTENER_TYPE=NodePort. KAFKA_LISTENERS is a comma-separated list of listeners and the host/IP and port to which Kafka binds to for listening. Then for each, it defines the port to listen to and the hostname to advertise in metadata responses. The Kafka broker will receive the number of messages by the Kafka topics. Otherwise, all listeners will use ssl.endpoint.identification.algorithm. SASL-SSL (Simple Authentication and Security Layer) uses TLS encryption like SSL but differs in its authentication process. The default is 0.0.0.0, which means listening on all interfaces. Learn more about Teams The default access method for external listeners is LoadBalancer, thus, in the example below, accessMethod: LoadBalancer. /bin/kafka-topics --create --topic topic-name --bootstrap-server localhost:9092 - will create topic. Different security (SSL and SASL) settings can be configured for each listener by adding a normalised prefix (the listener name is lowercased) to the config name. The Apache Kafka is nothing but a massaging protocol. Next commands should be executed on the kafka container, so first log in into the container by typing: docker-compose exec kafka bash to enter kafka`. Instead of creating separate CGROUP for each Broker node in Kafka cluster, we can use kafka env to make it working. Additionally, a specific service per kafka pod will be created. We will extend this so that the broker first looks for an entry with a lowercased listener name followed by a dot as a prefix to the existing name. In order to access Kafka Brokers from outside the cluster, an additional listener and advertised listener must be configured. There are two ways of configuring external access. listeners * properties are only applicable if you have INTERNAL:// as a listeners configuration, and you want to override the default. Many cloud providers differentiate between public and internal load balancers. For more complex networking, this might be an IP address associated with a given network interface on a machine. The default is 0.0.0.0, which means listening on all interfaces. From docs. listeners Q&A for work. To make Kafka accessible to external client applications, we added an external listener of type loadbalancer. ZooKeeper For more complex networking, this might be an IP address associated with a given network interface on a machine. /bin/kafka-console-consumer --topic topic-name --from . To disable the external access via load balancers: kubectl kudo . localhost:9092kafkalocalhost9092kafkaKAFKA_ADVERTISED_LISTENERSlocalhost9092brokerslocalhost . docker-compose up -d to setup project. To configure Kafka to advertise FQDN and listening on all the IP addresses, add the following text to the bottom of the kafka-env-template. We can create a property file with all the . We need to set the listener configuration correctly. For the CLIENT listener example, the broker would first look for client.KafkaServer with a fallback to KafkaServer, if necessary. This maps 2 names EXTERNAL and INTERNAL (you can use any name you like, I just reused names from your question) to the PLAINTEXT security protocol. Teams. One of the main reasons you might choose SASL-SSL over SSL is . As Kafka distinguished the source of your connections (INTERNAL vs EXTERNAL) by port (e.g., in my case, any connection to the docker container port 9092 would be seen as internal and any connection to port 19092 would be seen as external), the advertised IP addressed will be the one that corresponds to the port (e.g., if you do a connection to . listeners # Configure Kafka to advertise IP addresses instead of FQDN HOST_FQDN=$ (hostname -f) But if you have a complex network, for example, consider if your cluster is on the cloud which has an internal network, and also external IP on which rest of the work can connect to your cluster, in that case, you have to set advertised.listeners property . This is fine when you are connecting in from within an AWS account as that is what this port range and listener is for but from a . KAFKA_LISTENERS is a comma-separated list of listeners and the host/IP and port to which Kafka binds to for listening. Connect and share knowledge within a single location that is structured and easy to search. Just keep in mind that the advertisedPort option doesn't really change the port used in the load balancer itself. listener.name.internal. So for all internal cluster communication happens over what you set in listeners property. The public load balancers will get a public IP address and DNS name . the initial connection occurs on 7000 but then Kafka reports back to the client that it should be using the PRIVATE listener and the traffic reconnects onto the 6000 Private port range (confirmed with tcpdump). The default is 0.0.0.0, which means listening on all interfaces. Edit the KafkaCluster custom resource. can be omitted from the external listener config: listenersConfig: externalListeners: - type: "plaintext" name: "external1" externalStartingPort: 19090 containerPort: 9094 accessMethod: LoadBalancer. While working with the Kafka listeners, we need to set the "advertised.listeners" property. KAFKA_LISTENERS is a comma-separated list of listeners, and the host/ip and port to which Kafka binds to on which to listen.