In this post, we will describe the steps we followed to enable Kafka broker pods within kubernetes cluster using microk8s in Ubuntu 18.04.<\/p>\n
First step was to enable firewalls in ubuntu using ufw if not done already. Lets start with Install microk8s.<\/p>\n
\u00a0<\/strong><\/p>\n\n $ sudo snap install microk8s –classic<\/strong> configure firewall to allow pod-to-pod and pod-to-internet communication<\/p>\n $ sudo ufw allow in on cni0 && sudo ufw allow out on cni0<\/strong> enable addons for microk8s<\/p>\n $ microk8s.enable dashboard dns<\/strong><\/p>\n Enable loadbalancer addon. We require to assign a pool of IP address to assign for new loadbalancer services created within the cluster. We configured a range of 20 IP addresses.<\/p>\n $ microk8s.enable metallb<\/strong><\/p>\n start the cluster.<\/p>\n $ microk8s.start<\/strong><\/p>\n $ microk8s.status<\/strong><\/p>\n\n If you havent received any error message, then the cluster is started successfully and we can proceed to access the dashboard.<\/p>\n\n\n $ microk8s.kubectl get all –all-namespaces<\/strong><\/p>\n\n\n <\/p>\n\n\n the above command will give us the cluster ip address to access the dashboard. The name of the services will be service\/kubernetes-dashboard. The port will be 443 unless you change them through configuration. $token=$(microk8s.kubectl -n kube-system get secret | grep default-token | cut -d ” ” -f1)<\/strong> On successful login to dashboard, we can start to create the services. kind: Deployment<\/em> name: zookeeper-deployment-1<\/em> selector:<\/em><\/p>\n matchLabels:<\/em> metadata:<\/em> app: zookeeper-1<\/em> containers:<\/em> ports:<\/em><\/p>\n metadata:<\/em> labels:<\/em> spec:<\/em><\/p>\n ports:<\/em> selector:<\/em> Once we click the upload button, unless you have any format error reported, we should be able to see a successful zookeeper service running in the dashboard overview screen. Format errors are generally alignment issues and are simple to fix through the guidance provided by the error message.<\/p>\n To create kafka pods, click the + sign on the dashboard to provide the configuration for the service. kind: Deployment<\/em> spec:<\/em><\/p>\n selector:<\/em><\/p>\n matchLabels:<\/em> template:<\/em><\/p>\n metadata:<\/em><\/p>\n labels:<\/em> – name: kafka-service<\/em> ports:<\/em><\/p>\n – containerPort: 9092<\/em><\/p>\n env:<\/em><\/p>\n – name: KAFKA_LISTENERS<\/em> Once we click the upload button, unless you have any format error reported, we should be able to see a successful kafka pods running in the dashboard overview screen. Alternatively, you can view the information through the below command from console window.<\/p>\n\n\n $microk8s.kubectl get pods –output=wide<\/strong><\/p>\n\n Format errors are generally alignment issues and are simple to fix through the guidance provided by the error message. If you have any errors, you may check the Finally we require to configure a loadbalancer service for our pods to route the request to the pods that originate from outside the cluster. Execute the following command to create, link the loadbalancer service to our pods.<\/p>\n\n\n $ microk8s.kubectl expose deployment kafka-service –type=LoadBalancer –name=my-service<\/strong><\/p>\n\n You can view the status of the new loadbalancer service in the dashboard. Alternatively enter the below command to view the details of the service.<\/p>\n\n\n $ sudo microk8s.kubectl get services<\/strong> Name: my-service<\/em> If you happen to see all the configurations are correct in the loadbalancer service as shown above and the dashboard does not report of any failures across the three items we started, we are good to use the kafka brokers deployed in our pods. Notice the endpoints value above. It is the IP address assigned to the pods. Every time we restart the pods, the IP address will get changed. This may not be liked by applications that require IP address to be almost static and in such scenarios, we will be having the benefit of the loadbalancer service. <\/p>\n But you may be inclined to test it before you start to use it. We used a test utility called kafkacat. install it like below.<\/p>\n\n\n $ apt-get install kafkacat<\/strong><\/p>\n Once it is installed, we start the consumer first in a seperate console.<\/p>\n $kafkacat -b 10.182.183.230:9092 -t ceyark-test<\/strong><\/p>\n we start the producer in seperate console.<\/p>\n $ echo hello | kafkacat -b 10.182.183.230 -t ceyark-test<\/strong><\/p>\n If everything goes fine, we should see the “hello” message in the consumer console. Once you are done, issue the following command to stop the cluster.<\/p>\n $ microk8s.stop<\/strong><\/p>\n\n With the configuration files in hand ready, we were able to create the kafka docker clusters in less than <\/p>\n","protected":false},"excerpt":{"rendered":" In this post, we will describe the steps we followed to enable Kafka broker pods within kubernetes cluster using microk8s in Ubuntu 18.04. First step was to enable firewalls in ubuntu using ufw if not done already. Lets start with […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[138],"tags":[139],"class_list":["post-4214","post","type-post","status-publish","format-standard","hentry","category-ceyarkblogs","tag-ceyark"],"_links":{"self":[{"href":"https:\/\/www.ceyark.com\/index.php\/wp-json\/wp\/v2\/posts\/4214"}],"collection":[{"href":"https:\/\/www.ceyark.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ceyark.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ceyark.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ceyark.com\/index.php\/wp-json\/wp\/v2\/comments?post=4214"}],"version-history":[{"count":10,"href":"https:\/\/www.ceyark.com\/index.php\/wp-json\/wp\/v2\/posts\/4214\/revisions"}],"predecessor-version":[{"id":4791,"href":"https:\/\/www.ceyark.com\/index.php\/wp-json\/wp\/v2\/posts\/4214\/revisions\/4791"}],"wp:attachment":[{"href":"https:\/\/www.ceyark.com\/index.php\/wp-json\/wp\/v2\/media?parent=4214"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ceyark.com\/index.php\/wp-json\/wp\/v2\/categories?post=4214"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ceyark.com\/index.php\/wp-json\/wp\/v2\/tags?post=4214"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
$ sudo usermod -a -G microk8s $USER<\/strong>
$ su – $USER<\/strong><\/p>\n\n
$ sudo ufw default allow routed<\/strong><\/p>\n
\nWhen you access the dashboard in a browser, we require to login. We use the token approach to login and we obtain token through the two commands below executed in sequence.<\/p>\n\n\n
$microk8s.kubectl -n kube-system describe secret $token<\/strong><\/p>\n\n
\nTo create Zookeeper service, click the + sign on the dashboard to provide the configuration for the service.
\nThe configuration we provided is given below.<\/p>\n\n\n
apiVersion: apps\/v1<\/em>
metadata:<\/em><\/p>\n
spec:<\/em><\/p>\n
app: zookeeper-1<\/em>
template:<\/em><\/p>\n
labels:<\/em><\/p>\n
spec:<\/em><\/p>\n
– name: zoo1<\/em>
image: bitnami\/zookeeper<\/em><\/p>\n
– containerPort: 2181<\/em><\/p>\n
env:<\/em><\/p>\n
– name: ZOOKEEPER_ID<\/em>
value: “1”<\/em>
– name: ZOOKEEPER_SERVER_1<\/em>
value: zoo1<\/em>
– name: ALLOW_ANONYMOUS_LOGIN<\/em>
value: “yes”<\/em><\/p>\n
—<\/em>
apiVersion: v1<\/em>
kind: Service<\/em><\/p>\n
name: zoo1<\/em><\/p>\n
app: zookeeper-1<\/em><\/p>\n
– name: client<\/em>
port: 2181<\/em>
protocol: TCP<\/em>
– name: follower<\/em>
port: 2888<\/em>
protocol: TCP<\/em>
– name: leader<\/em>
port: 3888<\/em>
protocol: TCP<\/em><\/p>\n
app: zookeeper-1<\/em><\/p>\n\n
\nThe configuration we provided is given below.<\/p>\n\n\n
apiVersion: apps\/v1<\/em><\/p>\n
metadata:<\/em>
name: kafka-service<\/em><\/p>\n
app: kafka-service<\/em><\/p>\n
app: kafka-service <\/em>
spec:<\/em>
replicas: 1<\/em>
containers:<\/em><\/p>\n
image: wurstmeister\/kafka<\/em><\/p>\n
value: INTERNAL:\/\/:32323,EXTERNAL:\/\/:9092<\/em>
– name: KAFKA_ADVERTISED_LISTENERS<\/em>
value: INTERNAL:\/\/kafka-service:32323,EXTERNAL:\/\/10.152.183.230:9092<\/em>
– name: KAFKA_ZOOKEEPER_CONNECT<\/em>
value: 10.152.183.182:2181<\/em>
– name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP<\/em>
value: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT<\/em>
– name: KAFKA_CREATE_TOPICS<\/em>
value: ceyark-test:1:1<\/em>
– name: KAFKA_INTER_BROKER_LISTENER_NAME<\/em>
value: INTERNAL<\/em><\/p>\n\n
\nlogs in the dashboard through the logs options against the pod. The logs are generally good enough to troubleshoot and fix the issues.<\/p>\n
$ sudo microk8s.kubectl describe svc my-service<\/strong><\/p>\n
Namespace: default<\/em>
Labels: <none><\/em>
Annotations: <none><\/em>
Selector: app=kafka-service<\/em>
Type: LoadBalancer<\/em>
IP: 10.182.183.167<\/em>
LoadBalancer Ingress: 10.182.183.230<\/em>
Port: <unset> 9092\/TCP<\/em>
TargetPort: 9092\/TCP<\/em>
NodePort: <unset> 31985\/TCP<\/em>
Endpoints: 10.1.2.122:9092<\/em>
Session Affinity: None<\/em>
External Traffic Policy: Cluster<\/em>
Events:<\/em>
Type Reason Age From Message<\/em>
—- —— —- —- ——-<\/em>
Normal IPAllocated 5s metallb-controller Assigned IP “10.182.183.230”<\/em>
Normal nodeAssigned 5s metallb-speaker announcing from node “ceyark-macserv1<\/em>“<\/p>\n\n
\n10 min and this utility helps with our development tasks.<\/p>\n\n\n\n