Kubernetes: Pod-to-Pod Communication
So, one thing that I’ve been working on at Admiral a lot recently is getting applications to communicate with one another inside our cluster, at first I found an array of issues. A primary example being how I assumed that our initial/demo cluster allowed network policies to exist, only after communicating with someone in the cloud team, I learned that the cluster didn’t allow for network policies to exist.
Anyhow, after talking with engineers at Microsoft, since we’re working with AKS, I found that in the examples provided they were using tools such as CoreDNS. I then learned a lot more, about networking inside the cluster, the variety of approaches & potential implementations, to be honest, a number of topics were a little too infrastructure heavy for my liking, although having a fundamental understanding of everything, I believe I was able to understand the contents. Anyhow, after conducting an array of research, I found that the solution was incredibly simple & I felt like a fool for not trying that approach first.
As it turns out, a simple approach is to just make a call from one pod to another, through the use of Kube-DNS. The steps that I followed was to include 4 pods, all in a different namespace, specific to the application, I had one application with no network policies at all, another that allowed incoming traffic from a specific app, using labels. Then of course another application that denied all traffic from all pods except for the essentials, aka kube-system.
So, I gained access to the one pod that was allowed to send data to another pod; however, the first pod I targeted was the one that denied all communication(s) except for the essentials. To do so, I gained access to the Linux server inside the pod, to do so, I entered the following command(s):
$ kubectl exec -it <pod name> --namespace=<relevant namespace> -- bash $ curl <deny_all_service>.<deny_all_namespace>:8080
After having tested this, I was thrilled to see that it worked, only I had expected a refused connection type error, only I discovered that it resulted in a time out. I proceeded to test the different possible varieties, to learn that what I thought would work, in fact did work, I was over the moon since this was really a matter of trial & error, I had done the research & whatnot, so it was a matter of just applying what I had learned.
Obviously, I learned a lot from this experience, I was initially quite stressed that the network policies didn’t work in the initial cluster, but after talking to a member of the clout team, I was actually somewhat thrilled that this was actually to be expected. I guess as a part of my conclusion, I should highlight how communication is essential, despite the fact that we’re working in different geographical locations, it just illustrates that clear & concise communication is key to progress. If I didn’t speak to one of the cloud guys, I would’ve probably been scratching my head all day, stressing about how ‘this should work… But it doesn’t?! What?!‘.
But beyond that, having learned from my own experience, I’d highly recommend trying to tackle the smaller & more simplistic approaches first, as this may seem like common sense anyway. But in my defence, my team isn’t currently in charge of how the cluster is being managed, so I was totally unaware that the ‘demo cluster’ didn’t allow network policies to be applied, hence why I dived deep into research. The pipelines to create these policies appeared as successful, if my memory serves me correctly, I could see them inside the cluster, etc, all seemed well, they just didn’t work. That’s why I tried looking into alternative solutions, the initial solution that you’d expect would work didn’t, so I tried to proceed by looking at load balancers, NGINX, etc.
Looking back, I can’t help but feel a little embarrassed or at least ashamed that I didn’t simply ask for more help from my colleagues. I’m not by any means a know it all, there’s many technical areas that I lack knowledge in, an example being how only recently I’ve stumbled across the term ‘RMI-IIOP‘, yet I’ve been trying my best to delve into how to develop & architecture distributed applications aka microservices.