Photo by Aldebaran S on Unsplash
Many developer teams working with Kubernetes in production have the same development issues. They try deploying their environment locally by using Docker-compose to deploy their entire stack locally. This can go wrong in many ways.
Your Docker-compose YAML file is not equal to the Helm/Kustomize charts you run in production. You will stumble upon differences between them after you deploy to production and forget about an environment variable that you added.
Other teams might try to go the extra mile and deploy a local Kind/K3s instance to solve this, and deploy the Helm/Kustomize files there, but this can lead to the following issues:
Developer laptops aren’t powerful enough. Each laptop needs to be able to run the whole stack including services, databases and more.
Your developers might have different operating systems and/or architectures. Your OS may not be the Kubernetes node OS, and other developers might not have the same architecture. When the Mac M1 came out, many development environments stopped working completely.
Your local environment might not have the same policies or requirements as your Kubernetes cluster. Network policies mutating webhooks and Policy managers such as Kyverno/Datree and cloud services will forever differentiate your local env from production.
Monitoring sucks or is nonexistent locally.
For these reasons, developers might deploy their code to a Kubernetes “dev” cluster, with an environment (namespace) per developer or feature branch.
Development with Kubernetes
This is definitely a good way to stay with a configuration which is similar to production but leads to a slower development cycle. To see real changes, developers need to:
Develop locally
Push code to git.
Wait for the code to be built and pushed to a repository.
Wait for tests.
Wait for the namespace and Kubernetes resources to be provisioned.
This is what’s called the “inner dev loop” and can take between a minute or two to 20 minutes or more depending on the pipelines they have in their organization, which can waste time and leave developers frustrated.
Fix the dev loop
To combat these issues there has been a rise in “Developer Experience” tools for Kubernetes debugging and speeding up the development cycle such as tilt.dev, garden.io and okteto. These tools are very helpful and focus on the creation and administration of preview and dev environments.
There is another branch of developer tools which are focused on connecting your Kubernetes cluster to your local machine, so developers can work “inside” the cluster.
With the developer running a service and making code changes locally, a tunnel transfers their development environment to the cluster with a 2-way connection. This means all networking services and DNS names available to the pod are now available to their local workstation, and their local process has tunnelled all traffic from the application’s service from the cluster. They can then debug networking issues, use profiling tools and add breakpoints to their code. This is different from other tools that “sync” code to a running pod as here all traffic to the application is actually tunnelled to the developer’s workstation, which works through a reverse proxy. Developers can then push code only when they need to, at the end of writing the feature or fix.
I’ll be talking about the architecture of one of these tools, which is a free and open-source tool called KubeTunnel, which as a disclaimer, I’m a maintainer of.
Architecture
There are 2 main components in KubeTunnel’s architecture:
A client-side component (CLI) that forwards some or all of the Kubernetes cluster services to your local environment and adds them to your local DNS. This achieves being able to connect from your local process to the cluster services using the local cluster DNS names such as “service”.”namespace”.
A cluster-side component that creates a tunnel on the cluster that forwards the local process to it. This makes it so other services can connect to the process on the developer’s workstation, completing the 2-way connection to the cluster. This component is deployed beforehand as a Kubernetes controller with all RBAC permissions needed, so it can create tunnels instead of the specific developer who might not have the required permissions to patch services, create deployments etc.
Sidecar or deployment?
The cluster-side “tunnel” component can be created through a sidecar-based or deployment-based approach. With a sidecar, a mutating controller patches the wanted deployment on the cluster and manipulates it to add another container. This can lead to a mess in your environment, as the “injection” leads to the restart of the application pod, changes in resource needs and more. For more information as to why sidecars can be invasive in your environment, you can check out Istio’s new ambient mode blog.
The advantage that KubeTunnel has over other tools in the space is that it does not use sidecars, but takes a different approach to the issue, adding a whole separate deployment and manipulating the service instead, meaning that the application deployment still exists in its entirety. A Kubernetes controller watches the client-server connection, and if it stops, the service then switches back to forward traffic to the running pod in the cluster. At no point does the existing deployment need to restart or be patched.
KubeTunnel Architecture
How do I adopt KubeTunnel into my Workflow?
For developers, using KubeTunnel makes for the fastest development cycle possible. No builds, file synching or code updates are needed. KubeTunnel can and should be used hand in hand with other tools that manage developer environments, or just installed on Non-production clusters as a development tool with CI/CD pipelines managing and creating the environments. It can also be installed just for debugging purposes, to be able to replicate issues from the cluster “locally”.
For general discussions or questions about KubeTunnel, you’re welcome to join our Slack. If you need any DevOps help, feel free to send us a message on our website. For more information about the inner dev loop, you can check out this blog. WeDcode is a boutique software and DevOps consulting firm. We have vast experience with companies and startups with complex issues in Kubernetes both on the cloud and on-premise.
Kommentare