I Need to Talk to You about Kubernetes GitOps
GitOps is one of the most impactful evolutions that has happened during Kubernetes’ rise to the top. We had been building our kubefirst instant cloud native platform for more than a year when we discovered GitOps. It wrecked us. We decided to throw away a bunch of our work and start over on the new GitOps discipline, and it was the right call.
In a world where microservices and microproducts endlessly blossom throughout your platform’s ecosystem, it becomes increasingly difficult over time to manage these tens, then hundreds and soon thousands of microcomponents. But GitOps is able to reel all this back under control with the simplicity of a single branch of a git repository and some files that describe exactly what’s deployed.
Let’s explore Kubernetes GitOps together.
What GitOps Isn’t
When first reading the term GitOps, many think it’s what they’re already doing. If you use git, and that’s automatically driving your DevOps pipelines — that’s GitOps right? Decidedly not.
Here are some indicators that you’re not doing GitOps:
- If you’re not using Kubernetes, you’re likely not doing GitOps. Oh, stop yelling.
- If you are using Kubernetes but you have a delivery pipeline that includes the command
kubectl applyorhelm install(or any other “push this to Kubernetes” scripting or “sync everything” job), you are doing GitOps’ predecessor that’s recently been coined ScriptOps. - If you are using Kubernetes but you manually run a
kubectl applyorhelm installto deliver content to your cluster, you are doing ClickOps (even when it’s a command). - Even if you’re logging into Argo CD and pointing apps at git directories using the UI, that UI activity is a ClickOps operation and should be avoided when doing GitOps. ClickOps should probably be done in ephemeral environments for tech spikes only.
What GitOps Is
GitOps marries your git provider with your Kubernetes engine and serves as an application control plane for your desired state, which you keep hosted in git. If set up correctly, a GitOps shop can establish a registry of all Kubernetes resources across your organization in a single main branch of a git repository.
When thinking about GitOps, it’s best to imagine an impenetrable wall separating CI (automation pipelines) from CD (delivery), and that barrier is this GitOps main branch. As an engineer, continuous delivery (CD) will no longer be your responsibility; that’s the job of your CD Engine. Your only delivery role in continuous integration (CI) is to establish that desired state.
Why You Should Commit to GitOps #gitpuns
Architecture Simplicity
Kubernetes operations are conducted by handing Kubernetes what you want in YAML format.
Git is very good at versioning a distributed system of flat files like YAML and tracking when and why they change.
When you add a GitOps engine that can pull the desired state from git, apply it to the cluster and report back with sync status in an endless reconciliation loop, it produces a powerful, yet simple architecture that’s rooted in proven distributed technologies that your engineers are already using.
Discoverability
The GitOps approach to Kubernetes asset management and how you scale your applications across your clusters is superior to any alternative. A good GitOps engineer can walk into a new GitOps environment and be almost immediately impactful.
The discipline allows you to register a tree of applications represented as YAML files in a single main branch of a single git repository. If you bootstrap your clusters this way, anyone who is familiar with GitOps will be able to follow the tree and discover all of the desired state and already know how the content is delivered.
Your GitOps engineer will become one of the most valuable, and somehow also the most replaceable, members of your organization due to the cluster registry’s discoverability.
This should give a feel for what a GitOps registry would look like: https://github.com/kubefirst/GitOps-template/tree/1.10.9/registry
Security
CI tools are common attack vectors for bad actors. Any access they have is at risk during a breach. With GitOps, your CI tool won’t need access to your cluster. Instead your cluster will pull its deployments and configurations using a read-only connection to your GitOps git repository. Securing that git repository becomes the new game, which is much easier to manage, especially if you manage your git repositories in Terraform.
System Audit Log
If every change to your infrastructure gets applied because of a pull request in a single repository, then the history of pull requests being applied is the audit log of everything that’s ever happened, along with who approved it and everything else that git provides on its own. This is a much more convenient process for engineers than what ScriptOps organizations can typically offer.
Rollback
Another advantage of hosting your declared desired state in git is its ability to roll back any problem that’s introduced, in many cases, by merely reverting the problematic commit in the GitOps repository. It’s a simple engineering skill that can now solve some big problems.
Disaster Recovery
If your cluster always gets and syncs apps from a single git repo registry, replacing that cluster is as simple as pointing your new cluster to that same registry point, and it will make itself the same cluster. There are some devils in those details with host names and traffic concurrency, but this is one of your goals as a GitOps admin.
If you manually delete a deployment in Kubernetes and it’s part of GitOps, it comes back immediately. The GitOps repo main branch source will continually attempt to become the actual state without any scripted jobs anywhere. This is an excellent posture to keep your service availability high.
How GitOps Works
Git Repositories
You need to create a GitOps repository that has a folder that you can register your cluster against. For a single cluster, you can just call this folder “registry.”
GitOps Engines
GitOps works by placing a GitOps engine like Argo CD or Flux CD into your Kubernetes cluster. You’ll need to configure the engine with read-only access to your GitOps git repository. When you install this GitOps engine, you’ll point it to this directory in order to hydrate your clusters with apps.
GitOps Architecture Decisions
Once you provision your management cluster (a centralized cluster that manages and orchestrates your management systems and infrastructure), you have a GitOps architecture decision to make about the workload clusters that run your applications (production cluster, preprod, etc.) and whether they should each have their own GitOps engine or use that of the management cluster.
Distributed GitOps Architecture
The distributed GitOps architecture (sometimes called bootstrapped or standalone) is the only model that prevents your management cluster from needing access to your production cluster. For organizations with higher security requirements or who are subject to a compliance boundary, this is an ideal posture.
The distributed GitOps architecture requires that each workload cluster have its own dedicated instance of Argo CD. This won’t be much of a burden to your admins, provided you have single sign on implemented throughout the Argo CD instances.
This pattern also distributes the resource demand on Argo CD to each cluster, allowing it to horizontally scale those GitOps computations across the ecosystem. The smaller blast radius when adjusting the GitOps engine itself is also a nice benefit.
Centralized GitOps Architecture
Some organizations go with a centralized GitOps architecture, where the management cluster Argo CD instance has push access to enforce desired state onto the downstream workload clusters. This architecture allows you to view all environments’ applications from a single Argo CD instance, which can be a convenience in a number of ways, lending itself to GitOps templating techniques like ApplicationSets.
Abstracting Secrets in GitOps
If all your system is defined declaratively in git, how do you deal with secrets? The external secrets operator is probably the best tool for this problem. It provides a custom resource definition (CRD) that lets you define external secret resources that map to your secret tool of choice like Hashicorp Vault or your cloud secret store. This allows you to reference your secrets in git, without actually placing them there.
Automating GitOps Delivery in Your CI Pipelines
To deliver your app to your environments in your CI pipeline, you simply need to update the YAML file that represents that application instance in your GitOps repository. This is really all that is meant by the term “desired state.” It’s a file in a folder where you can define what should be in Kubernetes.
Choosing Your GitOps Technology
Choosing the right GitOps driver is one of your most important architectural decisions when designing your new Kubernetes platform. OpenGitOps offers a set of principles that define the GitOps discipline in a vendor-agnostic way to help guide you through this decision-making process.
See GitOps Automation Workflows Locally in Under 5 Minutes
Our kubefirst CLI provisions free and fully automated cloud native open source GitOps platforms.
We’re building a community of users who are using the same free open source cloud native tools in the same approximate ways. These users can contribute, help each other, and help produce a frictionless and fully automated cloud native core ecosystem.
We have a local platform and an expanding set of cloud platforms that are fully automated from the start. The open source tools that we provision are all preconfigured to work well with each other. You’ll be able to explore scalable, unwrapped open source tools like Argo CD, Argo Workflows, Vault, Terraform, Atlantis, External Secrets Operator, Cert Manager, External DNS and many others.
Because our architecture provides you with your own GitOps repository that powers your new environment, you’re free to take your new platform in any direction you choose, and you can always leave us behind if you choose to. We won’t be in your way. It’s a completely GitOps-based offering where you can remove our opinions and add your own with a pull request to the GitOps repository that you now own.
The fastest way to check out the platform is on the kubefirst local variation, where you can have a full, free cloud native ecosystem in just five minutes on your own localhost. Give your Docker runtime at least 5 CPU / 5 GB memory for a good time, if you can. Then just run:
brew install kubefirst/tools/kubefirst
kubefirst local
Our cloud platform details and other install types can be found at https://docs.kubefirst.io/
The cloud native platform includes a sample application called metaphor-frontend that demonstrates how to run GitOps delivery using Argo workflows that come with batteries included — prebuilt GitHub Actions that are running on private GitHub runners in your local cluster. We provide some automation to build and publish containers and charts, set GitOps-desired state and automatically manage versions for releases, even locally.
The metaphor-frontend app also demonstrates how to use the rest of the platform, like leveraging secrets from Vault, Helm values overrides, using an ingress, TLS certificate automation, DNS management automation, release management automation and so much more.
Run a quick install, shoot us a GitHub star for the free management platform, join our workspace and help us build an awesome community of engineers using the same approximate cloud native tools the same way. You really won’t believe what Kubernetes GitOps can do.
