GitOps explained: ArgoCD, desired state in Git, drift corrected

There are two ways to deploy an application on Kubernetes. The first: someone, or a CI script, runs kubectl apply or helm upgrade commands against the cluster. The second: you write in a Git repository what the cluster should contain, and a program inside the cluster makes it match. The second is called GitOps, and it fixes several problems that the first sweeps under the rug.

The principle in one sentence

Git is the source of truth for the desired state; a controller constantly compares that state with the actual state of the cluster and corrects the gap. A deployment is no longer an action you launch, it is a state you declare.

The community has formalised four principles, gathered under the name OpenGitOps:

• Declarative: the system is described by what it should be, not by a series of commands.
• Versioned and immutable: the desired state is stored in a way that keeps the full history, which Git does very well.
• Pulled automatically: an agent pulls changes from the source, rather than an external tool pushing them.
• Continuously reconciled: the agent compares permanently and corrects divergences.

Pull rather than push

In a classic “push” deployment, the CI holds credentials able to change the cluster. Whoever compromises the CI compromises the cluster, and those credentials live outside of it. GitOps reverses this: the controller, installed in the cluster, has read access to the repository, and it is the one applying the changes. The cluster needs no open door for deployments, and the CI no longer needs any access to it.

ArgoCD, the central piece

ArgoCD is one of the most widespread GitOps controllers for Kubernetes. It is itself made of a few services: an API server and web interface, an application controller that compares and synchronises, a service that reads the repository and generates manifests (the repo server), and a cache. The unit of work is called an Application, which describes four things:

• the source: the repository address, the folder path and the revision to follow (a branch, a tag);
• the content type: raw manifests, a Helm chart or a Kustomize configuration;
• the destination: the target cluster and namespace;
• the sync policy: manual, or automatic with options.

Each application has two states to watch. The first is sync status: Synced if the actual state matches Git, OutOfSync otherwise. The second is health: Healthy if the resources are working, Degraded or Progressing otherwise. An application can be perfectly synced and broken, because Git describes a broken configuration. The two are independent.

Drift, and how it disappears

Drift is the gap between what Git describes and what is really running. Its origin is mundane: at three in the morning, someone edits a deployment by hand with kubectl edit to resolve an incident, and forgets to carry the change back. A few weeks later, nobody knows why production does not look like the repository.

With automatic sync, two options deal with the problem:

• selfHeal: true: if the actual state departs from Git, for instance after a manual change, the controller puts it back in line;
• prune: true: a resource removed from Git is deleted from the cluster, instead of staying behind as an orphan.

This changes the way you work. Fixing an incident goes through a commit, which stays in the history with its author and date. Manual changes no longer stick, and that is intended. It is not always comfortable the day you want to try something in an emergency, but it is what guarantees the repository tells the truth.

If it is not in Git, it is not production. If it is in Git, it is what should be running.

At the scale of a small k3s cluster

GitOps is not reserved for large platforms. On a single-node k3s cluster hosting a site and a few services, the pattern comes down to few pieces:

1. A configuration repository holds one folder per application: a Helm chart or Kustomize manifests, with the environment’s values.
2. ArgoCD is installed in its own namespace.
3. One Application per service points to its folder, with automatic sync.
4. To ship a new version, you change the image tag in the repository, through a pull request.

ArgoCD is not free in resources: several processes run permanently, and on a small machine you notice. That is the price of continuous reconciliation, to be set against what you save in incidents caused by a state nobody knows.

Shipping a new version

The usual flow separates two repositories, or two folders: the application code, and the deployment configuration. The code’s CI builds the image, publishes it, then changes the tag in the configuration repository, either by opening a pull request or through a dedicated tool. ArgoCD sees the change and deploys.

Two habits help a lot. First, use immutable tags, or better, the image digest, so that what is deployed is exactly what was tested. Second, never use the latest tag: it makes the desired state ambiguous, which is precisely what you want to avoid.

Rollback, almost free

Undoing a deployment amounts to undoing a commit: git revert, a merge, and the controller brings the cluster back to the previous state. No special procedure to dig out of a wiki, no command to reconstruct from memory under pressure. Git’s history is also the history of deployments.

Secrets, the delicate point

You cannot simply commit a password in clear text to a configuration repository. Three common approaches: encrypt secrets in the repository (Sealed Secrets, SOPS), reference them from an external vault (External Secrets Operator), or create them outside the GitOps flow for what is really sensitive. None is free, but all beat a clear-text secret in the history.

The pitfalls you meet

• Permanent differences on fields that other controllers change (a replica count managed by an autoscaler, for example). You declare them as ignored in the application’s configuration.
• Creation order: some resources must exist before others (a custom resource definition before its use). ArgoCD offers sync waves for this.
• `prune` enabled too early, on a folder badly reorganised, deleting more than expected.
• A repository turned into a junk drawer, with no structure, where you can no longer tell what belongs to what.

To start, one application, one repository, manual sync for a few days to look at the differences, then automatic mode. You learn faster by watching the gap appear than by reading the documentation.