Deploys
A deploy gets a new version of an app into a Kubernetes cluster. Six ways are common, and Argo CD's agent mode adds a seventh. They differ in where the credentials live, which side opens the connection, what corrects drift, which branches can deploy, what runs in the cluster, and what stops when the central side is down. Start with a service account token in CI: it's the simplest to follow and needs nothing in the cluster. Move to a GitOps controller per cluster when the cluster has to match git, or when nothing may connect into it. The signals under Choosing cover the other options.
Installing a runner, agent or controller in a cluster is platform setup, which the Cloud Handbook covers.
Push and pull
In a push, a pipeline job runs kubectl apply or helm upgrade against the cluster's API, and the change reaches the cluster when the job runs. In a pull, a controller in the cluster reads the deployment configuration from git and applies it again and again, so it can also undo changes made by hand. A runner or agent in the cluster sits between the two: the cluster opens the connection, as in a pull, but a pipeline job still decides what to deploy and when, as in a push.
Comparison
| Option | Credentials | Opens the connection | Corrects drift | Branches that can deploy | Runs in the cluster | When the central side is down |
|---|---|---|---|---|---|---|
| Service account token in CI | A CI variable, for one namespace | CI, into the cluster's API | Nothing | Protected ones, with a protected variable | Nothing | No deploys while CI is down |
| Cloud token in CI | A CI variable, for every cluster the token sees | CI, into the cloud's API and the cluster's | Nothing | Protected ones, with a protected variable | Nothing | No deploys while CI or the cloud's API is down |
| Runner in the cluster | The runner's service account, in the cluster | The runner, out to CI | Nothing | Protected ones, with a protected runner | A runner, and a pod per job | No deploys while CI is down |
| Agent in the cluster | The agent's service account, in the cluster | The agent, out to GitLab | Nothing | Any branch on GitLab.com, unless the environment is protected | An agent | No deploys while GitLab is down |
| GitOps controller per cluster | Read access to git, in each cluster | The controller, out to git | The controller, where turned on | What merges to the branch the controller watches | A controller in every cluster | The cluster keeps what it last applied |
| Central GitOps controller | The central cluster, for every cluster | The central controller, into each cluster's API | The central controller, where turned on | What merges to the branch the controller watches | Nothing | No deploys or drift correction in any cluster |
| Central UI with agents | In each cluster, none in the central cluster | Each agent, out to the central cluster | Argo CD in each cluster, where turned on | What merges to the branch the controller watches | An agent and Argo CD | In autonomous mode, every cluster keeps deploying from git |
Service account token in CI
A pipeline job runs kubectl apply or helm upgrade --install with the token of a Kubernetes service account. A Role and a RoleBinding limit that account to the app's namespace (RBAC), and the token sits in a CI variable. A token from a kubernetes.io/service-account-token Secret never expires (long-lived API token).
Upsides
- The deploy is a command in the pipeline file, so the team reads what it does there and sees what ran in the job log.
- Nothing extra runs in the cluster.
- RBAC limits the token to one namespace, so a leaked token can't change other apps or the cluster itself.
- In GitLab, a protected variable reaches only pipelines on protected branches and tags, on every tier (protected variables), so a job on a feature branch can't deploy.
- The deploy job can wait for the rollout and fail the pipeline when it doesn't finish, as in Deploying.
Downsides
- The token is a long-lived credential, and rotating it means changing the cluster and the CI variable together.
- Anyone who can read the variable, or run a job that gets it, can also deploy outside the pipeline, from their own machine.
- CI connects into the cluster's API, so an API that only accepts an allow list of addresses, such as GKE's authorized networks (authorized networks), has to list the runners. GitLab.com's hosted runners have no static IP addresses (IP range), so that takes runners of your own.
- Each cluster, and each namespace with its own service account, needs a token in its own variable.
- Nothing corrects drift: a change made by hand stays until a deploy sets that field again, and a field the manifests don't set stays until someone removes it.
Cloud token in CI
A pipeline job holds a token for the cloud's API and fetches the cluster's kubeconfig with it, such as with gcloud container clusters get-credentials or doctl kubernetes cluster kubeconfig save. The job can also trade a short-lived ID token for cloud credentials through OpenID Connect, so no token is stored in CI at all. GitLab issues ID tokens on every tier, and each one expires with its job (ID tokens).
Upsides
- One token deploys to every cluster in the account, so a new cluster needs no new CI variable.
- Access to the clusters is granted and revoked with the cloud's roles, like the rest of the account.
- With OpenID Connect, no long-lived credential exists to leak or rotate.
Downsides
- One token reaches every cluster it can see: on GKE, "an IAM role grants privileges across all clusters in the project" (access control). A leaked token exposes all of them.
- On GKE, either a cloud role or Kubernetes RBAC can grant access, so limiting the token to one namespace takes a minimal cloud role plus RBAC for that namespace.
- Deploys depend on the cloud's API as well as the cluster's.
- CI still connects into the cluster's API, so the allow list has to include the runners, as with a service account token.
- Nothing corrects drift.
Runner in the cluster
A CI runner runs inside the cluster and starts every job as a pod there, such as GitLab Runner's Kubernetes executor or GitHub's Actions Runner Controller. The runner connects out to the CI service and asks it for jobs. A deploy job gets its rights from the service account of its pod.
Upsides
- CI stores no cluster credential: the rights live in a service account inside the cluster.
- Nothing connects into the cluster, so the cluster's API needs no allow list entry for CI.
- In GitLab, a protected runner only takes jobs on protected branches and tags, on every tier (protected runners). On GitHub, a runner group can be limited to selected repositories, and in GitHub Enterprise Cloud to selected workflows on a branch.
- Build and test jobs can run on the same runner, on the cluster's spare capacity.
Downsides
- Every job the runner takes runs inside the cluster, so a broken or malicious job runs next to production.
- A runner that takes jobs from unprotected branches gives every branch the deploy rights of its service account.
- Job pods use the cluster's CPU and memory, and the runner itself needs rights to create pods in its namespace.
- Nothing corrects drift.
Agent in the cluster
An agent runs in the cluster and keeps a connection open to the CI service, and pipeline jobs send their kubectl or Helm commands through that connection. The GitLab agent for Kubernetes works this way on every tier, Free included, and its configuration lists the projects and groups whose jobs may use it.
Upsides
- Nothing connects into the cluster, and CI stores no cluster credential.
- Jobs on any runner, GitLab.com's hosted runners included, can reach a cluster behind a firewall or NAT.
- Jobs from every project and group that its configuration authorizes can use the same agent.
Downsides
- The agent's Helm chart gives its service account
cluster-adminby default, and every CI job that uses the agent inherits those rights (install, CI/CD workflow). Narrowing the agent takes a role of your own, set withrbac.useExistingRole, and giving each job other rights through impersonation needs Premium. - On GitLab.com, any branch of an authorized project can use the agent.
protected_branches_onlyis only documented for GitLab Self-Managed and Dedicated, and marked as not ready for production use there. - Limiting the agent to jobs for one environment works on Free, but any branch can name that environment unless it's a protected environment, which needs Premium.
- The agent only works with GitLab, so moving to another CI service means another way to deploy.
- Nothing corrects drift. For GitOps with the agent, GitLab recommends Flux.
GitOps controller per cluster
A GitOps controller, such as Flux or Argo CD, runs in each cluster, reads the deployment configuration from git and applies it. The pipeline builds the image and commits its tag to the configuration repository, as in projects and configuration, and the controller deploys it. GitOps covers that repository, promotion and rollback.
Upsides
- Nothing connects into the cluster: the controller connects out to git and the registry, and CI holds no cluster credential.
- Whatever merges to the branch the controller watches gets deployed, so branch protection and review in the configuration repository decide who deploys.
- The controller can correct drift: Flux applies each Kustomization again at every interval (interval), and Argo CD does with
selfHeal: true(self-healing). - Each cluster deploys on its own, so a broken controller in one cluster doesn't stop deploys to the others.
- While the git host is down, each cluster keeps running what it last applied.
Downsides
- Every cluster runs a controller. Flux v2.10.0's four default controllers request 350m CPU and 256Mi of memory together (install manifest). Argo CD v3.6.1's default install runs seven workloads (install manifest) and sets no resource requests, so you size them yourself.
- Not every controller corrects drift by default: a Flux HelmRelease corrects it only with
spec.driftDetection.mode: enabled(drift detection), and Argo CD only withselfHeal: true. - The pipeline ends at the commit, so a rollout that doesn't finish can't fail it, and the controller's notifications have to report it instead.
- Every release takes a commit to the configuration repository.
- Seeing what runs everywhere means checking the controller in each cluster.
Central GitOps controller
One Argo CD deploys to many clusters: it reads git and applies changes through each cluster's API. argocd cluster add creates a service account in the target cluster, binds it to an admin-level ClusterRole, and stores its token in the central cluster (register a cluster, clusters).
Upsides
- One UI shows every app in every cluster, with syncs and rollbacks in one place.
- The target clusters run no controller.
- Single sign-on, access rules and notifications are set up once.
- With
selfHeal: true, it corrects drift in every cluster.
Downsides
- The central cluster holds a credential for every cluster, with admin rights by default, so whoever controls it controls all of them.
- It connects into every cluster's API, so each cluster's allow list has to include it.
- Every deploy depends on it: while it's down, no cluster gets deploys or drift correction.
- It watches and caches the resources of every cluster, so its load grows with each cluster it adds (argocd-agent architecture).
Central UI with agents
Argo CD's agent mode, the argocd-agent project, reverses the connection of a central controller. An agent in each cluster connects out to a central principal, and Argo CD's application controller runs in each cluster. In managed mode, Applications are created centrally and the agent copies them into its cluster (managed mode). In autonomous mode, each cluster takes its Applications from git, and the central UI shows them and can start syncs (autonomous mode).
Upsides
- One UI shows every cluster, while nothing connects into the clusters and the central cluster holds no credentials for them.
- In autonomous mode, the central cluster is neither a single point of failure nor of compromise: each cluster keeps deploying from git while it's down.
- Argo CD in each cluster corrects drift with
selfHeal: true, as in a controller per cluster.
Downsides
- At v0.10.0, its documentation lists high availability for the principal and advanced RBAC for multi-tenancy as still under development (features).
- In managed mode, a compromised central cluster can affect every cluster, and it's a single point of failure when the clusters use its Redis or repository server.
- In autonomous mode, every cluster runs all of Argo CD except its API server and Dex, and the central UI can't change an Application.
- It's one more component to run and upgrade next to Argo CD itself.
Choosing
Start with a service account token in CI, and change when one of these signals keeps showing up. The Simple blueprint deploys this way, and the GitOps blueprint uses a GitOps controller per cluster.
Network
When the cluster's API can't accept connections from CI, because its allow list can't name the runners or the cluster sits behind NAT, let the cluster open the connection: use a runner, an agent or a GitOps controller in the cluster.
Credentials
When no cluster credential may live in CI, use a runner or agent in the cluster, or a GitOps controller, so that CI holds at most a token that can push to the configuration repository. When rotating a token per cluster is the problem, a cloud token through OpenID Connect stores nothing, but it reaches every cluster its role covers.
Branches
When only protected branches may deploy, use a protected variable or a protected runner, which GitLab offers on every tier. The GitLab agent only fits on GitLab.com when production is a protected environment, on Premium. With GitOps, branch protection on the configuration repository decides.
Drift
When the cluster has to match git after someone changes it by hand, use a GitOps controller with drift correction turned on. With a push, a change made by hand stays until the next deploy sets that field again.
Many clusters
When one team deploys to many clusters and wants one view of them, a central GitOps controller gives one UI, but one cluster then holds a credential for all of them. Argo CD's agent mode gives one UI without those credentials, within the limits listed under Central UI with agents.