GitOps blueprint
The GitOps blueprint moves the deployment configuration of every app to one configuration repository, which Flux or Argo CD watches and applies to the cluster. The project's pipeline builds the image and commits its tag to the configuration repository. Promotion and rollback are commits in that repository. The controller is part of the cloud platform, which the Cloud Handbook covers. The choices that differ from the Simple blueprint are the repositories, access, secrets, the deploy channel and delivery. Move to it from Simple when one of the needs under Choosing comes up.
Development
| Topic | Choice |
|---|---|
| Repositories | Projects and configuration: a repository per project for the app's code, one configuration repository for the deployment configuration of every app, and the cloud on its own, so configuration changes deploy without a build |
| Branches | A short-lived branch per change off the default branch, so each change stays small enough to review |
| Merging | Rebase onto the default branch before merging, so the pipeline tests each change on top of what it merges into |
| Merge requests | One approval and a passing pipeline before merging, so every commit on the default branch is reviewed and tested |
| Tickets | A ticket per piece of work in the project's issue tracker, closed by the merge request that finishes it |
| Changelog | A CHANGELOG.md in the Keep a Changelog format, with each entry added in the merge request that makes the change, so the person who knows the change writes it |
| Image | One image per app, built once per commit with no environment configuration in it, so staging and production run the same image |
Security (SecOps)
| Topic | Choice |
|---|---|
| Secrets | Kept in the cloud platform's secret store and synced into Kubernetes Secrets by External Secrets Operator, so the configuration repository holds only references to them |
| Access for people | Developers merge in the project repositories, and a smaller group merges in the configuration repository, so deploy access is split from code access |
| Access for CI | A token that can push to the configuration repository and no cluster credentials, since the controller deploys from inside the cluster |
| Code scanning | Static analysis and secret detection in every merge request pipeline, so findings show up before the merge |
| Dependencies | Renovate for updates, and dependency and image scanning in every pipeline, so a vulnerable package shows up before it ships |
Chat (ChatOps)
| Topic | Choice |
|---|---|
| Alert channels | One production alert channel for alerts around the clock, and new errors in each team's own channel, so the on-call engineer gets only the alerts that need a response |
| Deploy channels | One deploy channel where the controller's notifications post every production sync with its app and version, so an incident lead finds recent deploys in one place |
| Incident channels | A public channel per incident named incident-<date>-<summary>, or a private one for a security incident, so the channel holds the timeline for the report |
Finance (FinOps)
| Topic | Choice |
|---|---|
| Cost labels | Labels for app, team and environment on every resource, so cost splits along all three |
| Cost ownership | The team that owns an app owns its cost, with a budget per team, so someone notices when the cost grows |
| Environment costs | Staging at one replica per app, since staging serves no customer traffic |
Operations
Delivery
| Topic | Choice |
|---|---|
| Environments | Staging and production, so every change runs in staging before it reaches production |
| Configuration | A directory per app in the configuration repository, with base values and an override per environment on the default branch, since branches per environment drift apart |
| Controller | Flux or Argo CD, whichever the cloud platform runs, watching the configuration repository's default branch, so a merge there is a deploy |
| Pipeline | Test and build stages, then a job that commits the new image tag to the staging override, so staging runs every commit on the default branch |
| Promotion | A merge request in the configuration repository that sets production's image tag to the one staging runs, so every production change is reviewed |
| Rollback | A revert of the promotion commit in the configuration repository, so the controller deploys the previous tag and configuration |
| Drift | The controller reverts changes made by hand in the cluster, so the configuration repository is what runs |
| Production readiness | A review before an app's first production deploy, so its probes, alerts and dashboards exist before customers rely on it |
Kubernetes
| Topic | Choice |
|---|---|
| Probes | A readiness probe on every container that serves traffic, and a liveness probe that checks only the process, so a database outage doesn't restart every pod |
| Shutdown | A 5-second preStop sleep, so the app keeps serving until load balancers stop sending it requests |
| Autoscaling | A Horizontal Pod Autoscaler on CPU with at least two replicas in production, so a failing node leaves a pod serving |
Observability
| Topic | Choice |
|---|---|
| Versions | The version baked into the image, and shown as the error tracker's release, the app.kubernetes.io/version label and a build_info metric, so anyone with access sees which build runs |
| Logging | One JSON object per line on stdout, at info level and up in production, with the request's trace ID on each line, so the log store can filter by field and by request |
| Metrics | A Prometheus /metrics endpoint with a request counter by status and a latency histogram, since the error rate and latency alerts need both |
| Monitoring | Uptime checks on every domain, alerts on each app's error rate and latency, and a dashboard per app, so the team hears about a problem before customers report it |
| Error tracking | The Sentry SDK in every app, so each error reaches the error tracker with its stack trace and release |
Incidents
| Topic | Choice |
|---|---|
| Incident response | Three severity levels, and an incident lead who hands out the communications lead and scribe roles, so the people who fix the problem don't also coordinate |
| Incident reports | A blameless report from the template within 24 hours of every Critical and Major incident, so the team learns from it while the details are fresh |