Simple blueprint
The Simple blueprint is a starting setup for a team's apps. Each project has one repository that holds the app's code and its deployment configuration. The project's pipeline builds the image and deploys it with Helm, so no CD software runs in the cluster. The cloud platform has a monorepo of its own, as the Cloud Handbook describes. Move to the GitOps blueprint when one of the needs under Choosing comes up.
Development
| Topic | Choice |
|---|---|
| Repositories | One repository per project, with the app's code in src/ and its deployment configuration in deploy/, so a feature and the configuration it needs land in one merge request |
| Branches | A short-lived branch per change off the default branch, so each change stays small enough to review |
| Merging | Rebase onto the default branch before merging, so the pipeline tests each change on top of what it merges into |
| Merge requests | One approval and a passing pipeline before merging, so every commit on the default branch is reviewed and tested |
| Tickets | A ticket per piece of work in the project's issue tracker, closed by the merge request that finishes it |
| Changelog | A CHANGELOG.md in the Keep a Changelog format, with each entry added in the merge request that makes the change, so the person who knows the change writes it |
| Image | One image per app, built once per commit with no environment configuration in it, so staging and production run the same image |
Security (SecOps)
| Topic | Choice |
|---|---|
| Secrets | Protected, masked CI variables per environment, which the deploy job writes to a Kubernetes Secret, so secrets stay out of git without a secret store to run |
| Access for people | The project's developers review and merge code and deployment configuration alike, since both are in one repository |
| Access for CI | A cluster credential per environment that can change only the app's namespace, in a protected variable, so only the default branch can deploy |
| Code scanning | Static analysis and secret detection in every merge request pipeline, so findings show up before the merge |
| Dependencies | Renovate for updates, and dependency and image scanning in every pipeline, so a vulnerable package shows up before it ships |
Chat (ChatOps)
| Topic | Choice |
|---|---|
| Alert channels | One production alert channel for alerts around the clock, and new errors in each team's own channel, so the on-call engineer gets only the alerts that need a response |
| Deploy channels | One deploy channel where the deploy job posts every production deploy with its app and version, so an incident lead finds recent deploys in one place |
| Incident channels | A public channel per incident named incident-<date>-<summary>, or a private one for a security incident, so the channel holds the timeline for the report |
Finance (FinOps)
| Topic | Choice |
|---|---|
| Cost labels | Labels for app, team and environment on every resource, so cost splits along all three |
| Cost ownership | The team that owns an app owns its cost, with a budget per team, so someone notices when the cost grows |
| Environment costs | Staging at one replica per app, since staging serves no customer traffic |
Operations
Delivery
| Topic | Choice |
|---|---|
| Environments | Staging and production, so every change runs in staging before it reaches production |
| Configuration | Base Helm values and one values file per environment in deploy/, so each environment's differences are in one small file |
| Pipeline | Test, build and deploy stages, with every commit on the default branch deployed to staging, so staging always runs the default branch |
| Deploys | helm upgrade --install --wait in the deploy job, so a rollout that doesn't finish fails the pipeline |
| Promotion | A manual production job in the pipeline that deployed to staging, so production gets the image and configuration that staging ran |
| Rollback | Run the production job of the last good pipeline again, so the image and its configuration roll back together |
| Production readiness | A review before an app's first production deploy, so its probes, alerts and dashboards exist before customers rely on it |
Kubernetes
| Topic | Choice |
|---|---|
| Probes | A readiness probe on every container that serves traffic, and a liveness probe that checks only the process, so a database outage doesn't restart every pod |
| Shutdown | A 5-second preStop sleep, so the app keeps serving until load balancers stop sending it requests |
| Autoscaling | A Horizontal Pod Autoscaler on CPU with at least two replicas in production, so a failing node leaves a pod serving |
Observability
| Topic | Choice |
|---|---|
| Versions | The version baked into the image, and shown as the error tracker's release, the app.kubernetes.io/version label and a build_info metric, so anyone with access sees which build runs |
| Logging | One JSON object per line on stdout, at info level and up in production, with the request's trace ID on each line, so the log store can filter by field and by request |
| Metrics | A Prometheus /metrics endpoint with a request counter by status and a latency histogram, since the error rate and latency alerts need both |
| Monitoring | Uptime checks on every domain, alerts on each app's error rate and latency, and a dashboard per app, so the team hears about a problem before customers report it |
| Error tracking | The Sentry SDK in every app, so each error reaches the error tracker with its stack trace and release |
Incidents
| Topic | Choice |
|---|---|
| Incident response | Three severity levels, and an incident lead who hands out the communications lead and scribe roles, so the people who fix the problem don't also coordinate |
| Incident reports | A blameless report from the template within 24 hours of every Critical and Major incident, so the team learns from it while the details are fresh |