You build it, you operate it: Actually implementing something like that

The challenge: working monitoring, absent ownership

The client's production operations presented a diagnostic puzzle. The monitoring worked. Alerts were configured correctly and fired reliably. Yet when a service went down at midnight, it stayed down. The tech lead would intervene the following day, and recovery began only after a meeting had established whose responsibility the failure was. Customers experienced downtime or degraded service for hours at a stretch, and on occasion for days — the recorded maximum time-to-response was three days.

An engineering diagnosis would have found nothing to fix, since the tooling was sound. The failure was organisational: when an alert belongs to everyone, it belongs to no one. With ten engineers all nominally responsible for production, each could reasonably assume someone else was looking — the bystander effect, operating on an alert channel. The subsequent responsibility meetings compounded the cost: before any recovery work began, the organisation first ran a process to determine who should feel accountable, with the service still down.

The instinctive remedy in this situation is additional tooling — louder alerts, more dashboards, an escalation bot. We declined that path on the diagnosis itself: diffuse accountability is unaffected by alert volume. The intervention had to be policy and agreement first, with tooling in a supporting role. This made the engagement as much an exercise in organisational development as in operations: ownership would have to be negotiated with teams, several of which had reasons to want no part of it.

The approach: negotiate the ownership, then encode it

1. Establish one rule with no exceptions: one service, one owner

The policy core was a single constraint — a service can never be owned by two teams. Shared ownership recreates the bystander problem one level up, with teams in place of individuals. Every other element of the design was negotiable; this one was held fixed throughout.

2. Assign ownership by consensus, accepting that this means negotiation

We ran a manual service discovery across the estate and mapped every service to a candidate owner — with the teams participating in the assignment, so that ownership was reached by consensus rather than appointment. Ownership that is declared from above is complied with; ownership that a team has agreed to is acted on at midnight, which was the behaviour the engagement needed to produce.

Consensus was the difficult part, and the reason a facilitated process was required at all. Accepting ownership of a service means accepting its pages, its failure modes, and its history — and several services had unattractive histories. The discovery surfaced systems that no present engineer fully understood: built on niche frameworks by engineers who had since departed, they were precisely the services most likely to fail and least wanted by any team. Assignment ran through a few facilitated cross-team sessions, in which the candidate map was walked service by service and ownership agreed in the room while the trade-offs were visible to everyone. The resolutions took the general form of pairing ownership with a commitment: teams accepted these services alongside an agreed plan to make them ownable — the orphaned systems were documented for what they were and marked for rewrite, so that saying yes to ownership did not mean saying yes to indefinite liability for another person's decisions.

3. Set the policy bar; let each team choose its mechanics

The duty policy defined what had to be true: every service has a named owning team, and someone on that team is on duty for it at any given time. How each team met that bar was left to the team. Some ran daily rotations; others ran six-hourly shifts through the day; each shaped its schedule around its members' circumstances. The constraints were real: small teams and members spread across timezones made genuine round-the-clock cover impractical, so teams chose day-shift rotations with a defined escalation path for overnight rather than pretend to staff a 24/7 roster they did not have the people for.

The delegation was deliberate rather than permissive. Duty schedules that teams design for themselves stand a far better chance of being honoured than schedules imposed from above, which get gamed. The policy owned the outcome, and the teams owned the method — though, as below, it was the ownership itself, not the perfection of any rota, that ultimately carried the result.

4. Route alerts to the people who agreed to act on them

Alert channels were restructured at the team level. Kubernetes workloads were tagged by owning team, and a firing alert landed in the owning team's channel, where the duty rotation guaranteed a named person responsible for picking it up. The general alert channel — the place where accountability had diffused — ceased to be the destination. A phased triage process was established on top: the owning team responds first and escalates only when needed, replacing the previous pattern of escalating everything to the tech lead by default.

5. Make the ownership map a lookup, so it stays true

Ownership must be visible to be trusted, and it must be maintained to stay visible. The map was recorded in two forms: a central, company-wide document that anyone could consult, and — more durably — ownership tags encoded into the infrastructure itself through infrastructure-as-code, extending coverage to services running directly on EC2 instances outside the orchestrator. "Who owns this?" became a lookup where it had previously been a meeting.

The execution: sequence of the work

Phase one — diagnosis and discovery (the first days). Incident history reviewed and the baseline established; manual service discovery across the estate; candidate ownership map drafted.

Phase two — negotiation (the bulk of the engagement). Facilitated sessions with the teams to reach consensus ownership, including the resolution of the orphaned services; duty policy agreed.

Phase three — implementation (once ownership was agreed). Duty rotations set up in PagerDuty on each team's chosen cadence; alert channels restructured; workloads tagged by team; infrastructure-as-code updated to carry ownership tags across Kubernetes and EC2.

Phase four — operation (ongoing thereafter). Phased triage in effect; the ownership document and tags maintained as the standing record. New services could not ship without an owning-team tag in their infrastructure-as-code — enforced by a check in the pipeline — and the central ownership document was derived from those tags, so the map could not silently fall out of date.

What did not go smoothly

Not all of it went smoothly: the ownership negotiations for the orphaned services were the slowest part of the engagement.

A team initially declined an orphaned service. One team refused outright to take on a poorly-understood service whose author had departed. It was resolved by pairing acceptance with a bounded rewrite commitment and interim documentation, so that saying yes was fair rather than open-ended liability.

Historically shared services were the hardest to assign. The one-service-one-owner rule collided directly with services two teams had always shared — one consuming, another maintaining. Landing a single owner for these took the most negotiation, and was settled by naming the maintaining team as owner while formalising what the consuming team could expect from the interface.

A rotation needed revision after real incidents. One team's first schedule proved to under-cover a window once real incidents arrived, and was revised after contact with reality rather than on paper.

Tagging non-compute resources was tedious. Ownership tags were straightforward for Kubernetes workloads but laborious for everything else — EC2 instances and, more tediously, resources such as S3 buckets — which had to be attributed through infrastructure-as-code with far more manual effort.

The results

Technical outcomes

• Time-to-response fell from a recorded maximum of 3 days to an average of 2 hours, with customer-facing downtime reduced accordingly
• Triage begins the moment an alert fires, following a phased process — owning team first, escalation only when needed
• Ownership tagging produced per-team cost visibility as a side effect, at both the cloud and application level, enabling FinOps practices the organisation previously had no path to
• Technology sprawl was arrested as it emerged: the discovery surfaced services no one understood, which were documented and marked for rewrite

Human outcomes

• The responsibility meeting was eliminated. Failure ownership became a documented process consulted in seconds, in place of a "who last touched it?" discussion held while the service was down
• Teams became intentional about creating services. Because every new service arrives with an owner and a pager, spinning one up became a considered decision — the accountability structure changed building behaviour, without any rule about building
• Clear ownership carried the result, even when rotations slipped. The duty rotations did not always hold — capacity was tight, and team leads frequently had to step in — but the outcome never depended on flawless scheduling. Because ownership was recorded in a document searchable by the whole organisation, the bystander effect was gone: a named lead knew a failing service was theirs to escalate and surface, so a response began regardless of whose turn it nominally was. That, not a perfect rota, is what turned three-day silences into two-hour responses
• Incidents became occasions for learning. With ownership no longer in question, a surfaced incident led to understanding the service's behaviour and improving it, rather than to a discussion of who was accountable while it stayed down

What this means if you're running a startup or SME

When alerts go unanswered, the missing component is usually an agreement, and tooling cannot substitute for it.

The technical implementation here — rotations, routing, tags — was straightforward and took a fraction of the engagement. The work that determined the outcome was human: a diagnosis that resisted the instinct to buy more tooling, and a negotiation in which teams agreed to own things, including things nobody wanted, on terms that made the ownership fair. This is a recurring shape in SME engineering problems: the system's behaviour is downstream of an organisational arrangement, and it is the arrangement that has to be redesigned.

This is the work we do at Aijutsu under our organisational development and coaching practice, alongside platform engineering: diagnosing where operational problems are organisational, facilitating the agreements that resolve them, and encoding those agreements into the infrastructure so they outlast the engagement.

If services go down at night and stay down until a morning meeting decides whose problem they are, the fix is available — and it is unlikely to be another dashboard. Book a diagnostic conversation: [email protected]

Frequently Asked Questions

Why do alerts go unanswered even when monitoring works?

Usually because accountability is diffuse: when every engineer is nominally responsible for production, each can reasonably assume someone else is responding — the bystander effect applied to an alert channel. In this case, alerts were firing correctly and going unanswered for up to three days; the remedy was an ownership structure, with the monitoring left largely unchanged.

How do you reduce incident response time without new tooling?

Assign every service to exactly one owning team, put a duty rotation behind each team, and route each alert to the channel of the team that owns the failing service. In this case, those three changes reduced time-to-response from a recorded maximum of 3 days to an average of 2 hours using the existing monitoring stack plus PagerDuty.

How should engineering teams assign service ownership?

One service, one owning team, with no service owned by two teams — shared ownership recreates the accountability gap at the team level. Assignment works best by consensus: ownership a team has agreed to produces action at midnight, while ownership imposed from above produces compliance during business hours.

How do you design on-call rotations that teams actually follow?

Set the policy bar centrally and let each team design its own mechanics. In this case the policy required only that every service have a named owner with someone on duty; teams then chose their own cadence — daily rotations for some, six-hourly shifts for others — shaped around their members' circumstances. Even so, the rotations were not self-sustaining: capacity pressures meant leads sometimes had to step in. What reliably held was the ownership itself — with every service assigned to a named team in a searchable record, someone always knew an alert was theirs to answer, which is the property that actually reduces response time.

How do you handle services that no team wants to own?

Pair the ownership with a commitment that makes it fair. The services no team wanted in this case were systems built on niche frameworks by departed engineers; teams accepted them alongside an agreed plan — documentation of the current state and a marked path to rewrite — so accepting ownership did not mean accepting indefinite liability for decisions the team never made.

What are the side benefits of service ownership tagging?

Cost visibility is the principal one. Tagging every workload by owning team — in this case across Kubernetes and EC2 via infrastructure-as-code — attributes cloud spend to teams as a by-product, which is the prerequisite for FinOps practices such as per-team budgets and cost accountability.