Cloud

Monitoring & Observability

Knowing that something is wrong before your customers tell you

How this is usually engaged

Commonly a three to five week engagement to instrument, define alerts and hand over on-call practice, either standalone or alongside a hosting arrangement. Alerting response commitments, where we are the recipient, are agreed per engagement and written into the support agreement.

Practice
Cloud & Infrastructure
Sectors
4 served
Process
5 stages
Standard
Built and tested to WCAG 2.2 AA

Summary

Metrics, logs, traces and error tracking assembled into a small number of alerts that reach a person who can act, and dashboards that answer the questions asked during an incident.

01What this solves

The problem this addresses

There are two failure modes here and most organisations have both. The first is silence: the system goes down and the first report arrives from a customer, hours later. The second is noise: hundreds of alerts a day, a channel nobody reads, and a genuine outage sitting unnoticed among the routine ones. Alert fatigue is not a discipline problem. It is a design problem, and it is what causes the alert that mattered to be scrolled past.

02Capabilities

What is included

01

Service level indicators and alerting on symptoms

Alerts on what users experience — error rate, latency, failed transactions, jobs not completing — rather than on the resource metrics that produce most false alarms. A page at four in the morning should mean a person is affected.

02

Structured logging and correlation

Consistent structured logs with a request identifier carried across services, so an incident investigation is a query rather than a hunt through several unrelated log formats.

03

Error tracking and release correlation

Exceptions grouped, deduplicated and tied to the release that introduced them, with source context. Knowing that a spike began at the deployment forty minutes ago is usually most of the diagnosis.

04

Dashboards built for incidents

A small number of dashboards that answer the questions actually asked at two in the morning — what changed, what is failing, is it getting worse — rather than a wall of charts assembled because the data was available.

05

Alert routing and on-call design

Routing rules, escalation, quiet hours and a rota, so an alert reaches a named person with the access and authority to act. We also help write the runbook entry each alert links to, because an alert without a next step just transfers anxiety.

06

Retention and cost control

Log volume and retention set deliberately per stream, with sampling on high-volume traces. Observability bills quietly exceed hosting bills more often than people expect, and it is nearly always retention nobody chose.

03Use cases

Where this work usually starts

01

Finding out from customers

Outages are currently reported by the people affected. The first goal is not a dashboard; it is one reliable alert that fires before the first complaint.

E-commerce

02

Cutting an alert channel nobody reads

Hundreds of notifications a day and a team that has stopped looking. We delete the alerts with no action attached and rebuild the small set that means something.

Logistics & Transport

03

Diagnosing intermittent slowness

Something is slow some of the time and nobody can reproduce it. Tracing and correlated logs turn an argument between teams into a measurement.

Finance

04Approach

How we approach it

We start from the things your users care about — can they log in, can they complete an order, are jobs finishing — and instrument those as service level indicators rather than instrumenting everything and hoping. Logs are structured and correlated with a request identifier so a single user's journey can be followed across services, and traces are added where the latency question is genuinely hard to answer otherwise. Then we do the part that gets skipped: for every alert, we define who it wakes, what they are expected to do, and what makes it safe to delete. An alert with no action attached is deleted, not tuned.

05Process

How the work runs

The order matters more than the ceremony. Stages overlap in practice, but none of them is skipped.

  1. 01

    Define what matters

    The user journeys whose failure constitutes an outage, agreed with the people who answer to customers rather than assumed by engineers.

  2. 02

    Instrument

    Metrics, structured logs and error tracking added to the application and the platform, with redaction applied at the point of emission.

  3. 03

    Design the alerts

    A deliberately small set of alerts on symptoms, each with a threshold argued from real data, an owner and a runbook entry.

  4. 04

    Route to a human

    Escalation, rota and quiet hours configured and then tested with a deliberate alert outside working hours, because untested routing usually reaches nobody.

  5. 05

    Review and prune

    After a few weeks of real traffic we tune thresholds against what actually fired and remove the alerts that proved to be noise. Observability is maintained, not installed.

06Deliverables

What you receive

Everything below is handed over as part of the engagement. If you take the work elsewhere afterwards, the next team has what it needs.

  • Instrumented application and infrastructure metrics
  • Alert definitions with owners and runbook linksEvery alert states who receives it and what they should do.
  • Incident dashboards
  • Log retention, redaction and access policy
  • On-call rota and escalation configuration
  • Incident review template and a walkthrough with your team
07Technologies

What we use on this work

These entries are drawn from our managed technology directory, which records why each one is in our stack and what we reach for it for. Nothing outside this list is claimed for this service.

Languages & Runtimes

  • TypeScript

Data & Storage

  • Elasticsearch

Cloud Platforms

  • Amazon Web Services

DevOps & Operations

  • Docker
  • Kubernetes
  • Grafana
  • Sentry
The full technology directory

Highlighted entries are used routinely on delivery work

08Security

How this is kept secure

Logs are treated as data that will inevitably contain something sensitive. We redact credentials, tokens and personal data at the point of emission rather than trusting a downstream filter, and retention is set against your obligations rather than defaulting to whatever the tool offers. Access to logs and traces is role-controlled, because full log access is close to full data access. Where telemetry leaves your environment, we say which vendor receives what.

09Questions

Asked before we start

Which monitoring tool should we use?

It matters less than the discipline around it, and the cheapest good answer is often the one already included with your cloud provider plus an error tracker. We commonly use Grafana for metrics and dashboards and Sentry for application errors, both of which have workable hosted and self-hosted options. We would rather spend the budget on defining fewer, better alerts than on a platform whose value depends on someone curating it.

How many alerts should we expect?

Few enough that every one is read. For a typical single application, a working set is often under a dozen alerts that can wake someone, covering availability, error rate, latency, failed background jobs and the handful of dependencies whose failure stops the service. Everything else belongs on a dashboard or in a weekly review, not in a notification.

Can you be our on-call team?

Only within limits worth stating plainly. We can hold first-line response for infrastructure and platform failures under a support agreement, with hours and response commitments agreed with you in writing. We cannot make judgement calls that require knowledge of your business — whether to halt a promotion, contact a regulator or accept degraded service during a busy period — and those escalations need a named person on your side who can decide.

Will this slow our application down?

Metrics and structured logging have a negligible effect at the volumes most applications run at. Distributed tracing is heavier, which is why we sample it rather than capture every request, and we measure the overhead in staging before enabling it in production. If instrumentation ever costs more than the insight it provides, it should be removed, and we will tell you when we think that line has been crossed.

START HERE

Talk to us about Monitoring & Observability

The project brief takes about five minutes and gives us enough to have a useful first conversation about Monitoring & Observability rather than a generic one. If you would rather just ask a question, the short form is there for that.

We read every enquiry ourselves. If we are not the right fit, we will say so and tell you what to look for instead.