impact metrics

I stopped caring how many operations we served and started caring which ones mattered when they broke.

For a long time I tracked total traffic, average response times, aggregate error rates. Those numbers hid the story from me. The question I actually needed was which endpoints hurt a user most when they failed.

Someone is setting up the tool for the first time. If the onboarding call fails, they leave. That one failure cost us more than a thousand slow searches ever did, and it had nothing to do with volume.

Aggregate metrics lied to me

Three operations in one service:

• A: 10,000 requests, 10 failures. 0.1%
• B: 100 requests, 10 failures. 10%
• C: 100 requests, 50 failures. 50%

Combined, that reads as 0.7%. The dashboard was green while C was on fire.

So I stopped asking what the error rate was and started asking which operations fail, and what a user cannot do while they are failing. Account creation might run a hundred times a day and take the whole product down with it.

Rare and manual operations usually carry the highest impact when they break.

Weighting by impact

Not all requests are equal. A payment failure is not a search timeout. A profile update is not a recommendation.

I had to know the product well enough to sort them:

• critical: login, payments, data saves
• high frequency: search, feeds, navigation
• user blocking: account creation, password resets
• revenue: checkout, subscriptions, upgrades

Then I weight the endpoint by that instead of by traffic. That was the annoying part and I think it is also the point. Defining the metric made me learn what the product actually did.

So now I measure what users care about, and fix that first.