Vanity metrics: which numbers are worth measuring

A vanity metric is a number that looks attractive in a report but is weakly connected to a decision or business result. The test is simple: name who will do what after a specific threshold is crossed. If there is no answer, the metric may be diagnostic context, but it should not dominate the dashboard.

This article is about filtering. We write about which indicators are worth having in key performance indicators.

A three-question test for every metric

Before an indicator reaches a dashboard, it should pass:

1. Does any decision depend on it? Not “is it interesting?”. Is there a decision you would make differently depending on its value?
2. Can you influence it? A metric over which you have no lever is weather information.
3. Is it clear what a change means? If an increase can mean three different things, you cannot act on it without further investigation.

An indicator that does not pass all three can exist in a report — but it should not be in view or the subject of a weekly conversation.

Classic vanity metrics and what to use instead

Vanity metric	Why it misleads	What to use instead
Page-view count	can grow through traffic with no buying intent	entries into important journeys and their conversion
Follower count	does not show a channel’s value by itself	audience actions of quality and a result attributed to the channel
Number of emails sent	measures your activity, not the effect	number of replies and meetings
Total registered accounts	does not say whether anyone uses the product	accounts that performed the primary action
Average time on page	also rises when people cannot find something	completion of the intended action
Number of closed tickets	rises when there are more tickets	time to resolution and share of returning cases
Number of released features	measures work, not value	use of each feature after a month

Note the pattern: vanity metrics usually measure input or reach, while useful metrics measure an outcome or friction.

Three pairs worth keeping together

One number can hide a side cost. Pairs of indicators help balance the result:

• Volume and quality. Number of sales enquiries plus the share that matches the target profile. Volume alone is easy to increase by adding traffic that will buy nothing.
• Speed and effectiveness. Ticket-handling time plus the share of returning cases. Reducing time by sending any response becomes visible only in the second number.
• Growth and retention. New users plus those who returned after a month. Growth alone can mask a product that retains no one.

How many metrics to have

There is no universal right number. Leave as many metrics in the main view as the recipient needs to make a decision — each should have an owner, definition, and expected response.

An example hierarchy:

• a primary outcome or a small set of outcomes — these tell you whether an area is meeting its goal;
• operational and guardrail metrics — these show the levers and side costs;
• the rest in an archive — available when you need to investigate a cause, but not in view.

A large number of indicators on one screen increases interpretation cost. Details can remain in a diagnostic view; the main view should match the rhythm and decision scope of its particular audience.

Four common traps

Measuring what is easy to measure. A tool provides page views, so we look at page views. Something being available does not make it important.

An average without a distribution. An average handling time of four hours can mean everything moves similarly, or that some cases take minutes and some days. Show a median, selected percentiles, and sample size; the choice depends on the decision and the cost of extreme cases.

A metric without a benchmark. “We have 340 enquiries a month” says nothing without knowing how many there were before and how many there should be. That is why we measure the state before every automation change — see process automation.

A goal set on the metric rather than the outcome. A measure that becomes a target stops being a measure. A team assessed on the number of closed tickets will start closing tickets, not solving the matter.

How to choose metrics from scratch

The sequence we use in analytics projects:

1. List the decisions you make repeatedly. Every week, month, or quarter.
2. For each one, decide which number is missing to make it faster or more certain.
3. Check whether that number can be calculated today from the data you have. If not, describe the missing source, frequency, quality, and acquisition cost; only then design a data layer.
4. Discard everything not assigned to a decision.

The fourth point is hardest because it requires removing from the report things someone once requested. It is still worth doing: every number in view consumes attention, and attention is finite.

When a vanity metric is useful after all

To be fair, because this is not a black-and-white division:

• As an early indicator. Search impressions do not pay bills, but they can appear before clicks and business results. In a new topic, they are one of the first visibility signals, to be assessed together with queries, position, and later user behaviour.
• As an alarm signal. A sudden halving of page views does not tell you what to do, but it does tell you to look for the cause.
• In external communication. Reach can be an argument in a conversation with a partner. Just do not make a product decision from it.

Frequently asked questions

How often should we look at metrics?

Match the rhythm to decision frequency, data delay, and volatility. An operational indicator may need a near-real-time alert, while a strategic result may need monthly or quarterly review. Also agree on the minimum change to which the team should react.

Is a dashboard worth having if we have little data?

Yes, if the panel replaces a manual compilation or supports a recurring decision. With a small sample, show the sample size and avoid conclusions from random fluctuations. Sometimes a simple periodic report is better than a dashboard maintained continuously.

Who should choose metrics?

The decision owner together with a person who understands the data and process. The former specifies the action; the latter protects definitions, quality, latency, and the possibility of manipulating the indicator.

---

We build panels for decisions, not presentations — and measure our own analytics platform by the same set of principles. See data and analytics, or tell us which decisions you currently make by instinct.