The leaderboard was ranking adoption, not closing skill.

The situation

On a mattress sales floor the selling is personal. A sleep expert runs a tablet-based recommendation quiz with the customer, carts the suggestions, and, when the customer walks out to think it over, uses a follow-up workflow to bring them back. Every step leaves a trail: quiz sessions, carted recommendations, CRM opportunities, and eventually orders in the ERP. Almost nobody was reading that trail.

Then a retail leader asked a pointed question: are sleep experts using the follow-up workflow to close customers buying under $1,500, or only on premium tickets? Behind it sat real decisions: where to coach, whom to coach on what, and whether to push the tools harder on low-ticket shoppers. Complicating things, a vendor report claimed a follow-up close rate near 50% while the warehouse said roughly half that. Before anyone could decide anything, the number had to be made trustworthy.

What I did

First the plumbing. The official tracking spreadsheet turned out to hold about a third of the real lead volume, so I rebuilt the dataset from the warehouse, stitching quiz sessions to CRM opportunities to ERP orders for each salesperson, with every join rate measured and published rather than assumed. Then the definition: a close became a follow-up order written 1 to 30 days after the lead, with same-day sales stripped from both sides of the ratio. That one rule also settled the vendor dispute: nine-tenths of the vendor's numerator was same-day sales that never needed a follow-up.

On that foundation went a leaderboard that drills from division to region to store to the individual sleep expert, with the under and over $1,500 split as first-class columns and a fresh build every morning. The leader's question got its own permanently maintained tab.

The turn

The trap was the denominator. There are three defensible ways to compute a close rate, and they rank regions differently. The headline rate (closes over all eligible leads) correlates strongly with how often a region's salespeople engage the tools at all (r ≈ 0.7), so it punishes under-tooled regions. The engaged-only rate rewards them, because a region that tools up only its hottest prospects looks elite. Tool engagement ranged from under 30% of leads in some regions to nearly 50% in others. That is enough to scramble any single ranking.

So the dashboard shipped with a reading rule instead of a single ranking, and every callout behind it was z-tested with a raised bar so that only gaps robust to dozens of simultaneous comparisons drove a coaching conversation.

And the original question got its answer: salespeople were about half as likely to use the follow-up workflow on a sub-$1,500 mattress. But conditioned on engagement, close rates on the two sides of the price line matched to within a rounding error, while leads with no tool engagement closed at roughly a third of the rate. The skill was fine. The habit was the gap.

What happened after

Coaching changed shape. Regions with strong closing but thin engagement got process-adoption coaching, a region with the opposite pattern got closing coaching, and the region strong under every method became the benchmark. The dashboard runs on the internal hub with a daily morning refresh, drills to the individual salesperson, and kept growing on stakeholder asks: district-level cuts, price-line columns on the leaderboard, exports for the field.

The honest limit is stated on the page itself: salespeople choose whether to use the tools, so observational lift cannot be read as causal. The next step was designed accordingly: a randomized store-level assignment, roughly 1,800 stores stratified into matched halves, sits ready for the causal read on the new recommendation experience before a full rollout.