Short write-up · Predictive modeling

The best-performing feature was 90% data leakage.

Per-channel purchase-propensity models produced dozens of deploy-ready personalization signals. The project's most valuable output was the one signal that didn't ship.

When2026
RoleSole analyst
MethodPer-channel L1 logistic regression · out-of-fold validation
DestinationOn-site personalization platform

The question

Which on-site behaviors actually predict a purchase in each channel, and which are strong enough to drive personalization rules, instead of the gut-feel targeting the platform was running on?

What I built

One model per purchase channel (cart, phone, chat, store, and two more), trained on roughly a hundred thousand converters with a 30-day pre-purchase behavior window, L1-regularized so the model itself prunes weak signals, validated out-of-fold so every reported number is on unseen data. The output wasn't a black-box score: it was a ranked, vetted list of individual behaviors, each one deployable as a plain targeting rule.

The catch that mattered

The strongest feature in early runs was a delivery-status page view: spectacular predictive power, obvious retrospective logic ("people checking delivery are engaged buyers"). It was also ~90% leakage: most of those visits happened after the order, reached from shipping-confirmation emails. The model was predicting purchases that had already occurred. Strict timestamp windowing, counting only behavior that happened before the order, collapsed the feature to noise, and it was cut before anything reached production.

If that signal had shipped, the personalization program's first result would have been "targeting works brilliantly on people who already bought."
Top signals by out-of-fold predictive strength (one channel shown)
Illustrative ranking · the struck bar is the leakage feature, shown at its pre-windowing strength
The honest list is less dramatic. What survived validation are ordinary, actionable behaviors: repeat product views, saved-item returns, financing-page visits. Each is strong enough to act on, and none is too good to be true.

Where it landed

The vetted per-channel signal lists went to the personalization team as targeting rules, replacing intuition-based segments; the leakage-prone feature was documented as a warning for future models; and re-scoring was set up on a schedule so the rankings stay honest as behavior shifts.

Back to all work · featured case studies: the organic decline, the mart, the dashboard suite.

Signal names, channels, and figures are generalized or perturbed; the ranking shown is illustrative. The method, the leakage catch, and the deployment path are real.