How do you measure whether recommendations are working?

Use clicks and conversions as early signals, confirm changes with controlled live experiments, and track longer-term outcomes such as repeat purchase, returns and retention.

Clicks and add-to-cart rates are fast and useful, but a click doesn’t prove a recommendation was good. People click on whatever sits at the top of a list, and systems trained only on clicks learn to serve whatever is easiest to engage with.

A stronger approach combines three things: offline tests to screen ideas, controlled live experiments to confirm them, and longer-term measures such as repeat purchase, returns, retention and customer satisfaction. Several RecSys 2026 speakers, including Thorsten Joachims of Cornell and the team at Lyft, focused on optimizing for those longer-term outcomes.