What metrics are used to evaluate recommendation systems?

Common offline metrics are precision@k, recall@k, hit rate@k and NDCG@k. Businesses also track click-through, conversion, basket size, repeat purchase, returns and retention.

  • Precision@k: of the top k items shown, the share the customer actually wanted.
  • Recall@k: of all the items the customer would have wanted, the share that made it into the top k.
  • Hit rate@k: the share of customers for whom at least one of the top k items was a hit.
  • NDCG@k: similar to precision, but gives more credit when the hits appear near the top of the list.
  • Coverage and diversity: how much of the catalogue gets recommended at all, and how varied each list is.

These are useful for comparing models offline. Business outcomes such as conversion, basket size, repeat purchase, returns and retention, confirmed with live experiments, tell you whether recommendations are working for customers.