Personalization of digital services remains the holy grail of marketing. In this paper, we study an important personalization problem – that of search rankings in the context of online search.
We present a three-pronged machine learning framework that improves the ranking of search results by incorporating a users’ personal search history. We apply our framework to data from the premier eastern European search engine, Yandex, and provide evidence of substantial improvements in search quality using personalization. We quantify the heterogeneity in returns to personalization as function of user history, query type (Do-Know-Go), and query’s past performance. We also show that
our framework can perform efficiently at scale, making it suitable for real-time deployment.
Our paper makes five key contributions to the marketing literature. First, it presents a general machine learning framework that marketers can use to rank recommendations using personalized data in many settings. Second, it presents empirical evidence in support of the returns to personalization in the online search context. Third, it provides managerial insights on the role of heterogeneity in user-history and query type on the returns to personalization. Fourth, it demonstrates how big data can be leveraged to improve marketing outcomes without compromising the speed or real-time performance of digital applications. Finally, it provides insights on how machine learning methods can be adapted to solve important marketing problems that have been technically unsolvable so far (using traditional econometric or analytical models).
Nevertheless, our paper overlooks a bunch of issues that serve as excellent avenues for future research. First, the use of individual-level user data (albeit anonymized) raises questions regarding privacy. It is not clear how users trade-off the loss in privacy with the gains from an improved search experience. Experimental or conjoint research that measures user preferences over these experiences can help managers decide the implications of deploying personalization algorithms. Second, our model’s predictions are predicated on the assumption that consumers’ click and search behaviors will continue to be the same even after we deploy personalized algorithms. However, this may not be the case. For example, Goldfarb and Tucker (2011) show that consumers alter their response rates when advertisements are perceived to intrude on privacy. Third, because we have not deployed the algorithm in the field, we cannot comment on long-term consumer satisfaction metrics and switching behavior. These limitations can be addressed by running a large scale field experiment that shows personalized results to a sample of users and compares their behavior with that of users who are shown non-personalized results. We believe that such a study would be of immense value to the field. More broadly, we hope our work will spur research on the application of machine learning methods to not only personalization-related issues, but also on a broad array of marketing issues.
References
D. W. Aha, D. Kibler, and M. K. Albert. Instance-based Learning Algorithms. Machine learning, 6(1):37–66, 1991.
E. Amaldi and V. Kann. On the Approximability of Minimizing Nonzero Variables or Unsatisfied Relations in Linear Systems. Theoretical Computer Science, 209(1):237–260, 1998.
A. Ansari and C. F. Mela. E-customization. Journal of Marketing Research, pages 131–145, 2003.
N. Arora and T. Henderson. Embedded Premium Promotion: Why it Works and How to Make it More Effective.
Marketing Science, 26(4):514–531, 2007.
R. Baeza-Yates, B. Ribeiro-Neto, et al. Modern Information Retrieval, volume 463. 1999.
S. Banerjee, S. Chakrabarti, and G. Ramakrishnan. Learning to Rank for Quantity Consensus Queries. In Proceedings of the 32nd international ACM SIGIR Conference on Research and Development in Information Retrieval, pages 243–250. ACM, 2009.
P. N. Bennett, R. W. White, W. Chu, S. T. Dumais, P. Bailey, F. Borisyuk, and X. Cui. Modeling the Impact of Short- and Long-term Behavior on Search Personalization. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 185–194, 2012.
L. Breiman, J. Friedman, C. Stone, and R. Olshen. Classification and Regression Trees. The Wadsworth and Brooks-Cole statistics-probability series. Taylor & Francis, 1984.
C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to Rank Using Gradient Descent. In Proceedings of the 22nd International Conference on Machine Learning, pages 89–96. ACM, 2005.
C. J. Burges, R. Ragno, and Q. V. Le. Learning to Rank with Nonsmooth Cost Functions. In Advances in Neural Information Processing Systems, pages 193–200, 2006.
Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li. Learning to Rank: From Pairwise Approach to Listwise Approach. In Proceedings of the 24th International Conference on Machine Learning, pages 129–136.
ACM, 2007.
R. Caruana and A. Niculescu-Mizil. An Empirical Comparison of Supervised Learning Algorithms. In Proceedings of the 23rd International Conference on Machine Learning, pages 161–168. ACM, 2006.
O. Chapelle and Y. Chang. Yahoo! Learning to Rank Challenge Overview. Journal of Machine Learning Research-Proceedings Track, 14:1–24, 2011.
M. Ciaramita, V. Murdock, and V. Plachouras. Online Learning from Click Data for Sponsored Search. In Proceedings of the 17th International Conference on World Wide Web, pages 227–236. ACM, 2008.
N. Clayton. Yandex Overtakes Bing as Worlds Fourth Search Engine,
2013. URL http://blogs.wsj.com/tech-europe/2013/02/11/
yandex-overtakes-bing-as-worlds-fourth-search-engine/.
N. Craswell. Mean Reciprocal Rank, pages 1703–1703. Springer US, 2009. ISBN 978-0-387-39940-9. URL http://dx.doi.org/10.1007/978-0-387-39940-9_488.
N. Craswell, O. Zoeter, M. Taylor, and B. Ramsey. An Experimental Comparison of Click Position-bias Models.
In Proceedings of the 2008 International Conference on Web Search and Data Mining, pages 87–94, 2008.
V. Dang. RankLib. Online, 2011. URL http://www.cs.umass.edu/˜{}vdang/ranklib.html.
B. De los Santos, A. Hortac¸su, and M. R. Wildenbeest. Testing Models of Consumer Search Using Data on Web Browsing and Purchasing Behavior. The American Economic Review, 102(6):2955–2980, 2012.
P. T. De Vrieze. Fundamentals of Adaptive Personalisation. Number 2006-2022. Paul de Vrieze, 2006.
P. Donmez, K. M. Svore, and C. J. Burges. On the Local Optimality of LambdaRank. In Proceedings of the 32nd international ACM SIGIR Conference on Research and Development in Information Retrieval, pages 460–467. ACM, 2009.
R. O. Duda and P. E. Hart. Pattern Recognition and Scene Analysis, 1973.
D. Dzyabura and J. R. Hauser. Active Machine Learning for Consideration Heuristics. Marketing Science, 30
(5):801–819, 2011.
C. Eickhoff, K. Collins-Thompson, P. N. Bennett, and S. Dumais. Personalizing Atypical Web Search Sessions.
In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, pages 285–294, 2013.
J. Elith, J. R. Leathwick, and T. Hastie. A Working guide to Boosted Regression Trees. Journal of Animal Ecology, 77(4):802–813, 2008.
P. Ferragina and A. Gulli. A Personalized Search Engine Based on Web-snippet Hierarchical Clustering.
Software: Practice and Experience, 38(2):189–225, 2008.
M. Feuz, M. Fuller, and F. Stalder. Personal Web Searching in the Age of Semantic Capitalism: Diagnosing the Mechanisms of Personalisation. First Monday, 16(2), 2011.
Y. Freund, R. Iyer, R. E. Schapire, and Y. Singer. An Efficient Boosting Algorithm for Combining Preferences.
Journal of Machine Learning Research, 4(Nov):933–969, 2003.
J. Friedman, T. Hastie, R. Tibshirani, et al. Additive Logistic Regression: A Statistical View of Boosting (with Discussion and a Rejoinder by the Authors). The Annals of Statistics, 28(2):337–407, 2000.
E. Gabbert. Keywords vs. Search Queries: What’s the Difference?, 2011. URL http://www.wordstream.
com/blog/ws/2011/05/25/keywords-vs-search-queries.
A. Ghose, P. G. Ipeirotis, and B. Li. Examining the Impact of Ranking on Consumer Behavior and Search Engine Revenue. Management Science, 2014.
M. Ginsberg. Essentials of Artificial Intelligence. Newnes, 1993.
A. G¨oker and D. He. Analysing Web Search Logs to Determine Session Boundaries for User-oriented Learning.
In Adaptive Hypermedia and Adaptive Web-Based Systems, pages 319–322, 2000.
A. Goldfarb and C. Tucker. Online display advertising: Targeting and obtrusiveness. Marketing Science, 30(3):
389–404, 2011.
Google. How Search Works, 2014. URL http://www.google.com/intl/en_us/
insidesearch/howsearchworks/thestory/.
Google Official Blog. Personalized Search for Everyone, 2009. URL http://googleblog.blogspot.
com/2009/12/personalized-search-for-everyone.html.
P. Grey. How Many Products Does Amazon Sell?, 2013. URL http://export-x.com/2013/12/15/
many-products-amazon-sell/.
I. Guyon and A. Elisseeff. An Introduction to Variable and Feature Selection. The Journal of Machine Learning Research, 3:1157–1182, 2003.
A. Hannak, P. Sapiezynski, A. Molavi Kakhki, B. Krishnamurthy, D. Lazer, A. Mislove, and C. Wilson.
Measuring Personalization of Web Search. In Proceedings of the 22ndInternational Conference on World Wide Web, pages 527–538. International World Wide Web Conferences Steering Committee, 2013.
E. F. Harrington. Online Ranking/Collaborative Filtering using the Perceptron Algorithm. In ICML, volume 20, pages 250–257, 2003.
T. Hastie, R. Tibshirani, J. Friedman, T. Hastie, J. Friedman, and R. Tibshirani. The Elements of Statistical Learning, volume 2. Springer, 2009.
H. Hong and M. Shum. Using Price Distributions to Estimate Search Costs. RAND Journal of Economics, 37 (2):257–275, Summer 2006.
E. Honka. Quantifying Search and Switching Costs in the US Auto Insurance Industry. The RAND Journal of Economics, 45(4):847–884, 2014.
D. Huang and L. Luo. Consumer Preference Elicitation of Complex Products using Fuzzy Support Vector Machine Active Learning. Marketing Science, 2015.
InstantWatcher. InstantWatcher.com, 2014. URL http://instantwatcher.com/titles/all.
B. J. Jansen, D. L. Booth, and A. Spink. Determining the Informational, Navigational, and Transactional Intent of Web Queries. Information Processing & Management, 44(3):1251–1266, 2008.
K. J¨arvelin and J. Kek¨al¨ainen. IR Evaluation Methods for Retrieving Highly Relevant Documents. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’00, pages 41–48. ACM, 2000.
K. J¨arvelin and J. Kek¨al¨ainen. Cumulated Gain-based Evaluation of IR Techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446, 2002.
Kaggle. Personalized Web Search Challenge, 2013. URL http://www.kaggle.com/c/
yandex-personalized-web-search-challenge.
Kaggle. Private Leaderboard - Personalized Web Search Challenge , 2014. URL https://www.kaggle.
com/c/yandex-personalized-web-search-challenge/leaderboard.
R. Kohavi and G. H. John. Wrappers for Feature Subset Selection. Artificial intelligence, 97(1):273–324, 1997.
S. Koulayev. Search for Differentiated Products: Identification and Estimation. The RAND Journal of Economics, 45(3):553–575, 2014.
A. Lambrecht and C. Tucker. When Does Retargeting Work? Information Specificity in Online Advertising.
Journal of Marketing Research, 50(5):561–576, 2013.
T.-Y. Liu. Learning to Rank for Information Retrieval. Foundations and Trends in Information Retrieval, 3(3):
225–331, 2009.
T. Y. Liu. Learning to Rank for Information Retrieval. Springer, 2011.
P. Masurel, K. Lefvre-Hasegawa, C. Bourguignat, and M. Scordia. Dataiku’s Solution to Yandexs Personalized Web Search Challenge. Technical report, Dataiku, 2014.
D. Metzler and W. B. Croft. Linear Feature-based Models for Information Retrieval. Information Retrieval, 10 (3):257–274, 2007.
D. Metzler and T. Kanungo. Machine Learned Sentence Selection Strategies for Query-biased Summarization.
In SIGIR Learning to Rank Workshop, 2008.
L. Mihalkova and R. Mooney. Learning to Disambiguate Search Queries from Short Sessions. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 111–127, 2009.
K. P. Murphy. Machine Learning: A Probabilistic Perspective. The MIT Press, 2012. ISBN 0262018020, 9780262018029.
S. Narayanan and K. Kalyanam. Position Effects in Search Advertising and Their Moderators: A Regression Discontinuity Approach. Marketing Science, 34(3):388–407, 2015.
O. Netzer, R. Feldman, J. Goldenberg, and M. Fresko. Mine Your Own Business: Market-Structure Surveillance Through Text-Mining. Marketing Science, 31(3):521–543, 2012.
H. Pavliva. Google-Beater Yandex Winning Over Wall Street on Ad
View, 2013. URL http://www.bloomberg.com/news/2013-04-25/
google-tamer-yandex-amasses-buy-ratings-russia-overnight.html.
T. Qin, T.-Y. Liu, J. Xu, and H. Li. LETOR: A Benchmark Collection for Research on Learning to Rank for Information Retrieval. Information Retrieval, 13(4):346–374, 2010.
F. Qiu and J. Cho. Automatic Identification of User Interest for Personalized Search. In Proceedings of the 15th International Conference on World Wide Web, pages 727–736, 2006.
J. R. Quinlan. C4. 5: Programs for Machine Learning, volume 1. Morgan kaufmann, 1993.
J. Reunanen. Overfitting in Making Comparisons Between Variable Selection Methods. The Journal of Machine Learning Research, 3:1371–1382, 2003.
F. Ricci, L. Rokach, and B. Shapira. Introduction to Recommender Systems Handbook. In Recommender Systems Handbook, pages 1–35. Springer, 2011.
P. E. Rossi and G. M. Allenby. Bayesian Statistics and Marketing. Marketing Science, 22(3):304–328, 2003.
P. E. Rossi, R. E. McCulloch, and G. M. Allenby. The Value of Purchase History Data in Target Marketing.
Marketing Science, 15(4):321–340, 1996.
R. E. Schapire. The Strength of Weak Learnability. Machine learning, 5(2):197–227, 1990.
B. Schwartz. Google: Previous Query Used On 0.3% Of Searches, 2012. URL http://www.
seroundtable.com/google-previous-search-15924.html.
S. Seiler. The Impact of Search Csts on Consumer Behavior: A Dynamic Approach. Quantitative Marketing and Economics, pages 1–49, 2013.
G. Song. Point-Wise Approach for Yandex Personalized Web Search Challenge. Technical report, IEEE.org, 2014.
J. H. Steckel and W. R. Vanhonacker. Cross-Validating Regression Models in Marketing Research. Marketing Science, 12(4):415–427, 1993.
D. Sullivan. Bing Results Get Localized & Personalized, 2011. URL http://searchengineland.
com/bing-results-get-localized-personalized-64284.
M. Surdeanu, M. Ciaramita, and H. Zaragoza. Learning to Rank Answers on Large Online QA Collections. In ACL, pages 719–727, 2008.
J. Teevan, S. T. Dumais, and D. J. Liebling. To Personalize or Not to Personalize: Modeling Queries with Variation in User Intent. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 163–170, 2008.
O. Toubia, T. Evgeniou, and J. Hauser. Optimization-Based and Machine-Learning Methods for Conjoint Analysis: Estimation and Question Design. Conjoint Measurement: Methods and Applications, page 231, 2007.
R. M. Ursu. The Power of Rankings: Quantifying the Effect of Rankings on Online Consumer Search and Purchase Decisions. Working Paper, 2016.
M. Volkovs. Context Models For Web Search Personalization. Technical report, University of Toronto, 2014.
Y. Wang, L. Wang, Y. Li, D. He, W. Chen, and T.-Y. Liu. A Theoretical Analysis of NDCG Ranking Measures.
In Proceedings of the 26th Annual Conference on Learning Theory (COLT 2013), 2013.
M. L. Weitzman. Optimal Search for the Best Alternative. Econometrica, 47(3):641–54, May 1979.
Q. Wu, C. J. Burges, K. M. Svore, and J. Gao. Adapting Boosting for Information Retrieval Measures.
Information Retrieval, 13(3):254–270, 2010.
J. Xu and H. Li. Adarank: A Boosting Algorithm for Information Retrieval. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 391–398. ACM, 2007.
Yandex. It May Get Really Personal We have Rolled Out Our Second–Generation Personalised Search Program, 2013. URL http://company.yandex.com/press_center/blog/entry.xml?pid=20.
Yandex. Yandex Announces Third Quarter 2016 Financial Results, 2016. URL http://ir.yandex.
com/releasedetail.cfm?ReleaseID=995776.
YouTube. YouTube Statistics, 2014. URL https://www.youtube.com/yt/press/statistics.
html.
Y. Yue and C. Burges. On Using Simultaneous Perturbation Stochastic Approximation for IR measures, and the Empirical Optimality of LambdaRank. In NIPS Machine Learning for Web Search Workshop, 2007.
J. Zhang and M. Wedel. The Effectiveness of Customized Promotions in Online and Offline Stores. Journal of Marketing Research, 46(2):190–206, 2009.