Industry Day

Welcome and Keynote: Monday Feb 3, 09:00-10:00



Liane Lewin-Eytan

Amazon, Israel

Derek Zhiyuan Cheng

Google, USA

[Keynote] Beyond Being Accurate: Solving Real-World Recommendation Problems with Neural Modeling


Ed Chi

Principal Research Scientist, Google, USA
View Abstract

Abstract: Fundamental improvements in recommendation and ranking have been much harder to come by, when compared with recent progress on other long-standing AI problems such as visual/audio machine perception and machine translation. Some reasons include: (1) large amounts of data making training difficult, yet having (2) noisy and sparse labels; (3) changing dynamics of context such as user preferences and items; and (4) low-latency requirement for a recommendation response. Beyond that, one recent big challenge is devising approaches to (5) learn more inclusive and fairer models. In this talk, I will touch upon many recent advances in neural modeling techniques for recommendations and their impact in Google products covering ~200 improvements over the last 3 years, including: (1) policy gradient RL techniques with off-policy correction in recurrent recommendation models; (2) multi-task models with gated mixture of experts; (3) diversification and slate optimization with determinantal point processes; (4) large output item spaces with Neural Deep Retrieval; (5) utilizing TPUs for large sparse models; (6) adversarial approaches for ML Fairness for Classifiers and Recommenders.

View Bio

Bio: Ed H. Chi is a Principal Scientist at Google, leading several machine learning research teams focusing on neural modeling, ML Fairness, reinforcement learning, and recommendation systems in Google Brain . He has delivered significant improvements for YouTube, News, Ads, Google Play Store, and other recommenders at Google with more than 200 product launches in the last 3 years. With 39 patents and over 120 research articles, he is also known for research on user behavior in web and social media.

Coffee Break: 10:00 - 10:30

Session 2: Monday Feb 3, 10:30-12:00

[Invited Talk 1] Search ML at Airbnb

10:30 - 11:00

Vanja Josifovski

CTO, Homes, Airbnb, USA
View Abstract

Abstract: Search ranking and recommendations are fundamental problems of crucial interest to major Internet companies, including web search engines, content publishing websites and marketplaces. However, despite sharing some common characteristics a one-size-fits-all solution does not exist in this space. Given a large difference in content that needs to be ranked and recommended and the parties involved, each search ranking problem is somewhat specific. Correspondingly, search at Airbnb is quite unique, being a two-sided marketplace in which one needs to optimize for both host and guest preferences, in a world where a user rarely consumes the same item twice and one listing can accept only one guest for a certain set of dates. In this talk, I will discuss our main challenges in Search and describe Machine Learning solutions we have developed for listing ranking and recommendations, both for our Homes and our Experiences Marketplace. I will shed the light on how we transitioned to using Deep Learning in Search, how we deal with bias in Search Data and enforce Diversity. Moreover, I will talk about how we jointly optimize for multiple objectives by leveraging listing quality, location relevance, reviews, host response time as well as guest and host preferences and past booking history. Finally, I will describe ML approaches we use for location ranking in type-ahead suggestions.

View Bio

Bio: Vanja Josifovski is the Chief Technology Officer, Homes at Airbnb where he leads efforts around developing technical vision and direction across the Homes business. He leads the Engineering, Data Science, Marketplace Dynamics, and Search Ranking functions. Vanja was most recently CTO at Pinterest, where he was responsible for setting the technical strategy of the company in areas like machine learning and search. Prior to this role, he held positions as the Head of Discovery, Ads Engineering, and Growth Engineering. Before joining Pinterest, Vanja worked on large scale machine learning and information extraction as a Technical Lead at Google Research. His career began with roles at Yahoo Research and IBM Research. Vanja holds a PhD in large scale database systems.

[Invited Talk 2] Food for Thought: Data Science and Marketing at Grubhub


Wai Gen Yee

Head of Data Science, GrubHub, USA
View Abstract

Abstract: My current position is in Marketing, where I have to either find or nurture customers, generally via email or paid digital advertising, etc. I will cover some of our recommendation techniques, but also describe how my work in Marketing is different than that I've done in prior roles.

View Bio

Bio: Wai Gen Yee is Head of Data Science at Grubhub. While he focuses on Marketing, he is really interested in building a strong and productive Data Science culture. In prior roles, he was Chief Data Scientist at Sears Holdings, focusing on pricing, and Chief Scientist at Orbitz, focusing on building their foundational Data Science capabilities. He also taught for a while at the Illinois Institute of Technology. He received his Ph.D. from the Georgia Institute of Technology.

[Invited Talk 3] Recommending Diverse Audio Content and Making Users Fall in Love


Mounia Lalmas

Head of Tech Research, Spotify, UK

Benjamin Carterette

Senior Research Manager, Spotify, USA
View Abstract

Abstract: Spotify is the biggest and most popular audio streaming platform available today, with as of January 2020, over 200 million monthly active users across 79 markets worldwide listening to over 50 million songs and 500,000 podcasts. We help this audio find the right audience via our recommendation products, which include playlist recommendation, playlist sequencing, and podcast show and episode recommendation. A large percentage of audio consumption is from Home or Search, which make them valuable spaces for surfacing new and diverse content. We talk about two recent research projects we completed on recommendation and search. In one, we track users as they go from zero podcast listening to “podcast lovers” thanks to Spotify’s recommendations. In the other, we compare groups of users by the diversity of music genres they listen to and how that makes them stick with Spotify over time. These two works will appear in their full form in papers at the Web Conference in Taipei in April 2020.

View Bio

Bio: Mounia Lalmas is a Director of Research at Spotify, and the Head of Tech Research in Personalization. Before that, she was a Director of Research at Yahoo, where she led a team of researchers working on advertising quality for Gemini, Yahoo native advertising platform. She also worked with various teams at Yahoo on topics related to user engagement in the context of news, search, and user generated content. Prior to this, she held a Microsoft Research/RAEng Research Chair at the School of Computing Science, University of Glasgow. Before that, she was Professor of Information Retrieval at the Department of Computer Science at Queen Mary, University of London. Her work focuses on studying user engagement in areas such as native advertising, digital media, social media, search, and now audio. She has given numerous talks and tutorials on these and related topics, including recently a WWW 2019 tutorial on "Online User Engagement: Metrics and Optimization". She is regularly a senior programme committee member at conferences such as WSDM, KDD, WWW and SIGIR. She was co-programme chair for SIGIR 2015, WWW 2018 and WSDM 2020.

Bio: Ben Carterette is a Sr. Research Manager at Spotify, leading a lab of research scientists in New York working on projects from offline and online experimentation and evaluation, to ML for recommendations and search, to the social impact of Spotify’s recommendations, to knowledge graphs and representations. Prior to joining Spotify, he was an Associate Professor of Computer and Information Sciences at the University of Delaware. He has won grants and several Best Paper Awards for his work on evaluation and experimentation in Information Retrieval, has served as General co-Chair for WSDM 2014 in New York, ICTIR 2016 in Newark, Delaware, and spearheaded the creation of the SIGIR/SIGKDD AFIRM Africa School on ML for Data Mining and Search in 2019. He is currently serving as elected Chair of ACM SIGIR.

Break: 12:00 - 12:05

Session 3: Monday Feb 3, 12:05-12:35

[Short Talk 1] AI the Next Step for Education: Tech Innovations Making Our Classrooms Smarter


Jerry Liu

Head of AI Solution, TAL Education Group, China
View Abstract

Abstract: With the recent development of AI, there has been tremendous changes in both offline and online education. Entire in-class interactions and behaviors between students and instructors have been structured and stored, which provide valuable information for analyzing class performance and improving the learning experience. In this talk, I will first show some successful applications we deployed in TAL's offline and online classrooms. Then I will outline the challenges we meet during the course of building real-world AI+Edu applications. After that, I will talk about the three initiatives we developed on (1) building a quality-assured online learning platform, which is able to utilize multimodal information from the online environment to conduct pedagogical monitoring and alerting; (2) studying a cost-effective and consistent approach of verbal fluency evaluation, which help elementary students improve their oral language skills after school; and (3) getting inside the black box of classroom by neural multimodal learning, which helps teachers get instant feedback on their pedagogical instructions.

View Bio

Bio: Jerry Liu is the Head of AI Solution at TAL Education Group (NYSE:TAL). His research focuses on machine learning and its applications in various domains: recommendation, advertising, and education. He has published in top conference proceedings, such as SIGIR, WWW, AAAI, ICDE, Ubicomp, etc. Before joining TAL, Jerry was a senior research scientist at Pinterest. He worked on building Pinterest large-scale real-time recommendation engine and designing various computational advertising marketplace algorithms of exploration, pacing, and auction. Jerry received his Ph.D degree in Computer Science from University of Pittsburgh.

[Short Talk 2] Beyond Keyword Matching in Twitter Search: Interpretable Query to Topic Tweet


Ajeet Grewal

Senior Engineering Manager, Twitter, USA
View Abstract

Abstract: Twitter is a fast growing search engine which enables the discovery of intriguing live conversations on various topics, events, and interests. As with any search engine, purely relying on keyword match will lead to a low recall of all engaging tweets relevant to the query. Because some of these tweets may not contain any of the query keywords, even though they are an indispensable part of the conversations relevant to the query. Such cases are more severe for tweets because tweets are often very short and many tweets only express themselves through images or videos. One solution to this low recall problem is to proactively recommend these missing tweets. To discover and recommend such twees, we propose to project the query and tweets into a common topic space. To avoid user confusion and increase interpretability, instead of adopting a latent topic space, we employ a manually pre-defined topic taxonomy. Following the unified topic taxonomy, tweets are annotated using algorithms with humans in the loop. Targeting a high precision, we map queries to a topic only when they pass an exact string match to the topic entry in the taxonomy. To further ensure the relevance of recommended tweets, we train and deploy a content understanding model based on fine-tuning BERT, which works together with an engagement-based learning-to-rank model, to rank both the recommended and keyword-matched tweets. During both candidate generation and ranking, we use recommendation signals such as the user’s graph, to personalize these results for our users. In this talk, I will describe the creation and maintenance of the topic taxonomy, the semi-automated tweet annotation, and the training and serving of the content understanding model in the production tweet search engine at Twitter. I will also talk about future work of expanding the topic taxonomy, fully automated model for a high precision tweet annotation, relaxing query to topic mapping, and improving the query-tweet content relevance model with semi-supervised learning.

View Bio

Bio: Ajeet Grewal is a Senior Engineering Manager at Twitter where he currently manages the Search and Recommendations group. The group works on building the best search engine and recommendation systems leveraging natural language processing, graph mining, and machine learning. The group is committed to building software infrastructure and machine learning models to improve the quality of content served to users. Prior to managing this group, he was a Senior Staff Software Engineer at Twitter. He has tech-led some of the largest scale and most impactful recommendations projects at Twitter including MagicRecs which optimizes notifications to be sent to a user given a budget, and “who-to-follow” a link-prediction service that suggests accounts that users would want to follow. He has over 14 years of experience in the field and obtained his Master’s degree in Computer Science from Simon Fraser University with a thesis on Statistical Machine Translation and his Bachelor’s degree from Indian Institute of Technology, Bombay.

Lunch Break: 12:35 - 14:00

Session 4: Monday Feb 3, 14:00 - 15:00

[Keynote] Search and Recommendation in the Modern Workplace


Nitin Agrawal

Corporate Vice President, Microsoft Search, Assistant and Intelligence, USA
View Abstract

Abstract: We have all heard the statistics and personally experienced the increasing amount of time spent on information related tasks in the workplace. More incoming and externally available information requires ever more effort to process. This information often arrives via email but also takes the form of documents, presentation decks, web pages, and textual chat conversations. Most tasks involve instances of information seeking, whether search or browsing. In responding to an email we may need to attach a requested presentation deck, in preparing for a meeting we may re-read a previous email thread, in authoring a document we may search for related content. At Microsoft we have seen the necessity of breaking down the boundaries between search and recommendation both in user experience and platform design to help users efficiently complete their tasks. In this talk we will present some of the technical challenges we are facing in this space ranging from applying deep learning techniques in production to doing ML while maintaining customer privacy.

View Bio

Bio: Nitin Agrawal is responsible for Cortana, Microsoft Search and AI Engineering teams across the Office 365 suite of products. Current key M365 AI initiatives include; search relevance, communication intelligence, meeting and calendar assistance, personal knowledge graph, language understanding, dialog and recommendations.

[Short Talk 3] Semantic User-Item Segments Search for Recommender Systems


Shiri Simon Segal

Applied AI Researcher, SparkBeyond, Israel
View Abstract

Abstract: Personalized product recommendations are a critical business strategy across a vast commercial range, from entertainment providers to financial organizations. Analyzing patterns of users’ preferences for items is essential for providing contextual recommendations that are supported by evidence. Yet existing methods, are limited in their ability to decipher these patterns because they lack any real understanding of the underlying characteristics that drive the correspondence or mismatch between certain users and certain items. Incorporating semantic knowledge into recommender systems is key to generating such contextual recommendations on which business stakeholders can trust and actually act upon. Understanding the underlying drivers is also a prerequisite for identifying market levers, and ultimately applying macro actions upon user and item segments rather than on individuals. Such macro actions may include planning better campaigns and designing dedicated items for specific users, based on discovered patterns. In this talk, I will describe a method to search and identify subpopulations of user-item pairs, together with the description of their common characteristics, for which the preference of the users for the items is either highly positive or highly negative. This is done by capturing the semantic meaning of the latent factors learned by the recommender system model, based on the users and items metadata. In addition, I will emphasize the ability to improve the accuracy of the model under the cold start scenario (when no historical data is available for either users or items), by utilizing new users’ or items’ characteristics to predict their latent factors and place those predictions in the recommender system model. The analysis can be done either at the individual latent factor level or alternatively, by clustering the users and items in the latent factors space and finding semantic attributes for the clusters. Several real-world examples in which this technique was applied will be presented as well. The audience will learn about these practical methodologies that enhance insightfulness from recommender models and help to deal with cold start scenarios.

View Bio

Bio: Shiri Simon Segal is an Applied AI Researcher at SparkBeyond who helps data scientists across a wide range of industries to apply new and better solutions to generate meaningful impact within their businesses. Before SparkBeyond, Shiri earned a PhD in Neuroscience at Tel-Aviv University, Israel, and worked in the Neuroscience industry for several years as a Data Scientist. She has a strong background in advanced analytics, machine learning, and statistics.

Coffee Break: 15:00 - 15:30

Session 5: Monday Feb 3, 15:30-17:00

[Invited Talk 4] Towards Large-scale Federated Conversational Intelligence


Sungjin Lee

Principal Scientist, Amazon Alexa AI, USA
View Abstract

Abstract: Conversational agents have become prevalent in every aspect of our lives. Conversational agents such as Alexa and Google Assistant are no longer closed applications but rather they have evolved to be an ecosystem featuring hundreds of thousands of voice skills and offer a rich set of tools to bring in unlimited number of skills, for instance, an AI-centric toolkit for voice skill authoring, seamless cold start of new skills, and traffic optimization based on user satisfaction. In this talk, I discuss recent industrial trends towards large-scale federated conversational intelligence and give a sneak peek of present challenges and approaches in industry.

View Bio

Bio: Sungjin Lee is a principle scientist at Amazon Alexa AI in Seattle. Prior to joining Amazon, he was a principal researcher at Microsoft Research, Yahoo Labs and a Post-doctoral researcher at Carnegie Mellon University. He received his PhD from Postech in South Korea. His research interests lie in various areas of speech and language processing, spoken dialog system and machine learning. His current research is mainly focused on autonomous learning for large-scale conversational AI by leveraging real user interactions.

[Invited Talk 5] Recent Advances and Challenges in E-Commerce Search and Recommendation Systems


Liangjie Hong

Director of Engineering, Data Science and Machine Learning, Etsy, USA
View Abstract

Abstract: Over the years, machine learning has been playing an increasingly important role in E-commerce applications such as search and recommendation systems. A large portion of efforts has focused on how to utilize machine learning to improve immediate sales and user engagement metrics in these systems. However, the nature of E-Commerce marketplaces places a number of unique challenges to help machine learning applications reach their full potential. In this talk, we will highlight some recent advances to tackle these challenges, especially paying attention to the interplay of search and recommendation systems as well as how to understand optimize long term user engagement metrics.

View Bio

Bio: Liangjie Hong is Director of Engineering, Data Science and Machine Learning at Etsy Inc. I drive Machine Learning and Data Science vision and deliver cutting-edge scientific solutions for Search & Discovery, Personalization & Recommendation, and Computational Advertising. Latest research results are published in SIGIR 2018, WSDM 2019, KDD 2019, WSDM 2020, WWW 2020 and other venues. Prior to Etsy, I was Senior Manager of Research at Yahoo Research.

[Short Talk 4] AI-Driven Email Subject Lines for Marketing Campaigns


Jun Ye

Senior Data Scientist, Microsoft Azure, USA
View Abstract

Abstract: Nowadays, more and more companies employ sending emails as a way to communicate and maintain relationship with customers: to keep customers up-to-date for security and performance updates, to inform them the latest products, to provide them with status updates to their accounts, etc.. When sending out emails to customers, companies have different strategies: some send the same email to every customer, some send customized emails to different cohorts or individuals; some send an email every day, some only send an email when new products are released; etc. Typically, companies use different strategies for different use case scenarios, aiming to improve the engagement with customers at different stages – customizing email subject lines to improve open rate, customizing email content body to improve click through rate (CTR), and customizing the contents to improve conversion rate. However, there is not yet a published method on an end-to-end solution where understanding customers’ preferences on what types of email subject lines they would open and creating such type of email subject lines are combined, together with an A/B testing system to evaluate the effectiveness of the solution. Without such end-to-end solution, it is difficult to scale up supporting fast-paced email campaigns, and it is also difficult to recommend the most appropriate subject lines for customers for different products. In our project, we aim to improve marketing email campaigns’ open rate for consumers by using different emotional attributes to email subject lines, and using a hybrid collaborative filtering model for recommendation.​ Previous studies have shown for promotional email campaigns in the consumer space, emotional attributes of the subject lines impacts customers’ willingness of opening an email. From historical data and domain knowledge, we identified 7 emotional attributes, such as exclusive, anxiety, curiosity, etc. Leveraging such knowledge, we employed a neural network architecture using encoder-decoder framework with attention mechanism and style encoding to generate email subject line variants with controlled emotional attributes. ​ Once the variants are generated, we use hybrid collaborative filtering as our recommendation system to determine which ones will provide higher open rate for a given cohort of users. The user-item matrix is conducted with historical data.​ To evaluate the results of the end-to-end solution, we use A/B testing, where a random set of users get the emails whose subject lines are written by editors (as the control group), and another random set of users get customized subject lines using our model (as treatment group). We collect the email open rate from these groups and compute a hypothesis test to understand if there is any statistically significant difference between the them.​

View Bio

Bio: Jun Ye is a senior data scientist with Microsoft Azure. He received the Ph.D. degree of Computer Science from University of Central Florida (UCF) in 2016. His research interests include cross-modality metric learning, multimodal data retrieval, style-controlled natural language generation and human action recognition. He has published more than 20 academic papers in top peer-reviewed computer vision and machine learning conferences and journals. Dr. Ye has also served as technical program committee members and reviewers for many academic conferences and journals in the fields of pattern recognition, machine learning, computer vision and multimedia.

[Short Talk 5] Making Content Discoverable is more than just a recommendation algorithm


Daniel Kershaw

Senior Data Scientist, Elsevier, UK
View Abstract

Abstract: At Elsevier, a lot of effort is focussed on content discovery for users, allowing them to find the most relevant articles for their research. This, at its core, blurs the boundaries of search and recommendation as we are both pushing content to the user and allowing them to search the world’s largest catalogue of scientific research. Apart from using the content as is, we can make new content more discoverable with the help of authors at submission time, for example by getting them to write an executive summary of their paper. However, doing this at submission time means that this additional information is not available for older content. This raises the question of how we can utilise the author’s input on new content to create the same feature retrospectively to the whole Elsevier corpus. Focusing on one use case, we discuss how an extractive summarization model (which is trained on the user-submitted summaries), is used to retrospectively generate executive summaries for articles in the catalogue. Further, we show how extractive summarization is used to highlight the salient points (methods, results and finding) within research articles across the complete corpus. This helps users to identify whether an article is of particular interest for them. As a logical next step, we investigate how these extractions can be used to make the research papers more discoverable through connecting it to other papers which share similar findings, methods or conclusion. In this talk we start from the beginning, understanding what users want from summarization systems. We discuss how the proposed use cases were developed and how this ties into the discovery of new content. We then look in more technical detail at what data is available and which methods can be utilised to implement such a system. Finally, while we are working toward taking this extractive summarization system into production, we need to understand the quality of what is being produced before going live. We discuss how internal annotators were used to confirming the quality of the summaries. Though the monitoring of quality does not stop there, we continually monitor user interaction with the extractive summaries as a proxy for quality and satisfaction.

View Bio

Bio: Daniel Kershaw is a Senior Data Scientist at Elsevier. Having completed his PhD in the Highlight DTC at Lancaster University in the Prediction of Information Diffusion in Online Social Networks, specifically the emergency and dynamics of new word formation e.g. fleek. Whilst at Elsevier he has worked on and delivered numerous Machine Learning for Personalised information discovery for Mendeley and Science Direct. Currently his focus is on automated text summarisation and knowledge graph construction for personalised information consumption and discovery.


Industry Chairs

Questions about the Industry Day should be directed to: