Industry Day

Welcome and Keynote: Feb 3, 09:00-10:00



Liane Lewin-Eytan

Amazon, Israel

Derek Zhiyuan Cheng

Google, USA

Keynote: Beyond Being Accurate: Solving Real-World Recommendation Problems with Neural Modeling


Ed Chi

Principal Research Scientist, Google, USA
View Abstract

Abstract: Fundamental improvements in recommendation and ranking have been much harder to come by, when compared with recent progress on other long-standing AI problems such as visual/audio machine perception and machine translation. Some reasons include: (1) large amounts of data making training difficult, yet having (2) noisy and sparse labels; (3) changing dynamics of context such as user preferences and items; and (4) low-latency requirement for a recommendation response. Beyond that, one recent big challenge is devising approaches to (5) learn more inclusive and fairer models. In this talk, I will touch upon many recent advances in neural modeling techniques for recommendations and their impact in Google products covering ~200 improvements over the last 3 years, including: (1) policy gradient RL techniques with off-policy correction in recurrent recommendation models; (2) multi-task models with gated mixture of experts; (3) diversification and slate optimization with determinantal point processes; (4) large output item spaces with Neural Deep Retrieval; (5) utilizing TPUs for large sparse models; (6) adversarial approaches for ML Fairness for Classifiers and Recommenders.

Session 2: Monday Feb 3, 10:30-12:00

Invited Talk 1: Search ML at Airbnb

10:30 - 11:00

Vanja Josifovski

CTO, Airbnb, USA
View Abstract

Abstract: Search ranking and recommendations are fundamental problems of crucial interest to major Internet companies, including web search engines, content publishing websites and marketplaces. However, despite sharing some common characteristics a one-size-fits-all solution does not exist in this space. Given a large difference in content that needs to be ranked and recommended and the parties involved, each search ranking problem is somewhat specific. Correspondingly, search at Airbnb is quite unique, being a two-sided marketplace in which one needs to optimize for both host and guest preferences, in a world where a user rarely consumes the same item twice and one listing can accept only one guest for a certain set of dates. In this talk, I will discuss our main challenges in Search and describe Machine Learning solutions we have developed for listing ranking and recommendations, both for our Homes and our Experiences Marketplace. I will shed the light on how we transitioned to using Deep Learning in Search, how we deal with bias in Search Data and enforce Diversity. Moreover, I will talk about how we jointly optimize for multiple objectives by leveraging listing quality, location relevance, reviews, host response time as well as guest and host preferences and past booking history. Finally, I will describe ML approaches we use for location ranking in type-ahead suggestions.

Invited Talk 2: Food for Thought: Data Science and Marketing at Grubhub


Wai Gen Yee

Head of Data Science, GrubHub, USA
View Abstract

Abstract: My current position is in Marketing, where I have to either find or nurture customers, generally via email or paid digital advertising, etc. I will cover some of our recommendation techniques, but also describe how my work in Marketing is different than that I've done in prior roles.

Invited Talk 3: Recommending Diverse Audio Content and Making Users Fall in Love


Mounia Lalmas

Head of Tech Research, Spotify, UK
View Abstract

Abstract: Spotify is the biggest and most popular audio streaming platform available today, with as of January 2020, over 200 million monthly active users across 79 markets worldwide listening to over 50 million songs and 500,000 podcasts. We help this audio find the right audience via our recommendation products, which include playlist recommendation, playlist sequencing, and podcast show and episode recommendation. A large percentage of audio consumption is from Home or Search, which make them valuable spaces for surfacing new and diverse content. We talk about two recent research projects we completed on recommendation and search. In one, we track users as they go from zero podcast listening to “podcast lovers” thanks to Spotify’s recommendations. In the other, we compare groups of users by the diversity of music genres they listen to and how that makes them stick with Spotify over time. These two works will appear in their full form in papers at the Web Conference in Taipei in April 2020.

Session 3: Monday Feb 3, 12:05-12:35

[Short Talk] AI the Next Step for Education: Tech Innovations Making Our Classrooms Smarter

Jerry Liu

Head of AI Solution, TAL Education Group, China
View Abstract

Abstract: With the recent development of AI, there has been tremendous changes in both offline and online education. Entire in-class interactions and behaviors between students and instructors have been structured and stored, which provide valuable information for analyzing class performance and improving the learning experience. In this talk, I will first show some successful applications we deployed in TAL's offline and online classrooms. Then I will outline the challenges we meet during the course of building real-world AI+Edu applications. After that, I will talk about the three initiatives we developed on (1) building a quality-assured online learning platform, which is able to utilize multimodal information from the online environment to conduct pedagogical monitoring and alerting; (2) studying a cost-effective and consistent approach of verbal fluency evaluation, which help elementary students improve their oral language skills after school; and (3) getting inside the black box of classroom by neural multimodal learning, which helps teachers get instant feedback on their pedagogical instructions.

[Short Talk] Beyond Keyword Matching in Twitter Search: Interpretable Query to Topic Tweet

Ajeet Grewal

Senior Engineering Manager, Twitter, USA
View Abstract

Abstract: Twitter is a fast growing search engine which enables the discovery of intriguing live conversations on various topics, events, and interests. As with any search engine, purely relying on keyword match will lead to a low recall of all engaging tweets relevant to the query. Because some of these tweets may not contain any of the query keywords, even though they are an indispensable part of the conversations relevant to the query. Such cases are more severe for tweets because tweets are often very short and many tweets only express themselves through images or videos. One solution to this low recall problem is to proactively recommend these missing tweets. To discover and recommend such twees, we propose to project the query and tweets into a common topic space. To avoid user confusion and increase interpretability, instead of adopting a latent topic space, we employ a manually pre-defined topic taxonomy. Following the unified topic taxonomy, tweets are annotated using algorithms with humans in the loop. Targeting a high precision, we map queries to a topic only when they pass an exact string match to the topic entry in the taxonomy. To further ensure the relevance of recommended tweets, we train and deploy a content understanding model based on fine-tuning BERT, which works together with an engagement-based learning-to-rank model, to rank both the recommended and keyword-matched tweets. During both candidate generation and ranking, we use recommendation signals such as the user’s graph, to personalize these results for our users. In this talk, I will describe the creation and maintenance of the topic taxonomy, the semi-automated tweet annotation, and the training and serving of the content understanding model in the production tweet search engine at Twitter. I will also talk about future work of expanding the topic taxonomy, fully automated model for a high precision tweet annotation, relaxing query to topic mapping, and improving the query-tweet content relevance model with semi-supervised learning.

Session 4: Monday Feb 3, 14:00 - 15:00

Keynote: Search and Recommendation in the Modern Workplace


Nitin Agrawal

Corporate Vice President, Microsoft Search, Assistant and Intelligence, USA
View Abstract

Abstract: We have all heard the statistics and personally experienced the increasing amount of time spent on information related tasks in the workplace. More incoming and externally available information requires ever more effort to process. This information often arrives via email but also takes the form of documents, presentation decks, web pages, and textual chat conversations. Most tasks involve instances of information seeking, whether search or browsing. In responding to an email we may need to attach a requested presentation deck, in preparing for a meeting we may re-read a previous email thread, in authoring a document we may search for related content. At Microsoft we have seen the necessity of breaking down the boundaries between search and recommendation both in user experience and platform design to help users efficiently complete their tasks. In this talk we will present some of the technical challenges we are facing in this space ranging from applying deep learning techniques in production to doing ML while maintaining customer privacy.

[Short Talk] Semantic User-Item Segments Search for Recommender Systems

Shiri Simon Segal

Applied AI Researcher, SparkBeyond, Israel
View Abstract

Abstract: Personalized product recommendations are a critical business strategy across a vast commercial range, from entertainment providers to financial organizations. Analyzing patterns of users’ preferences for items is essential for providing contextual recommendations that are supported by evidence. Yet existing methods, are limited in their ability to decipher these patterns because they lack any real understanding of the underlying characteristics that drive the correspondence or mismatch between certain users and certain items. Incorporating semantic knowledge into recommender systems is key to generating such contextual recommendations on which business stakeholders can trust and actually act upon. Understanding the underlying drivers is also a prerequisite for identifying market levers, and ultimately applying macro actions upon user and item segments rather than on individuals. Such macro actions may include planning better campaigns and designing dedicated items for specific users, based on discovered patterns. In this talk, I will describe a method to search and identify subpopulations of user-item pairs, together with the description of their common characteristics, for which the preference of the users for the items is either highly positive or highly negative. This is done by capturing the semantic meaning of the latent factors learned by the recommender system model, based on the users and items metadata. In addition, I will emphasize the ability to improve the accuracy of the model under the cold start scenario (when no historical data is available for either users or items), by utilizing new users’ or items’ characteristics to predict their latent factors and place those predictions in the recommender system model. The analysis can be done either at the individual latent factor level or alternatively, by clustering the users and items in the latent factors space and finding semantic attributes for the clusters. Several real-world examples in which this technique was applied will be presented as well. The audience will learn about these practical methodologies that enhance insightfulness from recommender models and help to deal with cold start scenarios.

Session 5: Monday Feb 3, 15:30-17:00

Invited Talk 4: Towards Large-scale Federated Conversational Intelligence


Sungjin Lee

Principal Scientist, Amazon Alexa AI, USA
View Abstract

Abstract: Conversational agents have become prevalent in every aspect of our lives. Conversational agents such as Alexa and Google Assistant are no longer closed applications but rather they have evolved to be an ecosystem featuring hundreds of thousands of voice skills and offer a rich set of tools to bring in unlimited number of skills, for instance, an AI-centric toolkit for voice skill authoring, seamless cold start of new skills, and traffic optimization based on user satisfaction. In this talk, I discuss recent industrial trends towards large-scale federated conversational intelligence and give a sneak peek of present challenges and approaches in industry.

[Invited Talk 5] Recent Advances and Challenges in E-Commerce Search and Recommendation Systems


Liangjie Hong

Director of Engineering, Data Science and Machine Learning, Etsy, USA
View Abstract

Abstract: Over the years, machine learning has been playing an increasingly important role in E-commerce applications such as search and recommendation systems. A large portion of efforts has focused on how to utilize machine learning to improve immediate sales and user engagement metrics in these systems. However, the nature of E-Commerce marketplaces places a number of unique challenges to help machine learning applications reach their full potential. In this talk, we will highlight some recent advances to tackle these challenges, especially paying attention to the interplay of search and recommendation systems as well as how to understand optimize long term user engagement metrics.

[Short Talk] AI-Driven Email Subject Lines for Marketing Campaigns

Jun Ye

Senior Data Scientist, Microsoft Azure, USA
View Abstract

Abstract: Nowadays, more and more companies employ sending emails as a way to communicate and maintain relationship with customers: to keep customers up-to-date for security and performance updates, to inform them the latest products, to provide them with status updates to their accounts, etc.. When sending out emails to customers, companies have different strategies: some send the same email to every customer, some send customized emails to different cohorts or individuals; some send an email every day, some only send an email when new products are released; etc. Typically, companies use different strategies for different use case scenarios, aiming to improve the engagement with customers at different stages – customizing email subject lines to improve open rate, customizing email content body to improve click through rate (CTR), and customizing the contents to improve conversion rate. However, there is not yet a published method on an end-to-end solution where understanding customers’ preferences on what types of email subject lines they would open and creating such type of email subject lines are combined, together with an A/B testing system to evaluate the effectiveness of the solution. Without such end-to-end solution, it is difficult to scale up supporting fast-paced email campaigns, and it is also difficult to recommend the most appropriate subject lines for customers for different products. In our project, we aim to improve marketing email campaigns’ open rate for consumers by using different emotional attributes to email subject lines, and using a hybrid collaborative filtering model for recommendation.​ Previous studies have shown for promotional email campaigns in the consumer space, emotional attributes of the subject lines impacts customers’ willingness of opening an email. From historical data and domain knowledge, we identified 7 emotional attributes, such as exclusive, anxiety, curiosity, etc. Leveraging such knowledge, we employed a neural network architecture using encoder-decoder framework with attention mechanism and style encoding to generate email subject line variants with controlled emotional attributes. ​ Once the variants are generated, we use hybrid collaborative filtering as our recommendation system to determine which ones will provide higher open rate for a given cohort of users. The user-item matrix is conducted with historical data.​ To evaluate the results of the end-to-end solution, we use A/B testing, where a random set of users get the emails whose subject lines are written by editors (as the control group), and another random set of users get customized subject lines using our model (as treatment group). We collect the email open rate from these groups and compute a hypothesis test to understand if there is any statistically significant difference between the them.​

[Short Talk] Making Content Discoverable is more than just a recommendation algorithm

Daniel Kershaw

Senior Data Scientist, Elsevier, UK
View Abstract

Abstract: At Elsevier, a lot of effort is focussed on content discovery for users, allowing them to find the most relevant articles for their research. This, at its core, blurs the boundaries of search and recommendation as we are both pushing content to the user and allowing them to search the world’s largest catalogue of scientific research. Apart from using the content as is, we can make new content more discoverable with the help of authors at submission time, for example by getting them to write an executive summary of their paper. However, doing this at submission time means that this additional information is not available for older content. This raises the question of how we can utilise the author’s input on new content to create the same feature retrospectively to the whole Elsevier corpus. Focusing on one use case, we discuss how an extractive summarization model (which is trained on the user-submitted summaries), is used to retrospectively generate executive summaries for articles in the catalogue. Further, we show how extractive summarization is used to highlight the salient points (methods, results and finding) within research articles across the complete corpus. This helps users to identify whether an article is of particular interest for them. As a logical next step, we investigate how these extractions can be used to make the research papers more discoverable through connecting it to other papers which share similar findings, methods or conclusion. In this talk we start from the beginning, understanding what users want from summarization systems. We discuss how the proposed use cases were developed and how this ties into the discovery of new content. We then look in more technical detail at what data is available and which methods can be utilised to implement such a system. Finally, while we are working toward taking this extractive summarization system into production, we need to understand the quality of what is being produced before going live. We discuss how internal annotators were used to confirming the quality of the summaries. Though the monitoring of quality does not stop there, we continually monitor user interaction with the extractive summaries as a proxy for quality and satisfaction.


Industry Chairs

Questions about the Industry Day should be directed to: