Industry Day

Abstract: Search ranking and recommendations are fundamental problems of crucial interest to major Internet companies, including web search engines, content publishing websites and marketplaces. However, despite sharing some common characteristics a one-size-fits-all solution does not exist in this space. Given a large difference in content that needs to be ranked and recommended and the parties involved, each search ranking problem is somewhat specific. Correspondingly, search at Airbnb is quite unique, being a two-sided marketplace in which one needs to optimize for both host and guest preferences, in a world where a user rarely consumes the same item twice and one listing can accept only one guest for a certain set of dates. In this talk, I will discuss our main challenges in Search and describe Machine Learning solutions we have developed for listing ranking and recommendations, both for our Homes and our Experiences Marketplace. I will shed the light on how we transitioned to using Deep Learning in Search, how we deal with bias in Search Data and enforce Diversity. Moreover, I will talk about how we jointly optimize for multiple objectives by leveraging listing quality, location relevance, reviews, host response time as well as guest and host preferences and past booking history. Finally, I will describe ML approaches we use for location ranking in type-ahead suggestions.

Bio: Vanja Josifovski is the Chief Technology Officer, Homes at Airbnb where he leads efforts around developing technical vision and direction across the Homes business. He leads the Engineering, Data Science, Marketplace Dynamics, and Search Ranking functions. Vanja was most recently CTO at Pinterest, where he was responsible for setting the technical strategy of the company in areas like machine learning and search. Prior to this role, he held positions as the Head of Discovery, Ads Engineering, and Growth Engineering. Before joining Pinterest, Vanja worked on large scale machine learning and information extraction as a Technical Lead at Google Research. His career began with roles at Yahoo Research and IBM Research. Vanja holds a PhD in large scale database systems.

Abstract: Spotify is the biggest and most popular audio streaming platform available today, with as of January 2020, over 200 million monthly active users across 79 markets worldwide listening to over 50 million songs and 500,000 podcasts. We help this audio find the right audience via our recommendation products, which include playlist recommendation, playlist sequencing, and podcast show and episode recommendation. A large percentage of audio consumption is from Home or Search, which make them valuable spaces for surfacing new and diverse content. We talk about two recent research projects we completed on recommendation and search. In one, we track users as they go from zero podcast listening to “podcast lovers” thanks to Spotify’s recommendations. In the other, we compare groups of users by the diversity of music genres they listen to and how that makes them stick with Spotify over time. These two works will appear in their full form in papers at the Web Conference in Taipei in April 2020.

Bio: Mounia Lalmas is a Director of Research at Spotify, and the Head of Tech Research in Personalization. Before that, she was a Director of Research at Yahoo, where she led a team of researchers working on advertising quality for Gemini, Yahoo native advertising platform. She also worked with various teams at Yahoo on topics related to user engagement in the context of news, search, and user generated content. Prior to this, she held a Microsoft Research/RAEng Research Chair at the School of Computing Science, University of Glasgow. Before that, she was Professor of Information Retrieval at the Department of Computer Science at Queen Mary, University of London. Her work focuses on studying user engagement in areas such as native advertising, digital media, social media, search, and now audio. She has given numerous talks and tutorials on these and related topics, including recently a WWW 2019 tutorial on "Online User Engagement: Metrics and Optimization". She is regularly a senior programme committee member at conferences such as WSDM, KDD, WWW and SIGIR. She was co-programme chair for SIGIR 2015, WWW 2018 and WSDM 2020.

Bio: Ben Carterette is a Sr. Research Manager at Spotify, leading a lab of research scientists in New York working on projects from offline and online experimentation and evaluation, to ML for recommendations and search, to the social impact of Spotify’s recommendations, to knowledge graphs and representations. Prior to joining Spotify, he was an Associate Professor of Computer and Information Sciences at the University of Delaware. He has won grants and several Best Paper Awards for his work on evaluation and experimentation in Information Retrieval, has served as General co-Chair for WSDM 2014 in New York, ICTIR 2016 in Newark, Delaware, and spearheaded the creation of the SIGIR/SIGKDD AFIRM Africa School on ML for Data Mining and Search in 2019. He is currently serving as elected Chair of ACM SIGIR.

Abstract: With the recent development of AI, there has been tremendous changes in both offline and online education. Entire in-class interactions and behaviors between students and instructors have been structured and stored, which provide valuable information for analyzing class performance and improving the learning experience. In this talk, I will first show some successful applications we deployed in TAL's offline and online classrooms. Then I will outline the challenges we meet during the course of building real-world AI+Edu applications. After that, I will talk about the three initiatives we developed on (1) building a quality-assured online learning platform, which is able to utilize multimodal information from the online environment to conduct pedagogical monitoring and alerting; (2) studying a cost-effective and consistent approach of verbal fluency evaluation, which help elementary students improve their oral language skills after school; and (3) getting inside the black box of classroom by neural multimodal learning, which helps teachers get instant feedback on their pedagogical instructions.

Bio: Jerry Liu is the Head of AI Solution at TAL Education Group (NYSE:TAL). His research focuses on machine learning and its applications in various domains: recommendation, advertising, and education. He has published in top conference proceedings, such as SIGIR, WWW, AAAI, ICDE, Ubicomp, etc. Before joining TAL, Jerry was a senior research scientist at Pinterest. He worked on building Pinterest large-scale real-time recommendation engine and designing various computational advertising marketplace algorithms of exploration, pacing, and auction. Jerry received his Ph.D degree in Computer Science from University of Pittsburgh.

Abstract: We have all heard the statistics and personally experienced the increasing amount of time spent on information related tasks in the workplace. More incoming and externally available information requires ever more effort to process. This information often arrives via email but also takes the form of documents, presentation decks, web pages, and textual chat conversations. Most tasks involve instances of information seeking, whether search or browsing. In responding to an email we may need to attach a requested presentation deck, in preparing for a meeting we may re-read a previous email thread, in authoring a document we may search for related content. At Microsoft we have seen the necessity of breaking down the boundaries between search and recommendation both in user experience and platform design to help users efficiently complete their tasks. In this talk we will present some of the technical challenges we are facing in this space ranging from applying deep learning techniques in production to doing ML while maintaining customer privacy.

Bio: Nitin Agrawal is responsible for Cortana, Microsoft Search and AI Engineering teams across the Office 365 suite of products. Current key M365 AI initiatives include; search relevance, communication intelligence, meeting and calendar assistance, personal knowledge graph, language understanding, dialog and recommendations.

Abstract: Conversational agents have become prevalent in every aspect of our lives. Conversational agents such as Alexa and Google Assistant are no longer closed applications but rather they have evolved to be an ecosystem featuring hundreds of thousands of voice skills and offer a rich set of tools to bring in unlimited number of skills, for instance, an AI-centric toolkit for voice skill authoring, seamless cold start of new skills, and traffic optimization based on user satisfaction. In this talk, I discuss recent industrial trends towards large-scale federated conversational intelligence and give a sneak peek of present challenges and approaches in industry.

Bio: Sungjin Lee is a principle scientist at Amazon Alexa AI in Seattle. Prior to joining Amazon, he was a principal researcher at Microsoft Research, Yahoo Labs and a Post-doctoral researcher at Carnegie Mellon University. He received his PhD from Postech in South Korea. His research interests lie in various areas of speech and language processing, spoken dialog system and machine learning. His current research is mainly focused on autonomous learning for large-scale conversational AI by leveraging real user interactions.

Abstract: Nowadays, more and more companies employ sending emails as a way to communicate and maintain relationship with customers: to keep customers up-to-date for security and performance updates, to inform them the latest products, to provide them with status updates to their accounts, etc.. When sending out emails to customers, companies have different strategies: some send the same email to every customer, some send customized emails to different cohorts or individuals; some send an email every day, some only send an email when new products are released; etc. Typically, companies use different strategies for different use case scenarios, aiming to improve the engagement with customers at different stages – customizing email subject lines to improve open rate, customizing email content body to improve click through rate (CTR), and customizing the contents to improve conversion rate. However, there is not yet a published method on an end-to-end solution where understanding customers’ preferences on what types of email subject lines they would open and creating such type of email subject lines are combined, together with an A/B testing system to evaluate the effectiveness of the solution. Without such end-to-end solution, it is difficult to scale up supporting fast-paced email campaigns, and it is also difficult to recommend the most appropriate subject lines for customers for different products. In our project, we aim to improve marketing email campaigns’ open rate for consumers by using different emotional attributes to email subject lines, and using a hybrid collaborative filtering model for recommendation.​ Previous studies have shown for promotional email campaigns in the consumer space, emotional attributes of the subject lines impacts customers’ willingness of opening an email. From historical data and domain knowledge, we identified 7 emotional attributes, such as exclusive, anxiety, curiosity, etc. Leveraging such knowledge, we employed a neural network architecture using encoder-decoder framework with attention mechanism and style encoding to generate email subject line variants with controlled emotional attributes. ​ Once the variants are generated, we use hybrid collaborative filtering as our recommendation system to determine which ones will provide higher open rate for a given cohort of users. The user-item matrix is conducted with historical data.​ To evaluate the results of the end-to-end solution, we use A/B testing, where a random set of users get the emails whose subject lines are written by editors (as the control group), and another random set of users get customized subject lines using our model (as treatment group). We collect the email open rate from these groups and compute a hypothesis test to understand if there is any statistically significant difference between the them.​

Bio: Jun Ye is a senior data scientist with Microsoft Azure. He received the Ph.D. degree of Computer Science from University of Central Florida (UCF) in 2016. His research interests include cross-modality metric learning, multimodal data retrieval, style-controlled natural language generation and human action recognition. He has published more than 20 academic papers in top peer-reviewed computer vision and machine learning conferences and journals. Dr. Ye has also served as technical program committee members and reviewers for many academic conferences and journals in the fields of pattern recognition, machine learning, computer vision and multimedia.