{"id":872,"date":"2023-12-05T14:21:09","date_gmt":"2023-12-05T14:21:09","guid":{"rendered":"https:\/\/www.wsdm-conference.org\/2024\/?page_id=872"},"modified":"2024-01-18T20:36:28","modified_gmt":"2024-01-18T20:36:28","slug":"wsdm-cup","status":"publish","type":"page","link":"https:\/\/www.wsdm-conference.org\/2024\/wsdm-cup\/","title":{"rendered":"WSDM Cup"},"content":{"rendered":"\n<h2 class=\"wp-block-heading has-text-align-center\"><mark style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-vivid-red-color\">WSDM CUP 2024 &#8211; Xiaohongshu &#8211; Conversational Multi-Doc QA<\/mark><\/h2>\n\n\n\n<h4 class=\"wp-block-heading\">Introduction:<\/h4>\n\n\n\n<div id=\"section-g46dbf3\" class=\"wp-block-gutentor-e1 section-g46dbf3 gutentor-element gutentor-element-advanced-text\"><div class=\"gutentor-text-wrap\"><p class=\"gutentor-text\">Despite progress in large language model-based chatbots, conversational question-answering (QA) is still challenging, especially with current or trending topics. A typical solution involves providing relevant documents for the models to reference. However, it&#8217;s often noted that these documents can overwhelm or mislead the language models. <br><br>In this challenge, we invite you to participate in a conversational QA challenge, featuring a mix of relevant and irrelevant documents from Xiaohongshu. Your systems will be trained on real-world data and assessed based on criteria evaluating both lexical and semantic relatedness. The top-3 teams will be awarded prizes of $1500, $1000, and $500 USD, respectively.<\/p><\/div><\/div>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Competition<\/strong>:<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Title\n<ul class=\"wp-block-list\">\n<li>Conversational Multi-Doc QA<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Rules\n<ul class=\"wp-block-list\">\n<li>Your models are required to answer user questions based on the conversational history and the provided reference documents \n<ul class=\"wp-block-list\">\n<li>Input: History, Reference Documents, Question.<\/li>\n\n\n\n<li>Output: Answer. <\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Model Scale Requirement: Ensure your model size is fewer than 14 billion (14B) parameters. The overall solution will be reviewed after the submission deadline.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Data:<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Description\n<ul class=\"wp-block-list\">\n<li>Format: both training\/eval\/test data will be given in `json` format, each sample includes the following fields: \n<ul class=\"wp-block-list\">\n<li>uuid: string, a unique identifier for each example <\/li>\n\n\n\n<li>history: list of tuples of strings, sequential QA pairs <\/li>\n\n\n\n<li>documents: list of strings, at most 5 reference documents <\/li>\n\n\n\n<li>question: string, user question &#8211; answer: string, reference answer (not given in test data) <\/li>\n\n\n\n<li>keywords: list of strings, reference keywords that should better be mentioned in the reference answer (not given in both training\/eval\/test set)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Example:<\/h4>\n\n\n\n<div id=\"section-g64fbb9\" class=\"wp-block-gutentor-e1 section-g64fbb9 gutentor-element gutentor-element-advanced-text\"><div class=\"gutentor-text-wrap\"><p class=\"gutentor-text\"><strong>Python<\/strong><br># Training example.<br>{<br>&nbsp; &nbsp; &#8220;uuid&#8221;: &#8220;xxxxx&#8221;,<br>&nbsp; &nbsp; &#8220;history&#8221;: [<br>&nbsp; &nbsp; {&#8220;question&#8221;: xxx, &#8220;history&#8221;: xxx},<br>&nbsp; &nbsp; {&#8220;question&#8221;: xxx, &#8220;history&#8221;: xxx},<br>&nbsp; &nbsp; &#8230;<br>&nbsp; &nbsp; ],<br>&nbsp; &nbsp; &#8220;documents&#8221;: [<br>&nbsp; &nbsp; &nbsp; &nbsp; &#8220;Jun 17th through Fri the 21st, 2024 at the Seattle Convention Center, Vancouver Convention Center.&#8221;, &#8220;Workshops within a \u201ctrack\u201d will take place in the same room (or be co-located), and workshop organizers will be asked to work closely with others in their track &#8230;&#8221;,&nbsp;<br>&nbsp; &nbsp; &nbsp; &nbsp; &#8230;<br>&nbsp; &nbsp; ],<br>&nbsp; &nbsp; &#8220;question&#8221;: &#8220;Where will CVPR 2024 happen?&#8221;,<br>&nbsp; &nbsp; &#8220;answer&#8221;: &#8220;CVPR 2024 will happen at the Seattle Convention Center, Vancouver.&#8221;,<br>&nbsp; &nbsp; &#8220;keywords&#8221;: # Will not be given.<br>&nbsp; &nbsp; [<br>&nbsp; &nbsp; &nbsp; &nbsp; &#8220;Vancouver&#8221;, &#8220;CVPR 2024&#8221;, &#8220;Seattle Convention Center&#8221;<br>&nbsp; &nbsp; ]&nbsp;<br>}<br>&nbsp;<br># submission example for eval\/test phase.<br>[<br>&nbsp; &nbsp; {<br>&nbsp; &nbsp; &nbsp; &nbsp; &#8220;uuid&#8221;: &#8220;xxxxx&#8221;,<br>&nbsp; &nbsp; &nbsp; &nbsp; &#8220;prediction&#8221;: &#8220;CVPR 2024 will happen at the Seattle Convention Center, Vancouver.&#8221;,<br>&nbsp; &nbsp; },<br>&nbsp; &nbsp; &#8230;<br>]<\/p><\/div><\/div>\n\n\n\n<h4 class=\"wp-block-heading\"><strong><strong><strong>Evaluation<\/strong><\/strong><\/strong>:<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Submission: \n<ul class=\"wp-block-list\">\n<li>We use codabench to hold the competition, please refer to https:\/\/www.codabench.org\/competitions\/1772\/ for details. <\/li>\n\n\n\n<li>Format: participants should submit their results in `json` format in which a line is actually an example that includes the following fields, \n<ul class=\"wp-block-list\">\n<li>uuid: int, a unique identifier for each test example <\/li>\n\n\n\n<li>prediction: string, your answer <\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Criterion \n<ul class=\"wp-block-list\">\n<li>Metrics: \n<ul class=\"wp-block-list\">\n<li>Keywords Recall: whether the answers contains the truths and the specific keywords by exact matching (see keywords filed in example data). <\/li>\n\n\n\n<li>ROUGE-L: whether the answers are similar to reference answers by fuzzy matching (see answer field in example data). <\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Ranking Procedure: \n<ul class=\"wp-block-list\">\n<li>The overall performance will be determined by examining the mean rank of the above metrics on the Phase 2 (Test set) leaderboard. <\/li>\n\n\n\n<li>In cases where teams have identical mean ranks, preference will be given to the team with the higher ROUGE-L score.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><strong><strong>Organization<\/strong><\/strong>:<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Homepage website: <a href=\"https:\/\/sites.google.com\/view\/wsdm24-docqa\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/sites.google.com\/view\/wsdm24-docqa<\/a><\/li>\n\n\n\n<li>Competition website: <a href=\"https:\/\/www.codabench.org\/competitions\/1772\/\" target=\"_blank\" rel=\"noreferrer noopener\">https:\/\/www.codabench.org\/competitions\/1772\/<\/a><\/li>\n\n\n\n<li>Organizer: Xiaohongshu.Inc<\/li>\n<\/ul>\n\n\n\n<section id=\"section-fcf29a7a-a3c5-4871-b5e5-5c8095cde507\" class=\"wp-block-gutentor-divider section-fcf29a7a-a3c5-4871-b5e5-5c8095cde507 gutentor-element gutentor-section gutentor-divider text-center\"><div class=\"grid-container\"><div class=\"gutentor-divider-box\"><span><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 240 40\" preserveAspectRatio=\"none\"><path d=\"M56.2 20c5.3-.1 10.6-.2 16-.3l16-.2c10.6-.1 21.3-.1 31.9-.2 10.6.1 21.3 0 31.9.1l16 .2c5.3.1 10.6.2 16 .3-5.3.1-10.6.2-16 .3l-16 .2c-10.6.1-21.3.1-31.9.1-10.6-.1-21.3 0-31.9-.2l-16-.2c-5.4.1-10.7 0-16-.1z\"><\/path><\/svg><\/span><\/div><\/div><\/section>\n","protected":false},"excerpt":{"rendered":"<p>WSDM CUP 2024 &#8211; Xiaohongshu &#8211; Conversational Multi-Doc QA Introduction: Competition: Data: Example: Evaluation: Organization:<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-872","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.wsdm-conference.org\/2024\/wp-json\/wp\/v2\/pages\/872","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsdm-conference.org\/2024\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.wsdm-conference.org\/2024\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsdm-conference.org\/2024\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsdm-conference.org\/2024\/wp-json\/wp\/v2\/comments?post=872"}],"version-history":[{"count":23,"href":"https:\/\/www.wsdm-conference.org\/2024\/wp-json\/wp\/v2\/pages\/872\/revisions"}],"predecessor-version":[{"id":986,"href":"https:\/\/www.wsdm-conference.org\/2024\/wp-json\/wp\/v2\/pages\/872\/revisions\/986"}],"wp:attachment":[{"href":"https:\/\/www.wsdm-conference.org\/2024\/wp-json\/wp\/v2\/media?parent=872"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}