{"id":5063,"date":"2024-11-18T08:07:42","date_gmt":"2024-11-18T08:07:42","guid":{"rendered":"https:\/\/www.wsdm-conference.org\/2025\/?page_id=5063"},"modified":"2025-03-05T08:48:38","modified_gmt":"2025-03-05T08:48:38","slug":"2025-wsdm-cup-lmsys-multilingual-chatbot-arena","status":"publish","type":"page","link":"https:\/\/www.wsdm-conference.org\/2025\/2025-wsdm-cup-lmsys-multilingual-chatbot-arena\/","title":{"rendered":"2025 WSDM Cup: LMSYS Multilingual Chatbot Arena"},"content":{"rendered":"
\n \n
\n \n \n
\n \n
\n \n
\n \n
\n
\n
\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n <\/div>\n \n
\n \n
\n \n
\n \n
\n
\n \n \n
\n \n
\n <\/path><\/svg><\/span><\/div>\n <\/div>\n <\/div>\n <\/div>\n
\n
\n \n \n
\n \n
\n \n
\n \n \n

Location: Konferenzraum 3<\/h2>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n
\n \n \n
\n \n
\n \n
\n \n
\n
\n \n \n
\n \n
\n \n
\n \n \n

Deadlines<\/h2>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n
\n \n
\n \n
\n
\n
\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n <\/div>\n \n
\n \n
\n \n
\n \n
\n
\n \n \n
\n \n
\n <\/path><\/svg><\/span><\/div>\n <\/div>\n <\/div>\n <\/div>\n
\n
\n \n \n
\n \n
\n \n
\n \n \n

November 18, 2024<\/h3>\n <\/div>\n <\/div>\n
\n \n
\n \n \n
Start of Competition<\/h6>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n
\n
\n
\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n <\/div>\n \n
\n \n
\n \n
\n \n
\n
\n \n \n
\n \n
\n <\/path><\/svg><\/span><\/div>\n <\/div>\n <\/div>\n <\/div>\n
\n
\n \n \n
\n \n
\n \n
\n \n \n

February 3, 2025<\/h3>\n <\/div>\n <\/div>\n
\n \n
\n \n \n
Submission Deadline<\/h6>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n
\n
\n
\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n
\n \n
\n
<\/div>\n <\/div>\n <\/div>\n <\/div>\n \n
\n \n
\n \n
\n \n
\n
\n \n \n
\n \n
\n <\/path><\/svg><\/span><\/div>\n <\/div>\n <\/div>\n <\/div>\n
\n
\n \n \n
\n \n
\n \n
\n \n \n

March 10-14, 2025<\/h3>\n <\/div>\n <\/div>\n
\n \n
\n \n \n
Award Ceremony<\/h6>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n
\n \n
\n \n
\n
\n \n \n
\n \n
\n \n \n
\n

All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted.<\/p>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n

\n \n
\n \n
\n
\n \n \n
\n \n
\n \n \n
\n

\n Participate Here<\/a>\n <\/h2>\n

We are delighted to announce the 2025 WSDM Cup: LMSYS Multilingual Chatbot Arena prediction contest, in collaboration with Kaggle and<\/span> \n LMSYS.org<\/a>.<\/h3>\n

\n
\n <\/p>\n <\/div>\n <\/div>\n

\n \n <\/div>\n
\n \n
\n \n \n

Background<\/h3>\n <\/div>\n <\/div>\n
\n \n \n
\n

As generative AI methods like large language models become increasingly critical to web-scale search and understanding, the problem of LLM evaluation has taken on a role of central importance. Static evaluation datasets have been\n fraught with challenges, both because of issues of contamination and leakage, and because of the difficulty of accurately rating free-text responses generated by these models. Utilizing human raters provides a useful gold standard,\n but this approach has obvious scalability limitations. Using auto raters<\/em> using one LLM as a judge for other LLM outputs is a path that promises scalability, but is only trustworthy when we have a strong basis for demonstrating\n that the auto-rater judge will provide a faithful proxy of human evaluations.<\/p>\n

The creation of useful auto-rater models also aligns with the concept of reward models or preference models in reinforcement learning from human feedback (RLHF). Previous research has identified limitations in directly prompting an\n existing LLM for preference predictions. These limitations often stem from biases such as favoring responses presented first (position bias), being overly verbose (verbosity bias), or exhibiting self-promotion (self-enhancement bias).<\/p>\n <\/div>\n <\/div>\n

\n \n <\/div>\n
\n \n
\n \n \n

The Challenge<\/h2>\n <\/div>\n <\/div>\n
\n \n <\/div>\n
\n \n \n
\n

In this iteration of the WSDM Cup, we challenge competitors to create useful and efficient auto-rater models. In order to provide a strong ability to ensure that these models are indeed reflective of human evaluations, we have\n worked in partnership with\n LMSYS.org<\/a>, which has become one of the leading resources for the field through their well known chatbot arena\n lmarena.ai<\/a>. This arena has led the way in open-ended, community based evaluations of LLMs in blind side-by-side comparison, and provides the perfect resource for training and evaluating auto-rater models.<\/p>\n

Because the WSDM community is global, we focus this competition on the creation of multi-lingual auto-rater models using chatbot arena data.  <\/p>\n

Kaggle is generously hosting the competition and is furthermore providing a prize fund of $50,000 USD. This competition begins on Monday, November 18th, and the final deadline for submissions is on Feb 3, 2025, so now is the time\n to get started.<\/p>\n <\/div>\n <\/div>\n

\n \n
\n \n \n

Get Started<\/h3>\n <\/div>\n <\/div>\n
\n \n \n
\n

To participate in the competition and for full details, please go to the\n WSDM Cup Multilingual Chatbot Arena competition page on Kaggle<\/a>.<\/p>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n

\n \n
\n \n
\n
\n \n \n
\n \n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"

Location: Konferenzraum 3 Deadlines November 18, 2024 Start of Competition February 3, 2025 Submission Deadline March 10-14, 2025 Award Ceremony All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. Participate Here We are delighted to announce the 2025 WSDM Cup: LMSYS Multilingual Chatbot Arena prediction contest, in collaboration with Kaggle […]<\/p>\n","protected":false},"author":10,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"page-templates\/full-width-page.php","meta":{"rank_math_lock_modified_date":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"class_list":["post-5063","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/www.wsdm-conference.org\/2025\/wp-json\/wp\/v2\/pages\/5063","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wsdm-conference.org\/2025\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.wsdm-conference.org\/2025\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.wsdm-conference.org\/2025\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wsdm-conference.org\/2025\/wp-json\/wp\/v2\/comments?post=5063"}],"version-history":[{"count":6,"href":"https:\/\/www.wsdm-conference.org\/2025\/wp-json\/wp\/v2\/pages\/5063\/revisions"}],"predecessor-version":[{"id":6456,"href":"https:\/\/www.wsdm-conference.org\/2025\/wp-json\/wp\/v2\/pages\/5063\/revisions\/6456"}],"wp:attachment":[{"href":"https:\/\/www.wsdm-conference.org\/2025\/wp-json\/wp\/v2\/media?parent=5063"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}