[dbjapan] CFP: NTCIR-9 Once Click Access task
- To: "dbjapan [at] dbsj.org" <dbjapan [at] dbsj.org>
- Subject: [dbjapan] CFP: NTCIR-9 Once Click Access task
- From: Tetsuya Sakai <tesakai [at] microsoft.com>
- Date: Tue, 2 Nov 2010 04:01:29 +0000
- Accept-language: ja-JP, en-US
- Thread-topic: CFP: NTCIR-9 Once Click Access task
お邪魔いたします。Microsoft Research Asiaの酒井と申します。
********** FIRST CALL FOR PARTICIPATION **********
NTCIR-9 1CLICK (ONE CLICK ACCESS) - A subtask of the INTENT task
Task participants registration due: December 20, 2010
Final workshop meeting at NTCIR-9: December 6-9, 2010 [NII, Tokyo]
"1CLICK" is a brand new information access challenge for NTCIR-9.
The goal of this task is to realize the following scenario:
the user enters a query and clicks on the search button -
and his/her information need is immediately satisfied with the
first system output that's displayed. No need to click any further -
hence "once click access".
Thus we step out of the "ranked list" and "document relevance" paradigm,
and enable IR, QA and summarization communities to solve a common and
As this is Round 1 of Once Click Access (“1CLICK”),
only Japanese textual output of length up to X characters will be considered.
The textual output may represent a Search Engine Result Page
(i.e. titles and snippets for a list of retrieved documents),
part of a single document, or any fragments of text.
Participating systems are allowed to use any existing web pages
as the knowledge source for producing the system output.
INPUT TO THE SYSTEM:
We will release 60 Japanese queries covering four query types
(15 CELEBRITY, 15 LOCAL, 15 DEFINITION and 15 Question Answering)
as input to the system. These queries have been manually
selected from a Japanese mobile query log and the
Yahoo! Chiebukuro (Japanse Yahoo! Answers) data.
For each query type, we assume that the user is looking for certain types
of factual information.
OUTPUT FROM THE SYSTEM:
We accept two types of "runs" (system output files):
DESKTOP runs (“D-runs”) which use up to X=500 characters (not bytes) for
each query; and
MOBILE runs (“M-runs”) which use up to X=140 characters (not bytes) for
We plan to design a nugget-based evaluation method.
We will provide a web-based tool for comparing a system output with a list of
nuggets. Thus nugget matches will be identified manually.
Unlike traditional summarization and question answering evaluation,
we plan to utilize the position of each matched nugget for computing evaluation
metrics. That is, a nugget match found near the end of the system output will
be discounted compared to one found near the beginning of another
output. Redundancy will be penalized. The goal is to cover the most relevant
nuggets within an X-character window, AND order them so as to minimize the
amount of text the user has to read. We assume that the user reads from top
to bottom, left to right.
FOR MORE DETAILS:
A link to the NTCIR-9 task particpants registration page
is now available from this page.
Please note that the registration deadline is December 20, 2010 (for all
Tetsuya Sakai (Microsoft Research Asia)
Makoto Kato (Kyoto University)
Youngin Song (Microsoft Research Asia)
Ruihua Song (Microsoft Research Asia)
Min Zhang (Tsinghua University)
Yiqun Liu (Tsinghua University)
Nick Craswell (Microsoft)
Please contact Tetsuya Sakai
tesakai [at] microsoft.com
- Prev by Date: [dbjapan] VLDB2010勉強会のご案内 (11/27) ※会場決定
- Next by Date: [dbjapan] 「日本データベース学会論文誌Vol.9,No.3」 自由投稿論文 募集のお知らせ