[dbjapan] CFP: NTCIR-9 Once Click Access task

[Tetsuya Sakai <tesakai [at] microsoft.com>]

お邪魔いたします。Microsoft Research Asiaの酒井と申します。

NTCIRの新しい日本語情報アクセスタスクの参加案内をお送りいたします。

参加をご検討いただければ幸いです。どうぞよろしくお願いいたします。

 

********** FIRST CALL FOR PARTICIPATION **********

 

NTCIR-9 1CLICK (ONE CLICK ACCESS) - A subtask of the INTENT task

 

http://www.thuir.org/1click/ntcir9/

 

Task participants registration due: December 20, 2010

Final workshop meeting at NTCIR-9:  December 6-9, 2010 [NII, Tokyo]

 

**************************************************

 

TASK OUTLINE:

 

"1CLICK" is a brand new information access challenge for NTCIR-9.

The goal of this task is to realize the following scenario:

the user enters a query and clicks on the search button -

and his/her information need is immediately satisfied with the

first system output that's displayed. No need to click any further -

hence "once click access".

 

Thus we step out of the "ranked list" and "document relevance" paradigm,

and enable IR, QA and summarization communities to solve a common and

important problem.

 

As this is Round 1 of Once Click Access (“1CLICK”),

only Japanese textual output of length up to X characters will be considered.

The textual output may represent a Search Engine Result Page

(i.e. titles and snippets for a list of retrieved documents),

part of a single document, or any fragments of text.

 

Participating systems are allowed to use any existing web pages

as the knowledge source for producing the system output.

 

 

INPUT TO THE SYSTEM:

 

We will release 60 Japanese queries covering four query types

(15 CELEBRITY, 15 LOCAL, 15 DEFINITION and 15 Question Answering)

as input to the system. These queries have been manually

selected from a Japanese mobile query log and the

Yahoo! Chiebukuro (Japanse Yahoo! Answers) data.

For each query type, we assume that the user is looking for certain types

of factual information.

 

 

OUTPUT FROM THE SYSTEM:

 

We accept two types of "runs" (system output files):

DESKTOP runs (“D-runs”) which use up to X=500 characters (not bytes) for

each query; and

MOBILE runs (“M-runs”) which use up to X=140 characters (not bytes) for

each query.

 

 

EVALUATION METHOD:

 

We plan to design a nugget-based evaluation method.

We will provide a web-based tool for comparing a system output with a list of

nuggets. Thus nugget matches will be identified manually.

 

Unlike traditional summarization and question answering evaluation,

we plan to utilize the position of each matched nugget for computing evaluation

metrics. That is, a nugget match found near the end of the system output will

be discounted compared to one found near the beginning of another

output. Redundancy will be penalized. The goal is to cover the most relevant

nuggets within an X-character window, AND order them so as to minimize the

amount of text the user has to read. We assume that the user reads from top

to bottom, left to right.

 

 

FOR MORE DETAILS:

 

Please visit

http://www.thuir.org/1click/ntcir9/

A link to the NTCIR-9 task particpants registration page

is now available from this page.

Please note that the registration deadline is December 20, 2010 (for all

NTCIR-9 tasks).

 

 

ORGANIZERS:

 

Tetsuya Sakai (Microsoft Research Asia)

Makoto Kato (Kyoto University)

Youngin Song (Microsoft Research Asia)

Ruihua Song (Microsoft Research Asia)

Min Zhang (Tsinghua University)

Yiqun Liu (Tsinghua University)

Nick Craswell (Microsoft)

 

 

QUESTIONS?

 

Please contact Tetsuya Sakai

tesakai [at] microsoft.com