CFP for NTCIR-6

[Atsushi Fujii <fujii [at] slis.tsukuba.ac.jp>]

*** Apologies for multiple copies ***


**********************************************************
         Call for Participation to NTCIR-6
Evaluation of Information Access Technologies: IR, QA and
     Cross-lingual Information Access Technologies

   http://research.nii.ac.jp/ntcir/ntcir-ws6/
      http://research.nii.ac.jp/ntcir/
Final Meeting: 15-18 May 2007, NII, Tokyo, Japan
**********************************************************


The NTCIR is a series of evaluation workshop to enhance the
research in information access technologies, including text
retrieval, question answering, cross-lingual information
access, etc, by providing infrastructure of evaluation
and research including test collections, evaluation metrics
and methodologies, and a forum of researchers.

NTCIR has used East Asian Language documents but attracted
international participation. For example NTCIR-5 has
participants from 15 different countries and areas including
Australia, Canada, China PRC, Germany, Hong Kong, Ireland, Japan,
Korea, Netherlands, Singapore, Spain, Switzerland, Taiwan ROC,
UK, United States.


TASKS:

NTCIR-6 selected the following 4 areas of research as
"Tasks" and 1 area as a "pilot workshop"; Other "pilot tasks"
can be started any time during the process of NTCIR-6.


1. Cross-Lingual Information Retrieval Task (CLIR)
    Multi- and Bi-lingual CLIR, and
    Single language IR
    Languages: Traditional Chinese, Korean, and Japanese.
    Simplified Chinese can be added. To conclude the CLIR to
    news documents, 4 CLIR test collections, NTCIR-3 through -6
    will be used and Cross-collection analysis will be done.
    New metrics for graded relevance judgments are used.
    Discussion about evaluation metrics and methodologies
    are also welcome.

2. Cross-Language Question Answering Task (CLQA)
    Focus on Named Entities, which are one of the problems
    in CL information access in Asian context.
    Traditional Chinese, English and Japanese. Korean language
    is under consideration. Volunteer for cooperation to
    organize Korean part is welcome.

3. Patent Retrieval Task (PATENT)
    Retrieval task: "Invalidity search" Using Japanese patents
    (10 year patent application fulltexts) and US patents.
    Use 4 patent collections of NTCIR-3 through -6.
    Classification task: Multi-viewpoint categorization. The
    purpose is to categorize target patent applications based
    on the F-term classification system.

4. Question Answering Task (QAC)
    Question answering beyond factoid questions and their evaluation
    method. Focusing to complicated questions like "WHY".

5. Pilot Tasks
    Any attempts to test the problems to be solved in short
    time, or feasibility studies for the future tasks. To be
    announced later. Opinion Extraction, Multilingual Multi-
    document Summarization, Evaluation of WEB Search engine
    are now under consideration, but not decided

6. Pilot Workshop: Multimodal Summarization of Trend Information (MuST)
     Focusing numerical information in the text. Extract
numerical information representing the trends from multiple
documents, analyze, summarize and display visully. In the
last MuST Workshop, we constructed an annotated corpus and
shared it. Research groups from various disciplinaries share
the corpus and work various direction of research using
the corpus. Currently using Japanese documents only.
To be announced later.



WHAT'S NEW to NTCIR-6:

(1) Cross-Collection Evaluation:

In IR related tasks, multiple test collections for the same
document genres and the same users' tasks will be used and
analyze the variability across the collections.


(2) Submission Raw Data and Evaluation Results:

All the submitted runs and their evaluated results using the
metrics set by the task will be available for the active
participants of the task. The purpose of this is that we
invite all the task participants to discuss, analyze or
examine the evaluation metrics and evaluation results of
the task.

Evaluation is very critical issue for all of the researchers.
So please examine how the evaluation is done and how the
metrics behave, or whether there are any methods to
overcome the limitation of current practice of the
evaluation. With your cooperation, we would like to
obtain fruitful examination of the evaluation results
and metrics.


OPEN SUBMISSION SESSION:

Conjunction with NTCIR-6 Final Meeting, a separate
conference called "Open Submission Session" will be held.
This is a refereed international conference on research
in Information Access and their evaluation.
Detailed information will be announced later.



IMPORTANT DATES:

    Registration Deadline:             May 31, 2006
    Document Set Release:              June 1, 2006
    Dry Run:                       July - September, 2006
    Formal Run：                   October - December, 2006
    Evaluation Results Return:      by February 1, 2007
    Paper for the Proceedings Due :    March 1, 2007
    Final Meeting, Tokyo, Japan:     May 15-18, 2007



** ONLINE REGISTRATION IS AVAILABLE NOW !!


INQUIRY:
Noriko Kando at kando (at) nii.ac.jp


Your participation is more than welcome. Please join and work together!


Noriko.
---
Noriko Kando
ntcir project