[dbjapan] NTCIR-13タスク提案募集 (締切: 2016年4月28日)

["Makoto P. Kato" <kato [at] dl.kuis.kyoto-u.ac.jp>]

日本データベース学会の皆様，
（重複して受け取られた場合にはご容赦ください）

京都大学の加藤です．

情報アクセス技術（情報検索，データベース，自然言語処理等）
に関わる共有タスクをNTCIR-13にて募集しております．
タスク提案の締切【4月28日】が近づいて参りましたので，
再度ご案内させていただきます．

NTCIR-12ではWeb検索，電子カルテ分析，音声検索，数式検索，ライフログ検
索，質問応答，対話文生成などのタスクが提案されました．
ある問題を広く周知したい，評価方法を確立したい，
他の方と問題を共有したい方などはぜひご提案いただければ幸いでございます．

どうぞよろしくお願いいたします．

============================================

   NTCIR-13 FINAL CALL FOR TASK PROPOSALS
    (Task Proposal Due: April 28, 2016)

============================================

NTCIR (NII Testbeds and Community for Information access Research) is a
sesquiannual series of evaluation conferences that mainly focuses on
Asian language information access.
The first NTCIR conference (NTCIR-1) took place in August/September
1999, and the latest NTCIR-12 conference will take place in June 2016.
Research teams from all over the world participate in one or more of the
NTCIR tasks to advance the state of the art and to learn from one
another's experiences.

As the tasks being run at NTCIR-12
(http://research.nii.ac.jp/ntcir/ntcir-12/index.html) are reaching the
final stages, it is time to call for task proposals for the next NTCIR
(NTCIR-13) which will start in June 2016 and conclude in December 2017.
Task proposals will be reviewed by the NTCIR Program Committee,
following the schedule below:


IMPORTANT DATES:
Apr 28, 2016	Task Proposals due
May 27, 2016	Acceptance Notification
Jun 10, 2016	Task Planning Sessions and Presentation at the NTCIR-12
conference

NTCIR-13 Tentative Schedule:
Dec 2016:	Task Registration Due
Jan 2017:	Dataset Release
Jan-May 2017:	Dry Run
Mar-July 2017:	Formal Run
Aug 2017:	Evaluation Result Release
Dec 2017:	NTCIR-13 Conference & EVIA 2017 in NII, Tokyo, Japan

Organizers of existing NTCIR-12 tasks (IMine, MedNLPDoc, MobileClick,
SpokenQuery&Doc, Temporalia, MathIR, Lifelog, QALab, and STC) are
required to submit a new proposal if they wish to continue them in
NTCIR-13. We also welcome new task proposals within the broad research
area of information access. New task organizers will be given many
opportunities to consult and learn from senior PC members for task
design and organization (see MENTORING). To organize an evaluation task
is to identify important research problems, tackle them strategically by
collaborating with other researchers (participants), build the necessary
evaluation framework to advance the state of the art, and make an impact
to the research community and to the future.


PROPOSAL TYPES:

We will accept two types of task proposals:

- Proposal of a Core task:
This is for fostering researches on a particular information access
problem by providing researchers with a common ground for evaluation.
New test collections and evaluation methods may be developed through the
collaboration between task organizers (proposers) and task participants.
At NTCIR-12, the core tasks were: IMine, MedNLPDoc, MobileClick,
SpokenQuery&Doc, Temporalia, and MathIR.

- Proposal of a Pilot task:
This is recommended for organizers who proposed to focus on a novel
information access problem and there are uncertainties in task designing
and organization. It may focus on a sub-problem of an information access
problem and may attract a smaller group of participating teams than core
tasks. However, it may grow into a core challenging task in the next
round of NTCIR. At NTCIR-12, the pilot tasks were: Lifelog, QALab, and STC.

Organizers are expected to run their task mainly with their own fundings
and to make the task as self-sustaining as possible. A part of the fund
can be supported by NTCIR, which we call "seed funding". It is usually
used for some limited purposes such as hiring relevance assessors. The
amount of the seed funding allocated to each task varies depending on
requirements and the total number of accepted tasks, but typical cases
would be: around 1M JPY for a core task and around 0.5M JPY for a pilot
task (note that the amount may be changed in NTCIR-13).

Please submit your task proposal as a pdf file to "NTCIR-13 Task
Proposal" track in the following EasyChair site by April 28, 2016 (UTC-12).
https://easychair.org/conferences/?conf=ntcir12


MENTORING:

We offer a mentoring program especially for those who propose a task for
the first time. If you would like help in your task organization, you
may ask for a mentor: a person who will help you organize a task usually
via e-mail. A mentor can also familiarize you with the standards and
schedules of NTCIR. He or she will also help you understand how to
design tasks, test collections, and evaluation methodology. Mentors are
volunteers who are familiar with successful proposals. To request a
mentor, please include a “Request for mentoring” section in your
proposal, which must include problems for which you need support (e.g.
scheduling, design of task, evaluation methodology, etc.). Since this
program just begins from NTCIR-13 and is still at a trial stage, we may
not fully meet your requests.


DIFFERENCE FROM THE PAST NTCIR TASK PROPOSAL:

- We have a plan to send acceptance notification before the NTCIR-12
conference, unlike the previous rounds where it was sent after the
conferences.
- Organizers at NTCIR-12 may keep the task name (e.g. TASKNAME-2) as
long as the task includes improvement based on the previous rounds.
- Please describe the data with as much detail as possible so that we
can help your data release after the proposal is accepted. In the past
NTCIR, it took much time to create memorandums, which sometimes slowed
down the task organization.
- We offer a mentoring program.
- The proposal should not exceed four pages in A4 single-column format.
- We encourage applicants to
  - Emphasize real-world applications of the proposed task by using
real-world data, real tasks, and addressing real-world problems.
  - Address a challenge in information access technology evaluation such
as a large amount of assessments required for evaluation,
privacy-preserving use of proprietary data, and live test with real users.


TASK PROPOSAL FORMAT:

The proposal should not exceed four pages in A4 single-column format.

Main part
- Task name and short name
- Task type (core or pilot)
- Abstract
- Motivation
- Methodology
- Expected results

Appendix
- Names and contact information of the organizers
- Prospective participants
- Data to be used and/or constructed
- Budget planning
- Schedule
- Request for mentoring (Include this section if you would like to apply
for the mentoring program; otherwise, exclude this section)
- Other notes


REVIEW CRITERIA:
- Importance of the task to the information access community and to the
society
- Timeliness of the task
- Organizers’ commitment in ensuring a successful task
- Financial sustainability (self-sustainable tasks are encouraged)
- Soundness of the evaluation methodology
- Language scope


NTCIR-13 PROGRAM COMMITTEE
Tat-Seng Chua (National University of Singapore, Singapore)
Nicola Ferro (University of Padua, Italy)
Kalervo Jarvelin (University of Tampere, Finland)
Gareth Jones (Dublin City University, Ireland)
Chin-Yew Lin (Microsoft Research Asia, China)
Mandar Mitra (Indian Statistical Institute, India)
Maarten de Rijke (University of Amsterdam, The Netherlands)
Mark Sanderson (RMIT University, Australia)
Ian Soboroff (NIST, United States)
Yiqun Liu (Co-chair, Tsinghua University, China)
Makoto P. Kato (Co-chair, Kyoto University, Japan)

NTCIR GENERAL CHAIRS:
Charles Clarke (University of Waterloo, Canada)
Noriko Kando (NII, Japan)
Tetsuya Sakai (Waseda University, Japan)
===