[dbjapan] NTCIR-16 タスク提案募集案内（11月27日）

[Takehiro Yamamoto <tyamamot [at] dl.kuis.kyoto-u.ac.jp>]

日本データベース学会の皆様，

兵庫県立大学の山本岳洋です．

情報アクセス技術（情報検索，データベース，自然言語処理等）
に関わる共有タスクをNTCIR-16にて募集しております．

NTCIR-15では対話評価，金融ツイート分析，議会会議録理解，多言語分類，Web検索，統計データ検索，ライフログ検索などのタスクが提案されました．

ある問題を広く周知したい，評価方法を確立したい，
他の方と問題を共有したい方などはぜひご提案いただければ幸いでございます．

タスク提案の締切は11月27日になっております．

どうぞよろしくお願いいたします．

=====
CALL FOR NTCIR-16 TASK PROPOSALS
(Task Proposals Due: November 27, 2020)

NTCIR (NII Testbeds and Community for Information access Research) is a 
series of evaluation conferences that mainly focus on information access 
with East Asian languages and English. The first NTCIR conference 
(NTCIR-1) took place in August/September 1999, and the latest NTCIR-15 
conference will take place in December 2020. Research teams from all 
over the world participate in one or more NTCIR tasks to advance the 
state of the art and to learn from one another's experiences.
As the tasks being run at NTCIR-15 
(http://research.nii.ac.jp/ntcir/ntcir-15/index.html) are reaching the 
final stages, it is time to call for task proposals for the next NTCIR 
(NTCIR-16) which will start in December 2020 and conclude in June 2022. 
Task proposals will be reviewed by the NTCIR Program Committee, 
following the schedule below:

* IMPORTANT DATES:
November 27, 2020 	Task Proposal Submission Due
January 15, 2021	Acceptance Notification of Task Proposals

* NTCIR-16 TENTATIVE SCHEDULE:
Jul 2021:		Dataset Release*
Jul-Dec 2021:		Dry Run*
Sep 2021-Feb 2022:	Formal Run*
Feb 1, 2022:		Evaluation Result Release
Feb 1, 2022:		Draft Task Overview Paper Release
Mar 1, 2022:		Draft Participant Paper Submission Due
May 1, 2022:		All Camera-ready Paper Submission Due
Jun 2022:		NTCIR-16 Conference & EVIA 2022 in NII, Tokyo, Japan
(* indicates that the schedule can be different for different tasks)

* WHO SHOULD SUBMIT NTCIR-16 TASK PROPOSALS?
Organizers of existing NTCIR-15 tasks are required to submit a new 
proposal if they wish to continue them in NTCIR-16. We also welcome new 
task proposals within the broad research area of information access.

To organize an evaluation task is to identify important research 
problems, tackle them strategically by collaborating with other 
researchers (both co-organizers and participants), build the necessary 
evaluation framework to advance the state of the art, and make an impact 
to the research community and to the future.

We encourage applicants to emphasize real-world applications of the 
proposed task by using real-world data, real tasks, and addressing 
real-world problems, and address a challenge in information access 
technology evaluation such as a large amount of assessments required for 
evaluation, privacy-preserving use of proprietary data, and live test 
with real users.


* PROPOSAL TYPES:
We will accept two types of task proposals:

- Proposal of a Core task:
This is for fostering researches on a particular information access 
problem by providing researchers with a common ground for evaluation. 
New test collections and evaluation methods may be developed through the 
collaboration between task organizers (proposers) and task participants. 
At NTCIR-15, the core tasks are: DialEval-1, FinNum-2, QA 
Lab-PoliInfo-2, SHINRA2020-ML, and WWW-3 (Details can be found at 
http://research.nii.ac.jp/ntcir/ntcir-15/tasks.html).

- Proposal of a Pilot task:
This is recommended for organizers who proposed to focus on a novel 
information access problem and there are uncertainties either in task 
designing or organization. It may focus on a sub-problem of an 
information access problem and may attract a smaller group of 
participating teams than core tasks. However, it may grow into a core 
challenging task in the next round of NTCIR. At NTCIR-15, the pilot 
tasks are: Data Search and MART (Details can be found at 
http://research.nii.ac.jp/ntcir/ntcir-15/tasks.html).

Organizers are expected to run their task mainly with their own funding 
and to make the task as self-sustaining as possible. A part of the fund 
can be supported by NTCIR, which is called "seed funding". It is usually 
used for some limited purposes such as hiring relevance assessors. The 
amount of the seed funding allocated to each task varies depending on 
requirements and the total number of accepted tasks. Typical cases would 
be: around 1M JPY for a core task and around 0.5M JPY for a pilot task 
(note that the amount is subject to change).

Please submit your task proposal as a pdf file via EasyChair by Nov 27, 
2020 (Anywhere on Earth). 
https://easychair.org/conferences/?conf=ntcir16taskproposal

* TASK PROPOSAL FORMAT:
The proposal should not exceed six pages in A4 single-column format. The 
first five pages should contain the main part and appendix, and the last 
page should contain only description about the data to be used in the 
task. Please describe the data with as much detail as possible so that 
we can help your data release process after the proposal is accepted. In 
the past NTCIRs, it took much time to create memorandums for data 
release, which sometimes slowed down the task organization.

Main part
- Task name and short name
- Task type (core or pilot)
- Abstract
- Motivation
- Methodology
- Expected results

Appendix
- Names and contact information of the organizers
- Prospective participants
- Data to be used and/or constructed
- Budget planning
- Schedule
- Other notes

Data (to be used in your task)
- Details
(Please describe the details of the data, which should include the 
source of the data, methods to collect the data, range of the data, etc.)
- License
(Please make sure that you have a license to distribute the data, and 
details of the license should be provided. If you do not have permission 
to release the data yet, please describe your plan to get the permission.)
- Distribution
(Please describe how you plan to distribute the data to participants. 
There are mainly three choices: distributed by the data provider, 
distributed by organizers, and distributed by NII.)
- Legal / Ethical issues
(If the data can cause legal or ethical problems, please describe how 
you propose to address them. e.g. some medical data may need approval 
from an ethical committee. e.g. some Web data may need filtering for 
excluding discriminative messages.)

* REVIEW CRITERIA:
- Importance of the task to the information access community and to the 
society
- Timeliness of the task
- Organizers’ commitment in ensuring a successful task
- Financial sustainability (self-sustainable tasks are encouraged)
- Soundness of the evaluation methodology
- Detailed description about the data to be used
- Language scope


* NTCIR-16 PROGRAM Co-Chairs
Takehiro Yamamoto (Co-chair, University of Hyogo, Japan)
Zhicheng Dou (Co-chair, Renmin University of China)

* NTCIR-16 GENERAL CHAIRS:
Charles Clarke (Co-chair, University of Waterloo, Canada)
Noriko Kando (Co-chair, NII, Japan)


--
Takehiro Yamamoto, Associate Professor
School of Social Information Science, University of Hyogo
Email: t.yamamoto [at] sis.u-hyogo.ac.jp
Tel: 078-794-5212
https://rerank-lab.org/