講演会：Evaluation Design for Retrieval of Conversations, JULY 10 13:00-14:30, Room2208

[Noriko Kando <kando [at] nii.ac.jp>]

(重複して受け取られた場合はご容赦下さい)

日本データベース学会の皆様：

国立情報学研究所の神門 典子と申します。

開催日が近づきましたので、改めて、Evaluation Design for Retrieval of 
Conversationsについての講演会について、ご案内申し上げます。

みなさまのご参加をお待ちしております。
また、周囲のご関心のある方々やメーリングリストなどに、適宜、ご紹介や転送 
をしていただければ、幸いです。

よろしくお願いいたします

神門 典子
----------------------------------------------------------------------
Title:  Evaluation Design for Retrieval of Conversations

Speaker:  Douglas W. Oard, University of Maryland, USA

Date:  Tuesday, July 10th, at 13:00-14:30

Place:  Room 2208, 22F, NII


Abstract:
Current approaches to the evaluation of information retrieval systems 
are based on several assumptions, including: (1) documents have a 
beginning and an end and they contain words, and (2) if we show the user 
the right documents they can recognize them.  These are reasonable 
assumptions when searching documents such as news stories, books, and 
patents that were written originally with dissemination as a principal 
goal.  But what about conversations?  Is a conversation one "document" 
or many?  Where does a conversation begin? Where does it end?  If we 
can't unambiguously identify the unit of retrieval, many current 
approaches to evaluation would be ruled out.  Even if we could somehow 
answer that question, ranked retrieval evaluation measures implicitly 
assume that the user can recognize relevant documents.  That may work 
well when searching your own conversations, but try searching a few 
recorded conversations from the Apollo program, or President Kennedy's 
recorded meetings during the Cuban Missile Crisis, or President 
Clinton's email.  You may need a "reader's guide" to figure out who the 
people are, and the words they are using mean.   If such "sensemaking" 
capabilities are needed, can we devise some insightful, affordable and
repeatable way of evaluating them?  Perhaps our greatest challenge, 
however, is that we often have an expectation of privacy when engaging 
in conversations.  Where, then, can we get the large collections of
conversations that we would need to build realistic test collections? If 
we get public conversations (e.g., political debates), we risk not
modeling private conversations well.  Alternatively, we could use policy
measures to protect sensitive collections used in our research, much as 
we presently do when working with copyrighted material.  We have, 
however, little experience protecting privacy, so we might have much to 
learn as a community about how to actually do this.  A third alternative 
would be to use an "algorithm deposit" model in which we send each 
retrieval algorithm to a secure data center and run it there, returning 
only the evaluation result.  We have some experience with such an 
approach, and thus some basis for assessing its strengths and 
weaknesses.  In this talk, I will step through each of these issues, 
fleshing each out with one or more examples. My goal will not be to 
answer any of these questions definitively, but rather to ask each of 
them clearly so that we can begin to work together on finding good answers.



About the Speaker:
Douglas Oard is a Professor at the University of Maryland, College Park,
with joint appointments in the College of Information Studies and the
Institute for Advanced Computer Studies, where he is the director of the
Computational Linguistics and Information Processing Lab.  He is a 
Visiting Professor at the National Institute of Informatics, and a 
General Co-Chair of NTCIR-10.  Dr. Oard earned his Ph.D. in Electrical 
Engineering from the University of Maryland.  His research interests 
center around the use of emerging technologies to support end user 
information seeking.  Additional information is available at 
http://terpconnect.umd.edu/~oard/

Academic Host: Noriko Kando, Professor, NII
Inquiry: Email to kando-secr [at] nii.ac.jp (ex. 2733)
------------------------------------------------------------------------