Jean-Baptiste Berthelin

Text mining applies information-extracting algorithms on large natural language text collections. The DEFT text-mining challenges have been an opportunity to demonstrate the diversity of techniques in this field, and yielded high-quality French written text corpora. The paper states possible definitions for text mining, along with its particular meaning within DEFT. Each campaign has been organized along definite steps, giving rise to specific problems, among which, the adjustment of measuring scales associated with opinion documents. Last, we examine matters in opinion meaning (present in the 2007 and 2009 campaigns) and the question of subjectivity in texts ant its processing by statistic or symbolic methods.

From 2005 onward, the French DEFT evaluation campaigns have been offering exploratory topics in text mining. The 2007 challenge was about classifying opinion texts, i. E., assigning an opinion class to each text in a corpus. The paper presents an analytic overview of results obtained by the competitors in the challenge, as well as a synthetic assessment of methods that were submitted to evaluation.

The relevance of human judgment in an evaluation campaign is illustrated here through the DEFT text mining campaigns. In a first step, testing a topic for a campaign among a limited number of human evaluators informs us about the feasibility of a task. This information comes from the results obtained by the judges, as well as from their personal impressions after passing the test. In a second step, results from individual judges, as well as their pairwise matching, are used in order to adjust the task (choice of a marking scale for DEFT-07 and selection of topical categories for DEFT-08). Finally, the mutual comparison of competitors' results, at the end of the evaluation campaign, confirms the choices we made at its starting point, and provides means to redefine the task when we shall launch a future campaign based on the same topic.

Abstract A question answering system will be more convincing if it can give a user elements concerning the reliability of its propositions. In order to address this problem, we chose to take the advice of several searches. First, we search for answers in a reliable document ...