|
The Challenge Programme was intended “to act as catalyst for both research and application development". A first call in Year 1 attracted a dozen submissions, of which six were accepted.
The First Pascal Challenge Workshop, held in Year 2 and gathering four diverse challenges on two and a half days, was an opportunity for many informal and lively discussions; it was generally considered as a highly cool and fun event (sixty participants, half being PASCAL members).
After this success, it was decided to have a quarterly call for Challenge proposals instead of a yearly one. In practice, about three calls per year were launched, each one attracting between 5 and 8 proposals.
The level of funding was very modest overall (about €10K on average), though some of the Challenges were highly visible in Machine Learning and neighbour communities, e.g., having their venue as satellite events in major NLP or Vision conferences.
In practice, three different types of Challenges were organised in Pascal:
• Applicative Challenges, which addressed either innovative or well known issues relevant to some application field. Two main applicative challenges were proposed, the Visual Object Recognition and the Textual Entailment Challenges, respectively concerned with Machine Vision and Natural Language Processing.
• Exploratory Challenges, which investigated new settings or questions which were beyond the standard Machine Learning framework. The Learning with different training and test distributions and Agnostic learning vs Prior knowledge Challenges are representative of exploratory challenges.
• Theoretical Challenges, which asked a question, though relevant to ML was too preliminary to lead to an exploratory challenge.
Some challenges scheduled for the first or second year of Pascal were significantly delayed due to one or more of the following reasons:
• The gathering and labelling of the dataset was more expensive/time consuming than expected. In applicative challenges, this difficulty was overcome through asking for additional funds. In some other cases, the challenge never went through.
|
• The challenge organisers wanted to propose “good enough" baseline procedures, which proved much harder than expected. For this reason, the review form and decision paid more attention to the existence and quality of the existing baseline procedures.
• The criteria for comparing participants were not sufficiently clear. This is the case in particular for some challenges involving the “user in the loop", thus introducing subjective aspects which proved a posteriori difficult to measure. Likewise, the importance of the evaluation criterion was emphasised in the reviewing process.
The expertise acquired by the Challenge Programme resulted in several initiatives or suggestions:
• Offer a perennial framework for getting the challenge datasets and assessing one's algorithms on the test datasets (avoiding overfitting and cheating).
• Offer a user-friendly environment for facilitating the organisation of challenges: i) getting the output proposed for every test entry; ii) ranking the contributions, including significance tests; iii) evaluating the difficulty of each test entry.
• Offering a prize and a fast track to a journal paper for the best contributions. As a conclusion, challenges proved to be an effective and fun way to guide research, contributing to the identification of the key dificulties in the field, ensuring that progress is made, and more generally, helping to build the scientific community at a fast pace.
Challenges might also present some dangers, e.g. favouring effective over deep approaches. However, challenges can also be adjusted to reach some desirable trade-off between accuracy and depth; organising several rounds around a given challenge theme proved quite effective in this respect.
Both the scale and variety of PASCAL research has made its management and support a challenging but enjoyable task for all involved. We are pleased to share our solutions to these issues. PASCAL is a distributed institute, but above all, it is an enabler. We aim to enable our members to achieve their groundbreaking research by fostering their creativity and ingenuity.
PASCAL may have come to an end in 2008, but we will continue to refine our methods in the new incarnation of this very successful network: PASCAL 2. Please join us in this exciting adventure! |