AutoML-Conf 2022

1st International Conference on
Automated Machine Learning

Call for AutoML Competitons

Dates

  • February 21, 2022 – Fast-Track Proposal submission deadline (moved 2 days back since having it on a Saturday was a bug)
  • March 1, 2022 – Decision notification
  • March 15, 2022 – Tentative start of the competition
  • May 16, 2022 – Launchpad Proposal submission deadline
  • July 25-27, 2022 – Announcement of the winners at AutoML-Conf 2022

Description

This Call for Competitions is an invitation for competition proposals to be affiliated with AutoML-Conf 2022. While traditional competitions are often won by short-term engineering efforts, with the competitions affiliated with AutoML-Conf we instead aim to facilitate mid-term scientific advances and open-source methods development. To reach these goals, we have two tracks this year: 

  • The Fast-Track targets organizers who are ready to launch their competition soon (tentatively March 15th) and announce the winners at AutoML-Conf 2022. While we also accept new competitions that are ready, we specifically invite organizers of previously-held competitions to reopen their competitions again, in order to track improvements in the field and allow competitors additional time to focus on novel research to address the problems targeted by the competitions.
  • The Launchpad Track targets organizers of competitions in the early design phase. These competitions will be launched at AutoML-Conf 2022, with results presented at the following AutoML conference, AutoML-Conf 2023. We encourage these competitions to have several intermediate phases; competition organizers should also request official winners to open-source their code. In combination, this will allow all competitors to continue from the top solutions in each phase, thereby facilitating continued progress throughout the phases.

We are looking for AutoML competition proposals that meet the following requirements: a relevant challenge for AutoML research, a fair setup, a broad outreach for the AutoML community, and long-term accessibility for the AutoML research community.

  • How to ensure that the competition poses a relevant challenge for AutoML research? The competition should be in scope for AutoML (for topics in scope at AutoML-Conf 2022, please see, e.g., the call for papers). It should also be novel in the sense of focussing on an AutoML research problem that has not been addressed exhaustively by a previous competition in  past years (except in the case of reopened competitions) or re-assess the state-of-the-art for an important problem. The problem should have the potential for great impact in the AutoML research community.
  • How to ensure a fair setup? The organizers should guarantee the availability of the data and the confidentiality of the test set (to prevent information leakages at any cost). The evaluation metrics should be meaningful for the AutoML problem and comparisons of submissions should be statistically sound. A baseline should be established to show that non-trivial results can be achieved. An estimate of what constitutes a significant difference in the performance will be much appreciated. 
  • How to ensure a broad outreach for the AutoML community? The competition should be accessible to the majority of machine learning practitioners who might not have access to large computational resources. Discussions about requirements of any domain knowledge and computational resources should be included in the proposal.
  • How to ensure long-term accessibility? The competition should remain accessible, either by keeping the competition open or by open-sourcing the competition environment and facilitating local evaluations. For the Fast-Track, due to the short timeline, this point is less critical.

Competitions that are also co-located with other conferences are welcome. For the Fast-Track, due to the short timeline, we prefer reopened AutoML competitions and almost-ready competitions. 

Proposal Template

Please follow the following template for your proposal submission:

  1. Problem description. Describe the AutoML problem. Justify why this is an important problem and discuss its novelty. In particular, please elaborate on how your proposed problem is different from previous AutoML competitions in recent years (except in the case of a reopened competition). Please include a discussion of the broader impact of this problem for the AutoML research community. If possible, include a brief discussion of related research  on this problem in the proposal.
  2. Evaluation. Describe how you plan to evaluate submissions.
  3. Timeline. Please provide a timeline for the following: 
    1. Start of the competition
    2. Duration of the competition
    3. Team registration
    4. Submission
    5. Evaluation, and
    6. Notification. 

Please consider multiple phases if suitable.

  1. Awards. We encourage you to think about awards beyond monetary prizes. You should prepare official certificates.
  2. Implementation
    1. Competition platform. Which competition platform do you plan to use (e.g., CodaLab, Kaggle, or your own)? Will the competition platform be accessible for participants from all over the world?
    2. Baseline. To what extent have you explored this problem and what is the baseline solution?
    3. Submission format. What type of fact sheets and/or reports do you require the final winners to submit? Please note that in order to be declared the final winners, competitors will have to open-source their submission.
    4. Privacy Issues. Are there any privacy concerns for the released data? Where applicable, have you obtained the right to release the data for the competition from your legal counsels?
    5. Q&A issues. How do you plan to handle Q&A and possible revisions during the competition? Do you have a forum for discussion during the competition?
    6. Dissemination. How do you plan to attract participants?
  3. Host information and previous experiences. Names, affiliations, email addresses, and short biographies of the organizers should be included in the proposal. We encourage the inclusion of details on previous experience  in organizing competitions, as well as experience with the proposed competition problem.

Competition proposals should follow the proposal template and be presented in PDF format.

Contact

Competition Chair: 

  • Wei-Wei Tu, 4Paradigm Inc.

Please send your competition proposal to competition-chair-2022@automl.cc