News

RSNA Announces New Machine Learning Challenge

RSNA has launched its second annual machine learning challenge. The RSNA Pneumonia Detection challenge invites teams to develop algorithms to identify and localize pneumonia in chest X-rays.

"The expectation that artificial intelligence will soon provide valuable tools for radiology continues to grow," says Luciano Prevedello, MD, MPH, chief of the division of medical imaging informatics at The Ohio State University and chair of the Machine Learning Steering Subcommittee of the RSNA Radiology Informatics Committee (RIC). "By organizing machine learning data challenges, RSNA is playing an important role in fostering and demonstrating these capabilities."

A machine learning data challenge is a competition among AI researchers to create applications that perform a defined task according to specified performance measures. The Pneumonia Detection challenge will be based on a publicly available data set published by the National Institutes of Health that has been carefully annotated by multiple expert reviewers. The RSNA Machine Learning Steering Subcommittee collaborated with volunteer specialists from the Society of Thoracic Radiology to annotate the data set, identifying abnormal areas in the lung images and assessing the probability of pneumonia.

"A successful machine learning challenge needs to begin with a data set accurate and large enough to provide 'ground truth,'" says Safwan Halabi, MD, medical director of radiology informatics at Stanford University and chair of the Machine Learning Data Standards Subcommittee of the RIC. "Developers build their applications by training them on the data set and challenge organizers use a segment of the data set to measure their performance."

This year's challenge will be conducted using a competition platform provided by Kaggle, Inc, a subsidiary of Alphabet, Inc, which is also the parent company of Google. The Kaggle platform will provide a home page for the challenge, controlled access to the challenge data sets, a discussion forum for participants, and the repository where they submit their results. Kaggle has recognized the RSNA Pneumonia Detection Challenge as a public good and will provide $30,000 in prize money for the winning entries.

The challenge will have two phases: training and evaluation. During the training phase, which started on August 27 and runs until October 17, participants will use the training portion of the data set to develop algorithms that duplicate the annotations provided by the radiologist observers. In the evaluation phase, which runs October 18–24, participants will run their algorithms on the testing portion of the data set, from which the annotations are withheld.

October 24 is the deadline for submission of test results, which will then be compared to the "ground truth" values supplied by the annotations of the expert observers, to determine the winners. The results will be announced in early November, and the top submissions will be recognized November 26 in a session at the Machine Learning Showcase during RSNA 2018 at McCormick Place in Chicago.

Last year's challenge had more than 200 participants, many of whom formed teams of data scientists and radiologists. "Radiologists have a critical role to play in the challenge, even if they don't have extensive experience with machine learning," Prevedello says. "They can provide insight, based on their clinical experience, to refine or supplement the annotations on the training data set and help developers identify systematic errors in the results of their algorithms." 

Radiologists interested in participating can set up a free account on the Kaggle site and express their interests and qualifications in the forum for the RSNA Pneumonia Detection challenge. Information is available on the challenge site: www.kaggle.com/c/rsna-pneumonia-detection-challenge or by contacting RSNA staff at informatics@rsna.org.