/data/universe/

The public data challenge has emerged as one response to the need for sophisticated data analysis in many sectors.  The prototypical example of these competitions is the Netflix Prize, which awarded $1M for improved predictions of user movie ratings¹.

Kaggle provides a platform for organizations to sponsor their own challenges.  The most high-profile is currently the $3M Heritage Health Prize; participants are asked to predict hospital admission rates from previous insurance claims.  Other competitions have focused on freeway travel time, the success of grant applications, and the progression of HIV infection.

One challenge caught my attention because of its astronomical content.  The Mapping Dark Matter challenge asks participants to predict the shape of simulated galaxies.  The scientific motivation for this measurement is a better understanding of dark matter.  While dark matter can’t be observed directly because does not emit light, the gravitational pull it exerts will bend light, causing small distortions in the observed shape of background galaxies.  A large statistical sampling of these distortions will constrain the dark matter distribution.

Unfortunately, the dark matter signal is much smaller than other distortions created when the data are recorded.  The Earth’s atmosphere and the telescope optics create blurring effects, and the data are pixellated when they are stored on the detector.  The Kaggle challenge (and related GREAT10 competition for astronomers) is to find an algorithm which will best infer the true ellipticity of the galaxy from the observed, noisy data.

Diagram of effects distorting galaxy shapes in observed images (courtesy GREAT competition).

I was initially skeptical that this challenge would yield improved results.  While many of the other crowdsourcing challenges were proposed by organizations unlikely to have many data science experts on staff, the professional astronomers proposing this competition are highly trained mathematically and have expert domain knowledge.  Moreover, I didn’t expect that the small prize (travel to an astronomical conference to present the results) would motivate many participants.

However, as of this writing 45 teams have joined the competition, submitting hundreds of attempts.  Already, participants have improved significantly on the benchmark results provided by the organizers, and the challenge will continue for another month.  One of the leaders is a PhD student who has employed techniques from his field of glaciology in the image analysis.  This success prompted a press release from the White House.

Why was this competition, with its small prize, successful in beating the efforts of numerate experts?  Psychological research has shown that monetary incentives are quite poor at motivating creative breakthroughs.  Instead, the potential for recognition, autonomy, and mastery are better motivators.  By satisfying these drives, crowdsourcing competitions provide means to access the cognitive surplus of skilled technical people at a cost far below the true value of their labor.  Moreover, by expanding the field of participants beyond professional astronomers, the challenge increases the cross section of the “adjacent possible,” described by Stephen Johnson as a key driver of innovation.  In this case, the familiarity of the Kaggle participants with machine learning approaches enabled a larger solution space compared to the imaging analysis approach favored by astronomers².

What lessons can we draw to encourage the success of future scientific crowdsourcing competitions?  First, the major problem to be solved was one where the scientists themselves were not experts.  In this case, the problem was algorithmic, but others could be related to computation, statistics, or engineering.  Second, the challenge was designed to allow low barriers to entry.  The scientists abstracted the problem sufficiently to remove the requirement for specialized domain knowledge.  Finally, the challenge was structured to enable clear, quantitative judgement of the success of proposed solutions.

Crowdsourcing challenges can provide new approaches to thorny technical problems in scientific research.  Funding agencies may not wish to allot grant funds for prize money, but even modest monetary rewards can elicit valuable participation.