Today it is possible to crowdsource a solution to just about anything: A new “Instagram for doctors” is enabling them to reach better diagnoses with help from their peers; governments across the world have asked members of the public to submit their ideas for government spending cuts or new ideas to solve problems across government; and almost any budding entrepreneur can appeal to the masses to crowd-fund their latest project. And many current media and research projects continue to harness information and participation from mass audiences.
While crowdsourcing may have risen in popularity in recent times, the science behind the wisdom — and power — of the crowd can be traced back over to ideas outlined a century ago. In 1907 scientist Francis Galton observed the results of a competition at a village fair to guess the weight of an ox. Guesses varied wildly amongst the 800 people that participated, but the average came within a single pound of the 1,198-pound ox. In his 2005 book The Wisdom of Crowds, journalist James Surowiecki provides many other examples of when the collective judgment of the many is wiser than even the expert judgment of the few, from stock markets to Google and Wikipedia to betting markets. However, Surowiecki also points out that the wisdom of the crowd is far from infallible. A key requirement for good judgment by a crowd is that each individual’s decision is independent from others.
Contemporary crowdsourcing practitioners would likely benefit from increased understanding of social science findings in this area and their implications, in order to help control for sources of error or bias. This is particularly necessary where strong inferences or conclusions are being derived and the need for scientific-level accuracy is heightened. Two studies published in the Proceedings of the National Academy of Sciences (PNAS) provide valuable perspective about two key issues: The typical distribution of participation on projects; and the way social influence can confound results.
Motivated participants and short attention spans
In a 2015 paper, “Crowd Science: The Organization of Scientific Research in Open Collaborative Projects,” Henry Sauermann of the Georgia Institute of Technology and Chiara Franzoni of the Polytechnic University of Milan investigate patterns relating to the increasing number of scientific projects that are using members of the public as unpaid volunteers or “citizen scientists” to perform tasks such as data collection, transcription or image coding. In fact, through their research Sauermann and Franzoni found more than 100,000 citizen scientists contributed over 129,000 hours of unpaid time, with an estimated value of over $1.5 million for the first six months of just seven projects studied. They analyze more than 12 million daily observations of users who participate in the seven projects hosted at the citizen science site Zooniverse.org.
The study’s findings include:
- Projects can attract “considerable volunteer effort [that] would be difficult and costly to procure through traditional or online labor markets,” the researchers note. “Using two different approaches, we estimate the average value of contributions received per project at more than $200,000 over the first 180 [days].”
- The top 10% of participants were responsible for almost 80% of the work on average — with the rate as high as 88%. “Although crowd-based projects generally appear to be characterized by strong inequality in individuals’ contributions, future work is needed to understand why the distributions of contributions are more skewed in some projects than in others.”
- Much of the participation is one-time only and involves “little effort”: “First, most users do not return to a project for a second time, with the share of those who return ranging from 17% to 40% (average, 27%)…. Second, when averaging the daily time spent across all active days for a given contributor, the mean ranges from 7.18 to 26.23 minutes across projects, indicating that visits tend to be quite short.”
The researchers conclude that “distributing work to the crowd is easier if projects can be decomposed into many small independent tasks such as the coding of individual pieces of data.… In contrast, complex tasks … require more coordination among individuals, likely limiting the number of participants who can get involved efficiently.” Further, the “high variability in project contributions over time suggests that crowd science may be more appropriate for tasks where the exact timing of effort is less important. For example, it is likely easier to ask the crowd to inspect a certain amount of archival data than to ask it to continuously monitor a stream of real-time data. Finally, challenges may arise for projects that require sustained participation by the same individuals.”
Social cues and skewed results
A 2011 PNAS study, “How Social Influence Can Undermine the Wisdom of the Crowd Effect,” analyzed experiments where participants were asked simple questions about geographical facts and crime statistics (about which they were unlikely to know the answer). Questions included the population density in Switzerland (where the study took place) and the country’s official murder rate in 2006. Participants were split into treatment and control groups, with those in the treatment groups receiving information on how others in the study had answered each question. Those in the control group received no such information. Participants (144 in total) were offered a financial incentive the closer to the correct answer they got, minimizing any incentive to change their response to one they thought was less accurate due to social pressure.
The study, by researchers Jan Lorenz, Heiko Rauhut, Frank Schweitzer and Dirk Helbing, found that knowing the answers of others influenced participants in three main ways:
- The diversity of guesses was diminished when participants could see the answers of others, without any improvement in their accuracy. The researchers termed this the “social influence effect.”
- Responses in the treatment group could gravitate toward the wrong answer where initial responses failed to capture the correct answer at the center of the different estimates. This was labeled the “range reduction effect.” The researchers compared this effect to someone in government, for example, seeking the advice of a group of experts. If all the expert predictions converged toward the wrong value, the person seeking advice would gain confidence in answers that were actually misleading.
- The combined psychological effect of the two trends above produced a “confidence effect.” Individuals were asked to self-report how confident they were about the accuracy of their answers. Those who received information about the answers of others reported a greater level of confidence in their responses, despite no improvement in their collective accuracy.
The authors conclude that the “results underpin the value of collecting individuals’ estimates in the absence of social influence. However, in democratic societies, it is difficult to accomplish such a collection of independent estimates, because the loss of diversity in estimates appears to be a necessary byproduct of transparent decision-making processes. For example, opinion polls and the mass media largely promote information feedback and therefore trigger convergence of how we judge the facts. The ‘wisdom of crowd’ effect is valuable for society, but using it multiple times creates collective overconfidence in possibly false beliefs.”
Keywords: Crowdsourcing, science, psychology, research, social influence
Expert Commentary