RETINOPATHY SCREENING

Crowdsourcing, now used for everything from designing T-shirts to funding biotech start-ups, could also be a useful tool for screening diabetic retinopathy, Christopher Brady MD told a session of the annual conference of the American Society of Retina Specialists in San Diego, California.

“We know we are not screening enough patients with diabetes for diabetic retinopathy. To do better will take a significant increase in human resources. In some areas, telemedicine is going to be part of the answer. Using computers and artificial intelligence to interpret fundus photos should also help reduce the human resources burden. But there is room for other novel methods such as crowdsourcing,” said Dr Brady, of Wilmer Eye Institute, Baltimore, Maryland, US.

He was inspired in part by a Spanish/South African collaboration that used the crowdsourcing approach to screen slides for signs of the malaria plasmodium parasite. The developers created a fun game interface and encouraged the public to zap parasites by clicking on them in sample slides.

The results gathered from the game help public health officials screen for malaria. The game, which can be found at malariaspot.org, can be played on any computer, iOS or Android platform.

LABOUR INTENSIVE

“You may remember from medical school that this is an incredibly labour intensive skill, requiring processing of huge amounts of data to screen large populations. Not only did people do very well, the developers saw additional benefits as young participants were developing into citizen scientists,” Dr Brady said.

For assistance, Dr Brady turned to tech giant Amazon, which has a crowdsourcing tool known as the Mechanical Turk (www.mturk.org). It enables researchers in any field to coordinate the use of human intelligence to perform tasks that computers are currently unable to do.

Dr Brady and colleagues used this platform to test the hypothesis that members of the public could be enlisted to help screen fundus photographs.

In the first phase of the study, anonymous graders were paid $0.10US to evaluate 19 fundus photos, of which 12 were abnormal and seven were normal. Participants underwent training in plain English during which they learned the basic terminology, and how to identify signs of retinopathy.

It took 10 anonymous graders 25 seconds to grade each image, or less than two hours for the entire task. The average individual grader was correct 81.5 per cent of the time. The accuracy improved to 90 per cent when all graders results were combined. There were no false negative grades.

“I get asked why anyone would go online and take $0.10US to participate, marking images with little idea what they are. But if you multiply it out, that amounts to $14 per hour, above the minimum wage in the US, and better than other Amazon crowdsourcing tasks,” he noted.

Based on the initial study, the researchers then tweaked the system and improved the interface based in part on user feedback. They then expanded the task to have graders mark four levels of retinopathy from normal to severe, based on UK screening guidelines.

At first the graders did no better than chance, but with feedback and experience their accuracy and specificity improved considerably. As in the first study, graders tended to over call the cases, so sensitivity was 100 per cent. “We heard from people all over the world. People enjoyed the work and they liked learning about diabetes. We saw more engagement than we anticipated. With minimal training this scalable around-the-clock workforce can rapidly determine whether these images are normal or abnormal. We believe this could prove to be a useful way to screen for diabetic retinopathy.”

The reach of such an approach is global. It would be possible to recruit graders from whatever region was being studied. This would have the added advantage of keeping some money in the local economy, he added.

British researchers recently reported promising results using a very similar approach. A study sponsored by the UK Biobank Eye and Vision Consortium reported that crowdsourcing provided “an accurate, rapid and cost-effective method of retinal image analysis” (Mitry et al, PLOS1, “Crowdsourcing as a Novel Technique for Retinal Fundus Photography Classification”, DOI: 10.1371/journal.pone.0071154).

Another research group is already using crowdsourcing to create a 3D map of the neurons in the retina. Dr Sebastian Seung and colleagues at the Massachusetts Institute of Technology have created an online game called EyeWire (eyewire.org) that has already recruited more than 140,000 players from more than 140 countries.

Participants complete a one-hour tutorial, then compete with others to see who can map the most cubes per week. Players compete for personal glory and unique prizes such as the right to name new neuron discoveries.

The neuroscientists benefit from the game players’ enthusiasm as these efforts help develop advanced artificial intelligence tools and computational technologies for mapping the connectome, the structural map of all neural connections.

* Dr Brady’s study was recently published at: Brady CJ et al, “Rapid grading of fundus photographs for diabetic retinopathy using crowdsourcing”, J Med Internet Res. 2014 Oct 30;16(10):e233. doi: 10.2196/jmir.3807. PMID: 25356929

Christopher J Brady:
christopherjbrady@gmail.com