To test the system, Watson was first tasked with answering questions taken from Doctor’s Dilemma, a competition for trainee doctors that takes place at the annual meeting of the American College of Physicians. Watson was given 188 questions that it had not seen before and achieved around 50 per cent accuracy – not bad for an early test, but hardly ideal.
To improve, Watson is now absorbing records – tens of thousands at Sloan-Kettering alone – of treatments and outcomes associated with individual patients. Given data on a new patient, Watson looks for information on those with similar symptoms, as well as the treatments that have been the most successful. The idea is it will give doctors a range of possible diagnoses and treatment options, each with an associated level of confidence. The result will be a system that its creators say can suggest nuanced treatment plans that take into account factors like drug interactions and a patient’s medical history.
Watson is answering basic questions based on the treatment guidelines that are published by medical societies and is showing “very positive” results, he adds.
The technology is particularly useful in oncology because doctors struggle to keep up with the explosion of genomic and molecular data generated about each cancer type. This means it can take years for findings to translate into medical practice. By contrast, Watson can absorb new results and relay them to doctors quickly, together with an estimate of their potential usefulness. “Watson really has great potential,” says Audeh. “Cancer needs it most because it’s becoming so complicated so quickly.”
The IBM system could also approve treatment requests more quickly. At WellPoint, one of the largest insurers in the US, nurses use guidelines and patient history to determine if a request is in line with company policy. Nurses are now training Watson by feeding it test requests and observing the answers. Progress is good and the system could be deployed next year, says WellPoint’s Cindy Wakefield. “Now it can take up to a couple of days,” she says. “We hope Watson can return the accurate recommendation in a matter of minutes.”