Engineers, AI experts, and researchers in several countries are urgently working to develop an application to rapidly and accurately assess whether the speaker exhibits reliable indicators of Covid-19. They plan to utilize the advanced audio capture capabilities and processing power of a smartphone to analyze audio from coughs and spoken test phrases to match individual micro-characteristics against a large database they are attempting to gather and build. A few weeks ago, a team of scientists and engineers from Carnegie Mellon University in Pittsburgh posted (and then were ordered to take it down pending FDA and CDC approval) a trial Application in an attempt to gather information to analyze voice characteristics from a large sample group including those that are infected with Covid-19. The CMU team has expertise in AI-driven voice forensic analysis and technologies and is profiling people from their voices. AI allows the sophisticated extraction of micro-signatures that can correlate to certain human parameters that may be relevant to Covid-19 and other diseases.
Thousands or even millions of people in virtually every country do not have easy access to testing for the virus, thus making it even more urgent to develop a voice-based application that could be administered easily, without cost, in any location. To complicate the issue, the blood tests for Covid-19 that are presently available are not entirely accurate, some with significant error rates in both false positives and false negatives. The research premise and idea for the use of artificial intelligence to detect disease is simple: determine whether the speaker has voice-dependent spectrographic criteria that result from changes in speech patterns that correlate with lung function and other respiratory parameters. Course or micro-changes occur when we breathe or cough. While many of the signatures detected and processed by AI algorithms can be very difficult to hear, they can nonetheless be identified and analyzed.
CMU is a global leader in robotics, AI, and software tools including the analysis of sound to detect abnormalities associated with certain physical conditions. In posting the App, they wanted to gather a large number of samples from healthy and sick people that not only had the virus but for other diseases as well, and then share the samples with trusted researchers worldwide. Before I was able to test the App, I contacted and interviewed Dr. Rita Singh at CMU. She is the team leader for faculty and student researchers, and has five years of experience in the development of AI algorithms and software to analyze characteristics of the human voice. She is an Associate Professor in the School of Computer Science Language Technology Institute and has extensive experience in developing complex algorithms that intersect the science of Artificial Intelligence and waveform analysis. She believes an application can ultimately result in the early and rapid detection of significant indicators for medical professionals for the assessment of health status.
Dr. Singh has been advised by CMU to arrange for clinical evaluation of the system and it can be provided to any entity who would like to use it, under an “evaluation license.” According to Dr. Singh, “once we have medical and other carefully curated evaluations conducted, it will go on for FDA approval. Once approved, it will be online.” Here is the application that was developed by Dr. Singh and her research team. It will allow the vital collection of data but prevents any results from being returned to the user. Nonetheless, the data collected can be of real value to the CMU researchers. The CMU application was straightforward and mimics others that are presently available at the University of Cambridge and in Israel. Whether it will ever receive FDA approval is uncertain given their current priorities. The question is whether such an application will work, can be verified in field trials, and would be accepted by the medical community.
I contacted Dr. Eric Hoffman at the University of Iowa Hospitals for his view on these AI applications. He is the Director, Advanced Pulmonary Physiomic Imaging Laboratory, and a Professor of Radiology – Division of Physiologic Imaging, as well as a Professor of Internal Medicine and Professor of Biomedical Engineering (BME). He noted that “I could believe it might be used to detect a pulmonary infection, but it would be hard to specifically analyze a cough. They should have CT images in order to see how extensive lung abnormalities may be present. It is very interesting research, but they just need to explore the parameters that limit its specificity. “ He further added that “I could believe they could detect inflammation. They need to do more work to say it is specific to Covid-19”.
I reached out to a colleague at the University of Cambridge Computer Security Lab, who put me in contact with Professor Cecilia Mascolo who is the Co-Director of the Centre for Mobile, Wearable Systems and Augmented Intelligence, at Cambridge. They have also developed a mobile application to gather information similar to what CMU has done. Her team also believes that audio can be used as a diagnostic tool for certain diseases including those that may be cardio, digestive and respiratory related. She reiterated that the Cambridge application is for pure data collection only, and that no indications of results are available. About 3000 people in a crowd source have submitted samples.
Dr. Singh and her team are collaborating with Dr. Shmuel Ur in Israel, who is a computer scientist with a Ph. D. from Carnegie Mellon. He has worked at IBM where his title was a Master Inventor and was granted 130 patents. I interviewed Dr. Ur, who was quite confident that researchers can develop a smartphone application to analyze different coughs to help detect Covid-19. He noted that because blood tests are not completely accurate, that voice analysis may add a vital analytical component. “A cough may provide evidence before a change in a patient’s temperature, which would provide a good screening tool together with a thermometer. It is clear that coughing spreads the virus, so early detection is vital.” He also has a website for collecting data. Dr. Ur believes that critical data may even be gathered over the telephone.
Dr. Singh provided a high resolution spectrograph of sample coughs for Covid-19 positive and negative individuals. The difference can clearly be seen, although subtle. It demonstrates how artificial intelligence is and will change diagnostic procedures in medicine, which were never possible before.
The Spectrographic (time-frequency) characteristics of cough sequences associated with different health conditions, which an AI program can evaluate. The horizontal axis represents time.
A: An (induced) cough sequence of a healthy person. Three cough instances are shown. In a healthy person, each cough often ends with some vocal fold activity, seen in the regions marked in yellow circles.
B: The characteristic cough sequence associated with allergy symptoms. The four coughs successively decrease in energy until they die down.
C: The characteristic cough sequence associated with a wet cough-turbulence-induced vocal fold activity occurs randomly, energy is spread over a wider frequency band. More energy in the cough sounds, and their longer spread over time, leads to a sharp characteristic intake of breath after each episode (marked with the green circle).
D: The characteristic cough sequence observed in the majority of Covid-19 patients who have respiratory symptoms: a prolonged, diffuse energy spread across frequencies preceded by a short catch (marked in red). Low frequencies are greatly affected, and the vocal folds show unusual oscillatory activity (pointed to by the red arrow), presumably caused by altered aerodynamics across the glottis due to respiratory distress. The frequency axis in each case ranges from 0-4000 Hertz.
Artificial Intelligence will play a significant role in medicine and can offer the capability of advanced diagnostics. It is clear that many applications will be run by patients with results that are uploaded to their doctors from smart devices, such as the Apple Watch and its ECG application. The world is looking for solutions to the Covid-19 virus pandemic in terms of rapid diagnostics and treatment. It would be a needed step forward if researchers in multiple venues can contribute to valid self-assessments, screening tools, and its early identification. The only way for that to happen is developers to build a large database in order to guarantee specificity in a correlation with patient input and results.