Stepwise development of a clinical expert system in rheumatology

Summary

The evaluation of computer expert systems, a promising diagnostic tool for future application in clinical medicine, is of great importance. We present here the evaluation of our expert system, “RHEUMA”. It is stressed, that repeated retrospective testing and updating of an expert system and its subsequent repeated assessment in clinical use and surroundings is mandatory. This increases the diagnostic accuracy of the system. For our system this is demonstrated under three separate conditions. In the first study the information available for the computer system (mainframe) came from medical histories only. Here an error rate of about 25% — similar to that of physicians themselves using the same information — was observed in 358 outpatients, compared to the final diagnoses of physicians also relying solely on information from medical histories. In a second step a completely new system on a personal computer was developed with all relevant diagnostic information. The error rate of this system (0.4%) was much too optimistic because the knowledge base was changed during the study, affecting about 30% of the 282 prospectively recruited outpatients. In a third step the efficacy of the expert system was tested in an additional hospital without the diagnostic involvement of the first testing clinic. The error rate of the system without changing the knowledge base reached 11% in 51 outpatients in this rheumatology clinic. This result reflects the diagnostic accuracy of the system today. Its ability to specify the same diagnoses which clinical experts reached approached 90%. Considerable time is needed for such prospective testing, with repeated updating of the knowledge base — in our case for both the two systems and field studies of 2 years each. Further prospective field testing with physicians not specialized in rheumatology and with a larger number of patients is necessary before the system can be used in clinical routine.