Manufacturers in many industries reduce development times by using COTS (commercial-off-the-shelf) software and hardware in their products, but medical device manufacturers may be reluctant to follow suit due to concerns that COTS implies SOUP (software of uncertain provenance) and thus may compromise device safety and premarket approval by the FDA and other regulatory agencies.
COTS and IEC 62304
IEC 62304 is becoming the de facto global standard for medical device software life cycle processes. The FDA has driven its development, and it is being harmonized with EU standard 93/42 EWG (MDD). Unlike IEC 61508 and EN 50128, IEC 62304 doesn’t address functional safety and therefore does not provide common numerical values for acceptable failure rates; conformity to IEC 62304 doesn’t imply a SIL rating as does, for instance, conformity to IEC 61508, which is meaningless without one.
Of particular importance in the context of this discussion, IEC 62304 assumes that SOUP will be used somewhere in the product. Section 5.1.1 "Software development plans," for instance, states that "The plan shall address … software configuration and change management, including SOUP CONFIGURATION ITEMS,” and SOUP is the explicit subject of sections such as 5.3.4 “Specify SYSTEM hardware and software required by SOUP item."
The question is not whether we may use COTS software and SOUP in medical device software, but a) how to decide whether a particular COTS software or SOUP item is appropriate for our medical device, and b) how to validate that this item supports our safety requirements for this medical device.
COTS and SOUP
Some software vendors make a rather simplistic distinction between COTS and SOUP. COTS, they say, has a company that has staked its reputation—and its financial future—on this software functioning as specified, while SOUP has no one standing behind it.
A more useful distinction is between (opaque) SOUP and clear SOUP, a distinction based on the artifacts available to support a safety case for the software, artifacts we need to support our claims about the risks and safety levels of the systems we build with the SOUP.
A COTS system may have a well-documented development process to which its vendor adheres, including tracking and documenting the software's failure history. However, if this information is not available for public scrutiny, we would consider it (opaque) SOUP.
In contrast, open source projects such as Apache and Linux have their source code and fault histories freely available, and after years of active service, this software's characteristics are well known. This software can be scrutinized with code symbolic execution and path coverage analysis, and the software's freely available history make findings from statistical analysis particularly relevant. Hence, we can consider this software clear SOUP; that is, SOUP that we can examine, verify, and validate as though we had written it ourselves.
Open source software may not be the best solution for medical systems, however, because the processes for open source development are neither clearly defined nor well documented—and this is precisely what concerns IEC 62304. Add to this that open source software may include more functionality than is needed, leaving dead code in the system, a practice that functional safety standards expressly discourage.
The best choice, therefore, may be COTS software from a vendor that makes available its product’s source code and fault history: COTS with clear SOUP. Better yet, some vendors provide a clear recipe for the SOUP: their detailed processes and complete development histories—an informal audit trail that helps substantiate claims about reliability and availability, and even the evidence they presented in order to obtain a certification (e.g. IEC 61508 SIL3) for their product.
The COTS software recipe for clear SOUP can prove invaluable for validation and approval following product upgrades. A large number of software defects are introduced after the product has gone to market, and our responsibilities end only when our products are no longer in use.
Page 2 of 2
Evaluating COTS software
At the highest level, the questions we must ask of any COTS software considered for a safety-related medical device is “What evidence does the software vendor provide that his product is what we need?” and "Will this COTS software support us in getting approval for our medical device?" To this end, we must examine:
- Safety claims—what are the vendor's claims, including context and whether the vendor quantifies his claims about availability and reliability?
- Process—does the vendor use a defined and documented process covering the entire software lifecycle from design to maintenance (ISO 9000, ISO 15504, and CMMI- Capability Maturity Model Intregration)?
- Fault-tree analysis—was the COTS software evaluated with fault-tree analysis, using a method such as Bayesian belief networks, and are the results of this analysis available?
- Static analysis—does the vendor use static analysis (syntax checking, fault probability analysis, and correctness proofs) to identify potential problems in his product, and what artifacts does it provide to support its findings?
- Proven-in-use data—can the vendor provide proven-in-use data; how far back do the data go; what is the sample size; and does the vendor provide fault analysis results with the proven-in-use data?
- Design artifacts—does the vendor provide architectural and detailed design documents, test plans and results, and a traceability matrix?
- Safety manual—does the vendor provide a safety manual, which states the safety claims for the COTS software, and does it define the context and constraints for these claims?
Erring on the side of caution is best when considering COTS software for medical devices, but it should be remembered that neither IEC 62304, nor the demands of safety, preclude its use.
If the critical distinction is made between opaque SOUP (which should be avoided) and clear SOUP is made—that is, SOUP for which source code, fault histories, and long in-use histories are available, we will find that COTS software may be the optimal choice for many safety-related medical devices.