What is Computer Software Assurance (CSA) and why are the FDA transitioning from traditional Computer System Validation?

Following the launch of their ‘Case for Quality‘ initiative in 2011, the FDA were uncertain why so few companies were investing in automated solutions and why so many continued to run long-outdated versions of software.  The initiative, which set out to study quality best practice in medical device manufacturing, found that the burden of Computer Systems Validation (CSV) deterred technology investments and as a result, inhibited quality best-practice (1).

On learning that the burden of CSV was holding companies back from realising their investment in technology, the FDA decided to partner with industry to strike a balance between promoting automation and value-add CSV activities.  The FDA aimed to improve quality, remove non-value add activities, and focus testing on high risk areas, therefore reducing validation cost and time by focusing on the software’s impact to patient safety, impact to product quality, and impact to quality system integrity (Direct or Indirect system).  Too often, testers spend time ensuring their protocol is error free, as opposed to spending time on automated solutions that verify the software meets it’s intended use.

The FDA’s new approach to CSV, Computer Software Assurance (CSA), represents a step-change in computer system validation, placing critical thinking at the centre of the CSV process, as opposed to a traditional almost one size fits all approach.

As its regulatory approach continues to mature, the FDA intends to focus on direct impact systems and not on indirect systems. The change allows manufacturers to focus testing rigor on areas that directly impact patient safety and device quality, as:

• ‘Direct system software’ (e.g. inspects or dispositions product, labeling systems) will require testing based on risk, and expected deliverables are similar to current expectations, i.e. the riskier the application, the more testing and documentation is required.
• ‘Indirect system‘ is software that does not directly impact the product or patient safety but does impact the quality system (e.g. Document Control, Complaint Management, Lifecycle Management tools). The same rigor is not needed for the assurance of these types of systems and they require less documentation.

The current approach to Computer System Validation (CSV)

CSV has morphed into an activity that is being done primarily to secure evidence for auditors, rather than to assure the quality of systems being validated. The current CSV approach is seen as:

• a deterrent to pursuing automation time, cost, use of automated testing tools, documentation generation etc
• regulatory burden (e.g., Data integrity) as an excuse for resisting progress; meanwhile, other regulated and nonregulated industries have moved forward and adopted frameworks for modern testing.
• all software is being validated as if it is product software.
• burdensome and the creation of complex Risk Assessments.
• focusing on gathering evidence for auditors.
• a duplication of vendor efforts at client site. 
• 80% of deviations due to tester or test script errors.
• Numerous post-go-live issues.

From CSV to CSA

Computer System Validation

• A focus on creating documentary records for compliance
• “Validate” everything (and miss higher risk areas)
• Ignoring previous assurance activity or related risk controls

Computer System Assurance

• A Focus on testing for higher confidence in system performance.
• Risk based “Assurance”, applying the right level of rigor for a given level of risk to patient safety and/or product quality.
• “Take credit” for prior assurance activity and upstream/downstream risk controls.
• Focus on testing, not scripting. Use unscripted testing for low / medium risk components.

The right level of documentation is; however, always remember:

•  “If it’s not documented, it didn’t happen.”
•  “Unscripted Testing ≠ No Documentation”
•  “Traceability is still required”

Scripted vs Unscripted Testing: what’s the difference?

Traditionally each test script was written in great detail, regardless of whether the system or feature was a Direct or Indirect system or feature. So, the same level of effort was being put into creating test documentation for low risk systems or features and high-risk systems or features. CSA introduces the terms Scripted Testing and Unscripted testing.

Let’s examine these terms: 

• ‘Scripted Testing’ is what we would know as traditional testing. Scripted tests as we know usually contain at a minimum a test Objective for the test script, a step-by-step test procedure, Expected Results and a Pass/Fail. Scripted Testing is to be used to test to higher risk (Direct) systems or features as the software does directly impact the product or patient safety.
• ‘Unscripted testing’ is testing that is carried out without the use of detailed test scripts. Unscripted Testing is to be used to test lower risk (Indirect) systems or features as the software does not directly impact the product or patient safety but does impact the quality system. There should be a Test Objective and a Pass/Fail, but no step-by-step test procedure.

Details regarding any failures or deviations and disposition regarding fails found while performing Scripted and Unscripted Testing will still need to be recorded to ensure failures or deviations are documented from the discovery of the failure or deviation to the successful implementation of the appropriate corrective action.

Computer Software Assurance Benefits

• A reduction in cycle times cycle times (test creation, review and approval)
• A system can be broken into features, and only the High-Risk features will require scripted testing.
• Reduced test script execution time
• Lower number of detected defects for example script errors & configuration
• A reduction in the number of generated documents for example the use of the combining deliverables and test scripts)
• Testing focused on ensuring SW Quality
• Better use of Supplier Qualification
• Maximized use of CSV and Project Resources expertise (e.g., SMEs)
• The release of the CSA guidelines will support companies who have taken the path to automation.

Transitioning from CSV to CSA

• Change the culture from of your organisation from a compliance-centric mindset to quality-focused culture.
• Leverage your software suppliers existing activities (perform supplier audits)
• Consider using computer system validation tools to automate assurance activities
• Know the intended use of your computer system(s).
• Know the high-risk features, operations and functions of the computer system(s).
• Review and update your current policies to align with the CSA approach.

Computer Software Assurance: 6 things you need to know

The FDA is expected to release it’s new guidance around CSA, ‘Computer Software Assurance for Manufacturing, Operations and Quality Systems Software’ before the end of 2020.

About the Author

Darren Geaney is a Process Engineer with over 20 years’
experience in Quality Assurance, specializing in Computer
System Validation. 

References

1. ‘Understanding Barriers to Medical Device Quality’, US Food & Drug Administration,[https://www.fda.gov/media/82284/download]