TRACEABILITY (Software / Firmware)

I have spent years with software V&V and documentation on US FDA submissions (IDE, 510(k) ..), remediations for other consultants, device, production and test equipment software and firmware V&V, et al, all reviewed by FDA personnel.   The key purpose of 'traceability', in the context of this discussion, is to prove that all the requirements have been addressed and proven to exist and function properly / repeatedly by means of the V&V test scripts / test cases. 

Traceability is required between the SRS (requirements) and other key documents in a SW V&V package, especially the test scripts/cases. This is always carefully reviewed by a reviewer / auditor. Whether COTS (commercial off-the-shelf) or custom SW. Traceability is required in glass box / white box reviews of code in custom SW V&V, usually defined by the developer in their test cases, but subject to review and questions by the FDA (or N-B) reviewer / auditor.  And where white box review / verification is conducted, black box also has to be conducted (to show software's proper operation in its platform / hardware / environment) in addition, which would also include additional traceability analysis / documentation.

The architecture block diagrams also provide a "traceability" function, in showing the interrelationships between the code modules, the hardware, the user interface(s), the output(s), any cloud interface, etc. 

I do traceability matrices routinely for my clients.  On some extremely complex software / firmware driven devices, the degree and depth is determined by risk to the patient / end-user. A hazard / risk analysis (device) is also an FDA requirement for software V&V (I use the ISO 14971 requirements for a Product Risk Managment File and Report to provide such analysis).  For a SW V&V documentation "model" see the FDA' guidance on 510(k) submissions for devices containing software - a useful overall SW V&V "model", which I discussed with the FDA author shortly after it was first published, and which is discussed briefly elsewhere in this blog. 

The US FDA states that SW V&V has to be a risk-based activity. The above form the basis of my many articles (peer reviewed), webinars and workshops on this subject. 

To answer this question from a different standpoint: The degree of importance placed on the traceability mentioned is not just based on patient risk. It also is based on the system's complexity, and the completeness of the other documentation in a SW V&V report, and may take on a greater or lesser role based on the depth of the other V&V activities documented. 

Please note that companies are given leeway as to approaches to take to satisfy the CGMPs and safety and efficacy issues, as long as those issues are, in fact, addressed. And acceptable solutions are not necessarily "rocket science", but just basic / methodical / appropriate testing / verification to prove all SRS (Software Requirements Specifications) have been met, including any applicable standards, guidance documents, and issues of safety / efficacy, properly documented. And yes, not everybody is doing the above as they should.

--  John E. Lincoln, jelincoln.com

Testing / V&V must not only prove requirements have been met (it does what it should do), but also that there are no "unintended consequences" (it doesn't do what it shouldn't do), a big issue with the FDA, and a major source of field problems. - JEL  05/12/24  

SOFTWARE / FIRMWARE  V&V  "MODEL"

The key document that supplies the model for all SW V&V documentation that I recommend is in Table 3 of the following:

http://google2.fda.gov/search?q=cache:Sp9J-y5akHwJ:www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm089593.pdf+guidance+on+510(k)+with+software&client=FDAgov&site=FDAgov&lr=&proxystylesheet=FDAgov&output=xml_no_dtd&ie=UTF-8&access=p&oe=UTF-8

Per that Guidance, a software / firmware V&V "package" / test report should must consist of the following 11 documents:

01.  Level of Concern;
02.  Software Description;
03.  Hazard List (I recommend the ISO 14971 Product Risk Management File / Report);
04.  Software Requirements Spec;
05.  Design Spec;
06.  Development;
07.  Architecture;
08.  V&V[T] (sic) -- the test cases, e.g., IQ, OQ (and Pt 11 if aplicable), PQs; or comparable;
09.  Traceability [Matrix], esepcially 04/05 Requirements to 08 test cases;
10.  Unresolved anomalies ("bugs");
11.  Revision(s) / Release Number.

While this is geared to devices and a 510(k) submission, I have successfully used it as a model for all regulated industries and all software / firmware V&V, in-, as-product, in test / production equipment, process, facilities, et al.  I have never had a problem in any US FDA audit, ISO audit, or vendor audit.  And it is a requirement for any device submitted to the FDA for IDE, 510(k), or PMA review.

Validation principles in general , e.g., Rqmts Specs, IQ, OQ, PQs, are readily available on the fda.gov website. 

21 CFR Part 11 (when using e-records and/or e-sigs to document CGMP actions / records:

http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/cfrsearch.cfm?cfrpart=11


The fda.gov website also has other guidance docs on SW V&V.  But the above two provide much of the "meat" for SW V&V.

You will find much contradictory "advice" on the Internet, including some that contradicts the above.  However, I emphasize that if the medical device has to undergo any US FDA review or audit, its software documentation MUST confirm to the above guidance document with its 11 documentation requirements.  And the original author of that guidance emphasized in a phone conversation with me when it first came out that it's requirements were basically taken from the non-regulated software industry itself, not created by the FDA in some 'vacuum'.

CROSS  POLLINATION


The US FDA, in its publications and/or on its website, fda.gov, provides much industry-specific material, guidance, and tools.   Why not look outside your own industry.

In my webinars and workshops, I tie my recommended "working" V&V definitions to CGMP and ISO requirements as a start.  Same applies to guidance docs and standards.  These list either legal requirements or recommendations, but require additional / working definition to make them usable, defined in SOPs ... and to be followed by the company.  In doing so, I  have not yet ever had push-back from an FDA or ISO/N-B auditor.  

I also always look at other regulated industries for the "C" [current] in CGMP.  The FDA agrees --  note their introduction to Process Validation, Jan 2011 (for pharma, BUT ...), especially the last sentence below:

"This guidance outlines the general principles and approaches that FDA considers appropriate elements of process validation for the manufacture of human and animal drug and biological products, including active pharmaceutical ingredients (APIs or drug substances), collectively referred to in this guidance as drugs or products. This guidance incorporates principles and approaches that all manufacturers can use to validate manufacturing processes."

Check the FDA's website, fda.gov, and enter validation or validation guidance or validation guidelines (also add software...)  in the upper right search box, and follow the leads, pulling out points that are of value, no matter what the industry.


In my webinars and workshops, I also mentioned QSIT (devices), ICH Q-series (pharma) and HACCP (foods) as all containing points that could be used in other regulated industries.

Be comfortable in "looking outside the  box" in adding the "C" to your "GMPs".  

-- John E. Lincoln

WHY  VALIDATE? 

The simple answers:  "It's the law." "The CGMPs require it." 

The real answer:  Yes it's the law, and the CGMPs require it.  But let's look beyond such negative motivators. 

Consider the real purpose of V&V. Not to satisfy a reg or audit, tho that's what motivates many. The real purpose is to prove that whatever you're verifying and validating (V&V) does what it's required to do, why it was acquired and installed (requirements)? That requires that before you spend time on other V&V activities, you verify (inspect, test, check, qualify) that it is installed properly, i.e., all applicable manufacturer's requirements have been met (electrical voltage, current, grounding, air exhaust, air treatment, safety, environmental prep, and so on). If this isn't done first, then subsequent V&V activities are compromised. 

If some process or equipment may have been running for X years, and will be retroactively validated, it's safe to assume that it's installed properly. However, "assume" is not in the QMS (Quality Management System) vocabulary, so even with an established system, some checks must be performed (IQ), and all QMS' require documentation of that activity. You may do so and find some problems which will have to be addressed. Then you can progress with verifying and/or optimizing the settings used (DOE may be involved), whether you call it an OQ or not. Then you challenge the system repeatedly with all the extremes of allowable inputs (material, personnel, times of day, utilities fluctuations, etc) to show (whether you call it a PQ or not) that the system being V&V is robust and will virtually always deliver expected levels of quality with known and acceptable levels of variation (see recent drug process validation guidance from FDA), which will change / improve over time / lifecycle. These should be done, as needed, all or part, over the life time of the product / process / equipment involved. 

-- John E. Lincoln, J. E. Lincoln and Associates LLC; jelincoln.com      

RETROSPECTIVE  VERIFICATION  AND  VALIDATION (V&V)

Based my 30+ years experience with the FDA, "deviations" and definitions as "legacy" won't cut it with them regarding omitting a validation. The FDA does accept retrospective V&V on processes and production / test equipment, and expects the documentation to be thorough, though much of your data can be reduced by a review of the last several lots through that process / equipment. With a retrospective validation, you do what you can, but I would certainly include an IQ, OQ, and several PQs (or DHR reviews, documented, covering varying operating conditions, shifts, operators, and RM lots) as part of that. Your procedures for such V&V should be addressed in an SOP and/or a Validation Master Plan, as well. 

Software must be fully validated, and hardware / equipment can be validated to the extent that there is no 100% downstream verification (QC / inspection) performed / documented. As part of Lifecycle considerations, you are required to re-V&V or at least re-evaluate the need for additional V&V, periodically (defined in your SOP / VMP) any way, so, if not done at all up to the Current time, you would be expected to do it starting as of the present. I've never seen the FDA give a company a "pass" on validation just because it wasn't done previously, and have lost several arguments along that vein with their auditors and the relevent FDA District Office.

However, the FDA prefers prospective V&V, because, by definition, retrospective means product was built and released previously using non-V&V'd processes / equipment, in violation of the CGMPs, and hence that product is technically "adulterated" -- challenges to such by companies have ended up in court, and no matter the outcome in the court, have been extremely costly to the company in terms of share value, lost market share, public reputation, and an adversairial relationship with the Agency, and have not changed the Agency's policies to such situations.

        

True, you can reference ASTM E2500 to justify not using the terms IQ, QO, PQ, but are still expected to "verify" (qualify, inspect, check ...) proper installation (or "IQ"), proper selection (and initialization) of settings / parameters / tolerance ranges (or "OQ"), and consistent reliability / reproducibility, et al, of your equipment / processes, under allowable "worst case" conditions (all worst shift variations, RM / lot variations, power / air fluctuations ...; or "PQs"). And as evident by some forum discussions, define your terms in your documents (validate, verify, qualify, commission, FAT, SAT, etc) with your own "working definitions" under the broader (but harder to "real world" implement definitions under the CGMPs and/or ISO.  -- John E. Lincoln, J. E. Lincoln and Associates LLC; jelincoln.com      

IMPORTANCE  OF  MULTIPLE  PQ  RUNS

Recently I had to disagree with several postings on a popular validation forum on the minimizing of the importance of several (more than 3) PQ (Performance Qualification) runs in a validation. 

The goal of a validation is to assure that the equipment and process produce acceptable product (CGMP and ISO definitions, e.g., 21 CFR 820.3z (1, 2, and3)), repeatedly, and reproducibly. The product is verified by adherence to validated process / equipment parameters and/or QC inspections (100% automated (validated) or sampled) / verifications. And to assure such, the PQ runs should incorporate all foreseable allowable variations in inputs, especially allowable worst case variations, e.g., RM (raw material), personnel, work shifts, power variations, tolerance ranges of all acceptable settings, et al. Failure to do any of this results in the many production "surprises" that appear to be behind most product recalls. So, getting back to the original question, one's production equipment / process validation should include several PQ runs with varying lots of RM -- which can be a challenge with silo / rail car / truck-trailer loads of the same lot of RM, different products expected to be run on that line, the different work shifts, and week ends / holidays, different acceptable power / compressed air loading, ..., if the line normally runs then, and similar considerations.

With large incoming RM lots, e.g., APIs (Active Pharma Ingredients) or plastic resin lots (molded plastic parts), I might choose to run three PQs at least to validate the process and that first RM bulk lot. Then release lots having that RM bulk lot could be released to the field.  Then as another bulk lot came in, I would run at least one more PQ using the new lot, to quality that FG lot using the new RM lot to be released.  I would then do one more PQ on the next bulk RM lot, to qualify it.  If there were no evidence of major variability in those three RM bulk lots, I would probably then consider the process with it's "worst case" allowable RM lots to now be fully validated, unless further QC / CAPA monitoring pointed to some additional problems.

Why a minimum of 3 lots?  The FDA has backed away from 3 as a de facto minimum for PQ runs; and with good reason.  One run proves nothing.  Two runs does not prove a pattern.  Three runs may if there's minimum variability between process inputs.  But in many cases, more that 3 may be required, as in the example of bulk RM mentioned above.  Other examples of allowable "worst case" inputs, were as alluded to in the first paragraph:  If your production runs 24/7/365+, then you might want to do 3 runs on 1st shift weekdays, to prove the process design, and add a 3rd shift, and/or a Sunday, and/or a holiday, or all, for a total of 4, 5 or 6 PQs to cover potential problems caused by reduced supervision, casual attitudes of operators, impaired behavior of operators, or similar, due to odd shifts and/or holiday work.  A further example of allowable worst case variables could be software / apps platforms, e.g., iPhones, Android phones, iPads, tablets, Macs laptops, Surface laptops, Android laptops, Mac desk computers, PCs, Unix, mainframes, servers, and so on, each with it's own PQ if required.

Another consideration alluded to above might be a company's electrical and/or compressed air variability in it's internal grid, e.g,. brownouts, especially when every one comes in to work at the same time and throws the power on to each of their workstations.  If product is produced at that time, some may be compromised, not getting the full amount of electricity or air that the validated parameter settings imply, until the grid(s) stabilize.  UPS and surge tanks may help, but these are some examples of considerations in validations I often see omitted.
  
Your company's unique situation and it's allowable varying worst case inputs would assist in making such a determination.  A cause and effect diagram can help in determining worst case allowable process inputs.  -- John E. Lincoln, J. E. Lincoln and Associates LLC; jelincoln.com  

Updated: 06/20/2022