Thank goodness for diligent regulators. Read on, and be horrified.

The US Food and Drug Administration (FDA) recently inspected a testing facility outside Los Angeles and found serious concerns about the quality and integrity of the data produced there. In its warning letter, which is publicly posted online, the agency’s inspectors outlined the objectionable conditions they found.

Anyone interested in the role of this powerful agency in protecting the American public health should read this letter. Prepare for a glimpse into the “wild west” of preclinical research. Prepare to be appalled!

Background

Drugs and medical devices undergo extensive testing before touching human beings in clinical trials. These studies are called non-clinical (or preclinical) research. In the United States these studies are regulated under the Good Laboratory Practice regulations. As part of its Bioresearch Monitoring Program, the FDA inspects testing facilities to ensure the data generated is scientifically accurate and valid.

These inspections can result in a range of actions. For a well-run facility, management actually benefits from such an inspection. They get feedback on weak areas and they better understand the agency’s thinking on certain scientific or organizational issues they face.

For a poorly run facility, the inspection could result in a warning letter at best or an immediate shutdown of the facility, followed by legal prosecution, at worst.

This letter indicates the facility is poorly run. Appallingly so. As such, all the scientific data it has generated on the investigational drug is in question.

Some choice passages from this warning letter

There were no records indicating the presence of a functional QAU, or records of any QA activities at the facility. In addition, you indicated during the FDA inspection that you were the study director as well as the head of the QAU. However, this practice undermines the QAU from performing its required functions separate from and independent of the personnel engaged in the conduct of nonclinical laboratory studies. You also indicated that a veterinarian for the study from (b)(4), was a member of the QAU; however, that veterinarian signed a statement denying that he was a QAU member.

It’s going to be hard to walk that one back!

Considering your test facility management’s overall lack of responsibility to implement basic essential elements of GLP compliance, the quality and integrity of the study data cannot be assured because there was no QAU oversight of the nonclinical laboratory studies conducted at your testing facility.

This is pretty bad. The QAU must monitor the conduct and reporting of the organization’s studies. When it comes to life-impacting drugs and devices, the integrity and acumen of the researcher alone is not enough.

Your testing facility failed to establish standard operating procedures (SOPs) in writing setting forth nonclinical laboratory study methods that are adequate to insure [sic] the quality and integrity of the data generated in the course of the studies, as well as to ensure appropriate handling and care of animals.

During the inspection, you indicated that as a surgeon, you knew all operation procedures, and therefore you did not agree that SOPs needed to be established.

Written SOPs, with appropriate approvals and control, are a bedrock of quality testing. It’s hard to overstate this.

For a reader who does not have a scientific background, let me make an analogy: Have you ever been in a court of law? What if you were on trial for a capital crime, and your lawyer showed up with no computer and no documents. You ask him, “What’s up, did you not prepare at all?” In response, he taps his temple and grins, saying, “Don’t worry, it’s all up here!”

I am impressed with the incredible capabilities of the scientists that I work with every day. But they and I know the importance of independent checks. The surgeon the FDA interviewed is saying his authority and expertise puts him above independent checks. He is saying we should trust his data because of his skill and expertise. Sorry, but this is not a medieval guild. Mastery is not a validated method, and modern drugs and devices require independent examination and comparison with a standard.

As the study director, you must ensure that all experimental data, including observations of unanticipated responses of the test system, are accurately recorded and verified.

You failed to ensure that all experimental data were accurately recorded and verified.

For the toxicity study in dogs, dose calculation worksheets for nine dogs show body weight measurements and test article administration in June and July 2016; however, purchase records indicate that eight of the nine dogs were purchased in August 2016, which makes the data entered before August 2016 invalid. In fact, during the FDA inspection, you acknowledged that the data entered before the dosing in August was inaccurate and did not reflect real-time data entry.

For all data changes, you failed to ensure: (1) that the original entries were not obscured; (2) that the reason for all changes was indicated; and (3) that all changes were dated and signed or identified at the time of the changes.

The study director’s role in a preclinical study is extremely important. He or she serves as the single point of control for the study. The passages above speak for themselves. They may even point to falsified data, though this is not explicitly suggested in the warning letter.

Basic data integrity principles and methods are taught to undergraduate science students within the first week of their laboratory courses. These include correcting handwritten entries without obscuring the original entry, signing and dating, documenting concurrently, et cetera. The warning letter shows these principles were not followed at this facility.

Because you failed to ensure that all experimental data were accurately recorded and verified, and that all entries and changes in entries were properly documented, FDA has concerns about the integrity of the data generated from the nonclinical toxicity studies conducted at your testing facility.

The investigators use professional-sounding understatement.

You failed to comply with the above requirements to retain and to provide archives for storage and retrieval of all raw data and specimens.

You were unable to provide all data requested during the inspection; you indicated that you were unsure where the data was, and that it might be at your home.

The absence of raw data and specimens collected during these nonclinical toxicity studies, and the absence of proper storage of raw data and specimens, raises significant concerns about the integrity of the study records and data.

Again, understatement. Archival procedures, controlled storage conditions, data completeness, and traceability are a bedrock of quality. These procedures would be in the SOPs, but as noted above, the facility’s SOPs were inadequate or absent.

Testing to determine the stability of the test and control articles in the mixture must be conducted, either before the study initiation or concomitantly, according to written SOPs that provide for periodic analysis of the test and control articles in the mixture. However, none of these analyses were done.

Does this sound a bit esoteric to you? It’s not. Before testing a compound in preclinical toxicity trials, the compound must be well-characterized. This ensures that the compound that eventually is dosed in humans matches the compound that was given to animals or to in vitro test systems. Without this characterization, you are putting humans in those safety trials at risk, or generating poor-quality data.

The protocols of nonclinical toxicity studies in mice, hamsters, rabbits, and dogs… did not have a date of approval by the sponsor and did not contain sufficient details, including but not limited to:

a. A statement of the purpose of the specific study

b. The name and address of the testing facility at which the study is being conducted

c. The procedure for identification of the test system

d. The methods for the control of bias

e. The frequency of tests, analyses, and measurements to be made

f. The records to be maintained

The study protocol is a central document. To name just two possible consequences of not having one: How do you identify a deviation from the study plan, if no plan was documented? How do you evaluate the outcome, if no predetermined endpoints were established? Not having an adequate protocol leads to weak scientific data and conclusions.

As evidenced by FDA’s inspection findings, your testing facility management failed to fulfill the primary responsibilities to establish appropriate policies and procedures intended to ensure the quality and integrity of nonclinical data for FDA submission. Furthermore, the deficiencies found in your oversight as the study director and the absence of an independent QAU indicate that your testing facility failed to fulfill the core responsibilities to remain GLP-compliant. As a result, FDA is concerned about the validity of nonclinical data generated by your testing facility.

Failure to address the violations noted above adequately and promptly may result in regulatory action without further notice.

More understatement. I haven’t checked, but this facility may already have been shut down! The FDA has the power to walk in and immediately halt operations. Especially since this is not their first warning letter.

Some caveats

I don’t wish to pile on. I don’t want to be part of an Internet mob. And I always assume good intent instead of nefarious fabrication and fraud. But people should be aware of the risks to health out there. The FDA cannot inspect every facility all the time. They cannot catch all the shoddy science out there. They can’t prevent people from wasting money on ineffective treatments.

I also feel sorry for the animals in these studies, which include mice, hamsters, rabbits and dogs. The letter indicates there were no SOPs describing their care and handling. What if some of them suffered from poor treatment? What if their treatment was fine, but the data from their studies was completely invalidated? It’s as if they underwent testing for nothing. If an animal was used in research, something good should come out of it. This may not have happened here.

I will write an article about how to use FDA warning letters and other materials to be proactive in your organization

In a future article I will show you how to use the FDA’s warning letter database to your advantage. The point is not just to gawk at the egregious ones. You can use the warning letters to get an edge and understand the agency’s thinking on many biotech and pharmaceutical subjects.

What do I mean by “getting an edge?” Let me explain. Have you noticed how regulations change very slowly? The GLP regulations that we follow were finalized in the 1970s and 1980s! There is nothing in those regulations mentioning cloud-based data capture, gene editing, or artificial organs. The FDA occasionally releases non-binding guidance documents that help industry, with its cutting-edge techniques, from falling afoul of these older regulations. But even those guidelines quickly become dated and cumbersome.

FDA warning letters, on the other hand, reveal findings and enforcement actions from inspections that occurred only weeks or months previously. The letters also indicate the reasoning behind the actions. A useful FDA inspection finding includes three things: a detailed description of the circumstance, the applicable standard (usually a regulation), and the possible consequences of not meeting the standard.

Read closely and you will find all three components in the letter I linked above. In my upcoming article I will break down the warning letters in a way that will make you a better auditor who will be even more valued by your organization or auditee.

Final comments

Visit the FDA warning letter database regularly. Stay tuned to warning letter news. Read my upcoming article on warning letters. Let me know if this article benefited you. And let me know if your jaw dropped as much as mine did!

Image credit

The image is from http://www.cancertreatmentus.org/

This past Saturday I sat for the Certified Quality Improvement Associate (CQIA) exam. I passed and now I can draw on this body of knowledge and cite this certification for the rest of my career in quality.

In this article I will explain what this certification is. I will describe my experience in preparing for and taking the exam. I will share what I have learned about certification pathways in quality. Finally I will offer some advice on whether you should pursue the certification yourself.

The Certified Quality Improvement Associate

The American Society for Quality (ASQ) introduced the CQIA certification in 2000 as a way to introduce the basics of quality to individuals and organizations that are not already in traditional quality roles. Getting such employees certified is a way to integrate quality throughout the company. Those already in a quality role can obtain the certification as a first step toward more intensive certifications such as Certified Quality Engineer.

The body of knowledge covered in the materials and coursework touches all aspects of quality. Some examples of learning objectives follow:

“Define quality and use this term correctly in various circumstances.”

“Define and distinguish between common and special cause variation in relation to quality measures.”

“Identify supplier performance measures, including quality, price, delivery and level of service.”

Succeeding on the exam requires remembering, understanding, and applying your learning from the materials I reference below. The exam reflects the body of knowledge presented in the CQIA Primer. Everything you need to know is in this textbook. There are additional materials that will enhance your learning and give you an edge. Actively participating in a prep course, doing as many practice questions as possible, and connecting with your instructors will also boost your chances and make your investment of time worthwhile.

Preparing for the exam

To prepare, I attended four three-hour courses presented by subject matter experts in quality. Each of the instructors were Certified Quality Engineers. At each session we:

• Did a practice quiz
• Went over two or three chapters from the CQIA Primer
• Did practice exam questions as a class
• Addressed any questions from our readings and end-of-chapter exams

You could prepare for the exam without this kind of course but that would mean doing a lot more solo work. In the class was at least one person who had failed the first time around. So, do whatever you can to get an edge!

Outside of class I read the Primer, took the end-of-chapter exams, and drilled myself again and again with the practice exam software.

It’s this last part that was especially valuable. The software is simple but has two particularly useful functions: it allows you to select the length of your exam and it breaks down your scores by section. This means you can practice for the exact length of the CQIA exam (110 questions in 3.5 hours). Once you are done you can see what areas you are weak in and revisit the Primer.

I read the Primer, highlighted key definitions, charts, figures and tables, and went over them again and again. I did not read the full Certified Quality Improvement Associate Handbook, but I visited it for some longer exposition of a couple of topics from the Primer. It also came in handy during the exam (more on that later).

During my many practice exam drills, I kept aiming for the 80% correct rate needed to pass the exam. I did some drills open book (as the official exam is) and some without. Both ways are valuable because you want to know the information and you want to know where to find it.

I had one classmate who did not have a college degree and was unaccustomed to the test-taking environment. Note-taking, reviewing, and computer-based exams were all unfamiliar to her. She was very anxious about what the exam would be like and she felt she wasn’t retaining anything during the course. If this sounds like you, anything you can do to become accustomed to that environment will help. Tell the instructor about your background. Take the practice exams. Create flashcards or other learning tools that work for you. Set up a timer and get used to how it feels to be timed while answering questions.

The exam itself

My exam was administered by Prometric, a third-party testing company. My exam was on Saturday morning. I had a good experience and found it was the same as Pearson Vue, where I took my EMT-B exam several years ago.

Read all the exam rules in advance and arrive with what you need. Make sure you have removed the practice exam questions from your CQIA Primer and that all your materials are bound.

The exam was the exact level of difficulty as the practice exam questions. As I said above, that program was very, very helpful for focusing on areas where my recall was weak.

You will spend lots of time flipping through the book looking for key tables and definitions. If you can find an answer within two minutes using the index or your own sticky flags (which are permitted), you will succeed on the exam.

There is plenty of time to finish the exam. I left with an hour to spare even though I went back to look up many flagged questions.

I used the exam software features to my advantage: a strikethrough to eliminate distracter choices, a timer to keep an eye on my progress, and a flag function that lets you revisit the question. I did not use the physical dry-erase board or the calculator that Prometric provided.

I did not take a break, but this is an option if you want one.

There was no math. At all.

I used the glossary of the CQIA Handbook several times. It was faster and sometimes more detailed than the index of the Quality Council of Indiana book. When a question was difficult and the choice depended on wording, I consulted the relevant section of both the Primer and the handbook.

ASQ certification pathways

My aim is for further certification. This graphic is pretty ugly but it gives you an idea of how to build on previous certifications as you grow more advanced and more focused on your area of quality.

Don’t take this chart as the final word. If you have questions, attend a chapter meeting of ASQ and talk to the presenter or another quality person. They will have firsthand knowledge of certifications. Tell them your situation and goals and ask for suggestions. Bring the same information to your manager and get their opinion as well, before committing to a specific certification.

As you can see, the CQIA is an initial certification that can lead to several professional ones.

Should you get the certification?

 

Yes, you should. Even if you are not in a traditional quality role, you may benefit from the CQIA. It may open the door to Six Sigma, lean, team dynamics skills, statistical process control, etc. Each of these is just a small section of the Primer but a vast field of practice of their own.

I am fortunate to be in a company that paid for my materials and for the exam as part of their dollars-and-sense interest in furthering my development. I hope you are in the same situation. If not, perhaps you can convince your manager to come around. If they won’t cover the cost, just pay for it on your own. The total will run you about $300. That might hurt a bit, but it will strengthen in your future earnings potential and pay off in the end.

Once you have the certification, you have it for life. You can keep the materials at your desk and share your learnings with coworkers. And if you progress to further, more expensive or labor-intensive certifications, you will walk in with greater confidence.

Final comments

What do I mean when I say I “boosted my credibility?” In quality we are not just people with sharp eyes or attention to detail. People who work in quality do not have a certain personality type. Rather we are following rigorous methods developed toward the end of the Industrial Revolution to ensure safe products, reliable service, unexpected delight and longer, healthier lives.

To study quality is to apply the scientific method in your day-to-day work. Getting a certification such as the CQIA helps you learn these methods and meet the experts who practice them. It thereby signals your credibility. It shows others in your organization that you are on the path toward expertise and that your work is based on the methods of a true profession.

As you consider the CQIA (or if you are preparing for the exam), feel free to reach out to me for more details. I have plenty of study tactics to share and some ideas about quality certifications that I didn’t cover in this article.

I wish you success!

 

Resources

Certified Quality Improvement Associate Primer

Certified Quality Improvement Associate Exam USB Flash

Find both at (http://www.qualitycouncil.com/cqia/)

Get both of these! The Primer is absolutely required. The practice exam software is a near-necessity.

 

The Certified Quality Improvement Associate Handbook, Third Edition 3rd Edition

This is not a necessity, but I would recommend it. It is more attractive and better organized than the Primer, and it will be useful during the exam.