Reviewing a scientific report is a great privilege. You are among the first to see new knowledge without having to do the sometimes-tedious investigative work itself. And if you’re like me, you might feel the joy of learning something new.
When reviewing, I encourage you to do two things. Firstly, commit to applying a method, just as the investigators applied a method when conducting their study. Secondly, consider the review, your comments, and the investigator’s response as a conversation between the two of you. This conversation has a subject. It may go off in an unexpected direction. And it may expand to include other people.
In this article I will describe how to approach the important role of reviewer.
Who you are
My primary assumptions about you are twofold:
- You do this for work (you are not a scholar doing peer review for free).
- You follow your organization’s standard operating procedures (SOPs) and national or international regulatory guidance.
I also assume that you are the only reviewer. If you are the Quality reviewer, but the report has already undergone a thorough technical review described in your SOPs, then your review will be greatly reduced in scope (and probably much easier!).
Lastly I assume you are properly trained. You must not only be familiar with the procedures of your reviewer or Quality unit, but also with the procedures the investigators followed and with the science underlying their methods. If some calculations were done in Excel, you should be familiar with Excel. If some statistics were done in Minitab, you should be familiar with Minitab.
Gather your materials
You should have four things at a minimum:
- The draft report
- The protocol (plus applicable SOPs)
- The raw data
- Your checklists and procedures and your form or software for documenting findings
Organize
The following setup is optimal to me: the draft report is on the computer screen in front of me. The raw data and the protocol are on my left. My materials (such as a checklist, inspections from during the study, and any other Quality forms) are on the right. These Quality materials may be software-based, in which case it should be open onscreen so you can document findings in real-time, while reviewing.
Check for completeness
Before beginning, make sure everything is there so that you can plan for a thorough review with minimal handoffs (handoffs are where mistakes often happen).
Read the intro/summary/abstract
Even if this section is at the end of the report, it is a good thing to check before getting deep in the weeds.
Read the body of the report and make sure all the sections specified in the protocol/SOPs are present
Title Page/Cover Page
This should include a unique identifier, the name of the study, and the sample name and lot number (if applicable). Your organization’s logo or letterhead may be required as well. The investigators, the dates of Quality inspections, the date of finalization, and the Quality reviewer (if applicable) may appear here or on a signature page. The client (if applicable) and testing facility are listed here as well.
Table of Contents
Depending on the length of the report, a table of contents may be helpful. Ideally this will automatically update in Word. If changes are anticipated, consider verifying the table of contents at finalization to avoid duplicating your effort.
Abstract
Peer-reviewed reports will require an abstract. The requirements are very specific to the journal. Your organization probably does not require one.
Introduction/Purpose
The introduction should basically be a high-level summary of what is in the protocol. If the writer briefly stated the scientific background, the goal of the current study, and the methods in simple language while still being descriptive, he or she has done a great job!
Some studies may state the overall conclusion in this section as well. If so, check it against the conclusion section below. If this section is labeled Purpose instead of Introduction it should correspond to the Purpose section of the protocol.
Materials and Methods
This section should mirror the protocol because the protocol is basically a promise or agreement about what will be done. The materials, such as reagents, instruments, and suppliers, may be in their own section or in tables interspersed with the text. Check each against the raw data as you go.
Pull up applicable SOPs as you review. Although these documents often read like work instructions, they may provide greater detail than the protocol about what was done. Try to avoid “filling in the blanks” though. The report should stand on its own.
Read for flow and for accuracy to what is described on the lab notebooks or worksheets. Check that all specifics, such as temperatures, weights, and incubation durations match what was recorded. If there are any deviations, these may need to be addressed in the report. The need for a retest or an investigation may first be discovered here.
Raw data and report disposition
This section is important! The retention period of the report and raw data must follow regulations of the FDA, EPA, or other agencies. The report should describe where the raw data will be held so that clients or regulators can compare the report to it during an audit or investigation.
In general, the Quality review documentation you are completing right now will be stored with this package and for the same retention period.
Validity criteria
I like to see validity as a standalone section because it is so important. This and the following (evaluation/acceptance) deserve your full attention.
Validity is the soundness of the entire study. In an animal test for irritation, a valid determination may require that animals in a positive control group developed irritation as expected. In a chromatography study, a valid identification of a compound may require that the signal-to-noise ratio be below a certain threshold. In an ELISA test, a valid assay may require that the calibration curve is linear and not just a jumble of dots.
In routine assays that seldom have validity problems, this section can be easy to overlook. The text for this section may be canned. In your review, take a moment to look at validity. There may be one validity criterion or there may be several.
Without a valid assay, the investigator cannot make an evaluation (below). Or, he or she cannot make an evaluation without a compelling justification.
Evaluation/acceptance criteria
Evaluation/acceptance criteria are very important as well. They might be something specific such as “to pass, the white blood cell count must be within 4,500-10,000 white blood cells per cubic millimeter.”
In a larger study, the evaluation may be much broader. The investigator may look at the totality of health observations, bloodwork, feed consumption, histopathology, and necropsy observations. In this case, a lot of judgement is involved on the part of the investigator. As a reviewer comparing the raw data to the report, you are verifying that the evaluation broadly matches what you see in the raw data.
In both cases, the evaluation should be based on what was described in the protocol. Because of its deciding nature, to alter it or introduce other evaluation criteria requires a protocol amendment or a deviation.
Results
Reviewing results may take up the bulk of your review. Depending on your SOPs you may check 100% of transcription or just do a spot check. The report may present every data point collected, or just summary tables. You should be checking that all of it is traceable to the correct place in the raw data, that any gaps are labeled so in the report tables, and that the text matches what is in the tables and what is in the raw data.
If there are voluminous data tables to check, you may choose to review these last so that you can read the report in one sitting while it is fresh in your mind.
Discussion
The discussion section is where the results are interpreted. It should connect to the introduction or purpose of the study and then move beyond. You should check it against everything that is applicable: the protocol, the results you have just reviewed, the evaluation and validity criteria, and the relevant citations. This section is free to be lengthy but it should not restate too much from elsewhere in the report. Instead, it should lean heavily toward context, synthesis and interpretation.
Depending on your training and role in the organization, you may be responsible for assessing the scientific rationale of the report. But a typical Quality reviewer will not be responsible for this. Instead, questions of scientific judgement are left to the investigator and his or her peers, who will have already reviewed the report or at least the associated raw data.
Conclusion
The conclusion is best kept brief! If it is lengthy, suggest in your comments that some of it be moved to the Discussion section.
Both short-and-simple and long-and-complex studies are free to have a one-sentence conclusion. As a reviewer, you should double-check that the conclusion matches the interpretation arrived at in the Discussion section and does not go beyond it.
Although the conclusion is one sentence, a word on scope and a qualifier are always scientifically justified. Consider the following one-sentence conclusion:
“Under the conditions of this protocol, the levels of endotoxin in the sample were found to be below the limit of detection.”
The investigator referenced the scope of the study (the current protocol and its limitations), provided a scientifically cautious qualifier (there could be endotoxin, but not enough to detect), and clearly stated the conclusion, all in one sentence.
References
For a non-peer-reviewed study, this is not as important. It is the responsibility of the investigators to base their study on sound science. However, you should check that the resources cited exist, are cited correctly, and are accessible. If your organization is issuing reports based on papers from obscure journals from the distant past that can’t be found online or in your reference library, you might have a problem.
Check transcription
A 100% transcription check is an unambiguous requirement. It is clear for the reviewer. It is clear to the investigator that their transcription will be verified. After establishing a 100% transcription check requirement, you will start getting higher-quality raw data by the time of review. This means less back-and-forth and less reprocessing of data (data transcribed incorrectly into Excel or statistical software causes incorrect results).
Spot-check calculations
Your organization may require spot-checks or that a certain percentage of calculations be checked, or that 100% be checked. Generally, you will not need to check calculations done within statistical software. If possible, you should have a digital copy of any Excel spreadsheets used so that you can display the functions and check the calculations the easy way.
See the investigator if needed
Consulting the investigator before your review is finished is a matter of judgement. Here’s what I mean: if you can’t find what you are looking for or can’t understand a calculation, you will definitely want to ask the investigator to clarify. However, if you identify a major deviation or some other issue, you want to ensure that this is documented and winds up in your quality system. You want each major finding to be documented and to have a documented response.
The same goes for minor errors. You don’t want to hand the report and raw data back and forth, for multiple review cycles, while they address issues that come up during your review. Instead, there should be one efficient but thorough review of the complete package, followed by a write-up. Subsequent reviews should be brief, involving only signing and sending.
You might also consult a third party such as a statistician or manager.
Write it up
Organize your findings cohesively and fill out any checklists you may have. If applicable, categorize your findings by major and minor. If possible, each of your findings should describe what was found, reference the standard it is being compared to (the protocol, an SOP, etc.), and the consequences of not meeting the standard.
If the report is not acceptable or not acceptable without additional experimental work, make this clear. If a major deviation was identified, you should let the investigator know in person so they can resolve it quickly. Any required investigations should be closed and addressed in the report when the report comes back to you for finalization.
At finalization, see that all findings are addressed
When the report comes back to you for finalization, you will ideally only be reviewing minor changes and the addition of finalization dates and cover pages, logos, etc. You will want to ensure that all investigations are closed and that all your findings are addressed. Each finding should be closed out individually.
Do a final look-over
Finally, check the boring stuff! Make sure the pagination is correct and that all the pages are there. Be ready for third and fourth reviews as minor printing or electronic signature issues are sorted out.
Sign off and then thank the investigator for their hard work.
Some further reading
An excellent and much-shared article geared toward peer-reviewed papers. There is good advice in the comments section as well:
A well-organized breakdown of the elements of a scientific report from the University of Waikato (New Zealand):
http://www.waikato.ac.nz/library/study/guides/write-scientific-reports
A detailed guide to writing a discussion section from the University of Southern California:
Quality Systems Analyst 