Quantitative Study

Guide to an Overall Critique of a Quantitative Research Report

NRP/513 Version 2

Introduction

Statement of the problem

Was the problem stated unambiguously, and was it easy to identify?

Is the problem significant for nursing?

Did the problem statement build a persuasive argument for the new study?

Was there a good match between the research problem and the methods used—that is, was a quantitative approach appropriate?

Were research questions and/or hypotheses explicitly stated? If not, was their absence justified?

Were questions and hypotheses appropriately worded, with clear specification of key variables and the study population?

Were the questions/hypotheses consistent with existing knowledge?

Was the literature review up-to-date and based mainly on primary sources?

Did the review provide a state-of-the-art synthesis of evidence on the research problem?

Did the literature review provide a strong basis for the new study?

Were key concepts adequately defined conceptually?

Was a conceptual/theoretical framework articulated—and, if so, was it appropriate? If not, is the absence of a framework justified?

Were the questions/hypotheses consistent with the framework?

Method

Protection of participants’ rights

Were appropriate procedures used to safeguard the rights of study participants?

Was the study subject to external review by an IRB or ethics review board?

Was the study designed to minimize risks and maximize benefits to participants?

Was the most rigorous design used, given the study purpose?

Were appropriate comparisons made to enhance interpretability of the findings?

Was the number of data collection points appropriate?

Did the design minimize biases and threats to the internal, construct, and external validity of the study (e.g., was blinding used, was attrition minimized)?

Was the population identified? Was the sample described in sufficient detail?

Was the best possible sampling design used to enhance the sample’s representativeness? Were sampling biases minimized?

Was the sample size adequate? Was the sample size based on a power analysis?

Were the operational and conceptual definitions congruent?

Were key variables measured using an appropriate method (e.g., interviews, observations, and so on)?

Were the specific instruments adequately described and were they good choices, given the study population and the variables being studied?

Did the report provide evidence that the data collection methods yielded data that were reliable, valid, and responsive?

If there was an intervention, was it adequately described, and was it rigorously developed and implemented? Did most participants allocated to the intervention group actually receive it? Was there evidence of intervention fidelity?

Were data collected in a manner that minimized bias? Were the staff who collected data appropriately trained?

Results

Data analysis

Were analyses undertaken to address each research question or test each hypothesis?

Were appropriate statistical methods used, given the level of measurement of the variables, number of groups being compared, and assumptions of the tests?

Was the most powerful analytic method used? (e.g., did the analysis help to control for confounding variables)?

Were Type I and Type II errors avoided or minimized?

In intervention studies, was an intention-to-treat analysis performed?

Were problems of missing values evaluated and adequately addressed?

Was information about statistical significance presented? Was information about effect size and precision of estimates (confidence intervals) presented?

Were the findings adequately summarized, with good use of tables and figures?

Were findings reported in a manner that facilitates a meta-analysis, and with sufficient information needed for EBP?

Discussion

Interpretation of the findings

Were all major findings interpreted and discussed within the context of prior research and/or the study’s conceptual framework?

Were causal inferences, if any, justified?

Was the issue of clinical significance discussed?

Were interpretations well-founded and consistent with the study’s limitations?

Did the report address the issue of the generalizability of the findings?

General Issues

Presentation

Was the report well-written, organized, and sufficiently detailed for critical analysis?

In intervention studies, was a CONSORT flowchart provided to show the flow of participants in the study?

Was the report written in a manner that makes the findings accessible to practicing nurses?

The following questions address statistical conclusion validity, internal validity, construct validity, and external validity. These questions come from Box 10-1 in chapter 10.

1. Was there adequate statistical power? Did the manner in which the independent variable was operationalized create strong contrasts that enhanced statistical power? Was precision enhanced by controlling confounding variables? If hypotheses were not supported (e.g., a hypothesized relationship was not found), is it possible that statistical conclusion validity was compromised?

2. In intervention studies, did the researchers attend to intervention fidelity? For example, were staff adequately trained? Was the implementation of the intervention monitored? Was attention paid to both the delivery and receipt of the intervention?

3. What evidence does the report provide that selection biases were eliminated or minimized? What steps were taken to control confounding participant characteristics that could affect the equivalence of groups being compared? Were these steps adequate?

4. To what extent did the study design rule out the plausibility of other threats to internal validity, such as history, attrition, maturation, and so on? What are your overall conclusions about the internal validity of the study?

5. Were there any major threats to the construct validity of the study? In intervention studies, was there a good match between the underlying conceptualization of the intervention and its operationalization? Was the intervention “pure” or was it confounded with extraneous content, such as researcher expectations? Was the setting or site a good exemplar of the type of setting envisioned in the conceptualization?

6. Was the context of the study sufficiently described to enhance its capacity for external validity? Were the settings or participants representative of the types to which results were designed to be generalized?

7. Overall, did the researcher appropriately balance validity concerns? Was attention paid to certain types of threats (e.g., internal validity) at the expense of others (e.g., external validity)?

Finally,

Despite any limitations, do the study findings appear to be valid—do you have confidence in the truth value of the results?

Does the study contribute any meaningful evidence that can be used in nursing practice or that is useful to the nursing discipline?