|Year : 2019 | Volume
| Issue : 2 | Page : 113-120
The quality of physiotherapy care: the development and application of quality indicators using scientific evidence and routinely collected data embedded in the process of clinical reasoning
Rob A.B Oostendorp1, J.W. Hans Elvers2, Emiel van Trijffel3
1 Scientific Institute for Quality of Healthcare, Radboud University Nijmegen Medical Centre, Nijmegen; Department of Manual Therapy, Faculty of Medicine and Pharmacy, Vrije Universiteit Brussel, Brussels, Belgium; Department of Physiotherapy, International Research Group - Pain in Motion-, Vrije Universiteit Brussel, Brussels, Belgium; Practice Physiotherapy and Manual Therapy, Heeswijk-Dinther, The Netherlands
2 Department of Public Health and Research, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands
3 Department of Manual Therapy, Faculty of Medicine and Pharmacy, Vrije Universiteit Brussel, Brussels; Department of Education, SOMT University of Physiotherapy, Amersfoort, The Netherlands; Department of Physiotherapy, Human Physiology and Anatomy, Faculty of Physical Education and Physiotherapy, Vrije Universiteit Brussel, Brussels, Belgium
|Date of Submission||23-Mar-2019|
|Date of Acceptance||17-Jun-2019|
|Date of Web Publication||9-Oct-2019|
PhD, PT, MPT Rob A.B Oostendorp
Oude Kleefsebaan 325, 6572 AT Berg en Dal, The Netherlands
Source of Support: None, Conflict of Interest: None
Quality improvement has become a central tenet of physiotherapy care. Quality indicators (QIs) as measurable elements of care have been used over the past 25 years to analyze and evaluate the quality of physiotherapy care. The aim of this article is to describe the state of the art regarding the development and application of QIs in physiotherapy primary care when embedded in a clinical reasoning process. In contrast to international clinical practice guidelines, Dutch physiotherapy clinical practice guidelines are generally based on the clinical reasoning process in combination with best available evidence. Information required to develop QIs is preferably derived by combining available systematic review-based scientific evidence, guideline-based recommendations, and routinely collected data with clinical evidence, professional expertise and standards, and patient perspectives. A set of QIs (n=28) in patients with whiplash-associated disorders was developed and embedded per step of the clinical reasoning process in physiotherapy care: (a) administration (n=2); (b) history taking (n=7); (c) objectives of examination (n=1); (d) clinical examination (n=4); (e) analysis and conclusion (n=2); (f) treatment plan (n=3); (g) treatment (n=2); (h) evaluation (n=5); and (i) discharge (n=2). The use of QIs represents a useful tool for measuring the (improvement of) quality of physiotherapy primary care, as many evidentiary gaps still exist in terms of diagnostics, prognostics, and treatment, and concerning patient-related outcome measurements in different patient groups such as patients with musculoskeletal pain. The recommended set of QIs embedded in the clinical reasoning process for patients with whiplash-associated disorders can be used as a starting point for the development of a general set of QIs that measure the (improvement of) quality of primary care physiotherapy.
Keywords: clinical practice guidelines, level of evidence, physiotherapy, quality indicator, routinely collected data, scientific evidence real-world evidence
|How to cite this article:|
Oostendorp RA, Elvers JH, van Trijffel E. The quality of physiotherapy care: the development and application of quality indicators using scientific evidence and routinely collected data embedded in the process of clinical reasoning. Bull Fac Phys Ther 2019;24:113-20
|How to cite this URL:|
Oostendorp RA, Elvers JH, van Trijffel E. The quality of physiotherapy care: the development and application of quality indicators using scientific evidence and routinely collected data embedded in the process of clinical reasoning. Bull Fac Phys Ther [serial online] 2019 [cited 2020 Feb 28];24:113-20. Available from: http://www.bfpt.eg.net/text.asp?2019/24/2/113/268685
| Introduction|| |
Quality improvement has become a central tenet of physiotherapy care and a statutory obligation in many countries . There are numerous reasons why it is important to improve the quality of physiotherapy care, and these include enhancing the clinical reasoning process and making best use of clinical practice guidelines (CPG) and scientific evidence, improving patient-related outcomes and safety, and aligning care to what patients want in addition to what they need. These factors have prompted many new initiatives to develop and apply quality measurements, that is, quality indicators (QIs), over the past decades .
The purpose of this article is to describe the state of the art regarding the development and application of QIs in primary care physiotherapy when embedded in a clinical reasoning process.
Defining quality indicators and clinical reasoning
QIs have been defined as ‘measurable elements of practice performance for which there is evidence or consensus that they can be used to assess the quality of the care provided’ . QIs may relate to structures (such as staff, equipment, and appointment systems), processes (such as clinical reasoning), or outcomes of care (such as a patient’s functioning, disability, and participation) . QIs have been used over the past 25 years to analyze and evaluate the quality of physiotherapy care ,. Rational development of QIs is preferably based on systematic reviews and CPGs, supplemented by expert clinical experience and patient perspectives and values. Process and outcome indicators are also often based on patient registration systems that encompass operationalized, guideline-based recommendations as measurable elements of clinical practice ,,. It is generally accepted that the quality of patient registration systems is a reliable indicator of the quality of care .
In contrast to international CPGs, for example, for whiplash-associated disorders (WAD) , low back pain , and neck pain , Dutch physiotherapy CPGs are generally based on the clinical reasoning process in combination with best available evidence . Most Dutch physiotherapy CPGs and evidence statements have been translated into the English language under the auspices of the Royal Dutch Association for Physiotherapy (KNGF), including those on low back pain  and neck pain .
Clinical reasoning has been defined ‘as a process in which the physiotherapist, interacting with the patient and significant others, structures meaning, goals and health management strategies based on scientific evidence, clinical data, client choices and professional judgment and knowledge’ ,. Clinical decisions based on this clinical reasoning process are not only related specifically to guideline-based recommendations but are also consistent with a more comprehensive approach to managing patients with, for instance, musculoskeletal pain, such as nonspecific low back pain or neck pain.
The clinical reasoning and decision cycle is an internationally accepted concept that facilitates problem solving and decision making in daily practice. The transparency of the clinical reasoning process, as provided by measurable elements such as QIs, is considered a cornerstone of the quality of care .
Development and application of quality indicators
Three issues are important when developing (content validity and reproducibility) and applying (acceptability, feasibility, reliability, sensitivity to change, and predictive validity) QIs: (a) which stakeholder perspective(s) are the indicators intended to reflect, (b) what aspects of health care are being measured, and (c) what evidence is available?
There are different stakeholders of physiotherapy care (physiotherapists, patients, practice managers, professional associations, health insurance companies, policymakers, and politicians). Different perspectives of stakeholders may need different sets of QIs, particularly as stakeholders have different perspectives about quality of physiotherapy care. Physiotherapists tend to focus on the quality of implementation and evaluation of CPGs, including process and outcome indicators.
The most commonly used method for development of QIs in the Netherlands is an iterated consensus rating procedure (similar to that used internationally) . A number of Dutch studies of different patient groups have generated a set of guideline-based QIs, expressed as percentages ranging from 0 to 100%, with the number of times a QI was met as the numerator and the number of patients assessed as the denominator ,,,,,,. We give a few examples from our study of patients with WADs . The numerator score for the number of patients subjected to a methodically performed history taking and recording of sociodemographic characteristics (noted as yes) was 365; the extent to which examination objectives were in agreement with patient’s history taking (noted as yes) was 319, and the extent to which treatment goals were in agreement with prognostic health profile and time phase since accident (noted as yes) was 411. The denominator was the number of patients who participated in the study (N=457). The QIs were 79.9, 69.8, and 89.9%, respectively. To allow for interpretation as performance targets, percentage scores of QIs were categorized as negligible (0–20%), weak (21–30%), very inadequate (31–40%), inadequate (41–55%), sufficient (56–65%), substantial (66–75%), good (76–85%), very good (86–95%), and excellent (96–100%). The achieved QIs in the given examples in our study were respectively good, substantial, and very good.
As mentioned previously, a desired performance target can be determined in consultation with different stakeholders. The Dutch Royal Association for Physical Therapy (KNGF), in consultation with physiotherapists working in primary care, has set the target standard for QIs concerning the steps of the clinical reasoning process to a minimum of ‘substantial’ (66–75%). This minimum has been chosen to prevent ceiling effects. Ceiling effects in this context refer to the percentage of physiotherapists that have the highest score possible, thus making it difficult to measure relevant changes in the quality of physiotherapy care over time.
Method of development and evidence supporting quality indicators
The preferred method of QI development consists of five steps: (a) extraction of recommendations from CPGs, patient-related outcome measurements, and literature, particularly systematic reviews; (b) transformation of recommendations into QIs by phrasing them as the average degree (in %) to which patients were subjected to a methodically performed clinical reasoning process, including the level of evidence supporting the formulated QIs graded from levels I–IV, based on a national consensus document ; (c) appraisal by an expert and user panel, including scoring of the set of QIs on a five-point Likert scale (1=not at all to 5=completely) based on acceptability, feasibility, clarity, and relevancy to the physiotherapy care process; (d) classification of process indicators into the nine steps of the clinical reasoning process; and (e) classification of outcome indicators in accordance with the International Classification of Functioning, Disability and Health (ICF)  such as body functions, activity and participation, and personal and environmental factors. For a detailed description of the methodology involved in the development of QIs for general practice, see Campbell et al. , and for physiotherapy, see Oostendorp et al. , Oostendorp et al. , and Scholte .
The methods used for indicator development in physiotherapy are now briefly explained by means of a recently published example on the quality of physiotherapy care in patients with WAD . Recommendations (n=96) in relation to the physiotherapy clinical reasoning process were independently extracted by two specialized physiotherapists from the Dutch CPG Physiotherapy Management and WADs ,, and the Quebec Task Force on WAD . These recommendations were then transformed into a set of 28 QIs by phrasing them as the average degree (in %) to which patients were subjected to a methodically performed clinical reasoning process, for example, the average degree (in %) to which patients underwent a methodically performed history taking, the average degree (in %) to which accident-related information was noted, the average degree (in %) to which treatment goals were determined and recorded in agreement with individual prognostic health profiles and the time phase since an accident, and the average degree (in %) to which physiotherapy modalities agreed with treatment goals and with time phases since an accident.
From quality indicators to the process of clinical reasoning and evidence support
In the aforementioned example, the set of QIs was classified per step of the clinical reasoning process in physiotherapy care, including the number of QIs and the level of evidence per step: (a) administration (n=2); (b) history taking (n=7); (c) objectives of examination (n=1); (d) clinical examination (n=4); (e) analysis and conclusion (n=2); (f) treatment plan (n=3); (g) treatment (n=2); (h) evaluation (n=5); and (i) discharge (n=2). A complete overview of these 28 indicators was recently published as a supplement to our study . [Table 1] presents the complete set of QIs (n=28) for the physiotherapy care process of patients with WADs .
|Table 1 Set of quality indicators for physiotherapy care process of patients with Whiplash-associated disorders: steps of clinical reasoning, number of indicators per step, item measured, indicator, and level of evidence|
Click here to view
Only two indicators were supported by level I evidence (psychometric quality of the outcome measures), whereas five indicators were partly supported by level II evidence (evidence combined with consensus). Twenty-one indicators were supported by level IV evidence (expert opinion and professional consensus or standards). Most QIs were therefore based on level IV evidence. [Table 2] presents the levels of evidence per step of the process of clinical reasoning.
|Table 2 The number of quality indicators (n=28) per step of the clinical reasoning and decision processa and per level of evidenceb|
Click here to view
Internationally, a number of clinical reasoning models in physiotherapy have been described, such as deductive reasoning versus narrative reasoning ,. In our study , the clinical reasoning and decision-making process was based on a combination of deductive and narrative reasoning, which can be compared to internationally accepted general instruments such as the Hypothesis-Oriented Algorithm for Clinicians (HOAC II) ,. Recently, a critical review described the utility of the ICF model in facilitating clinical decision making for physiotherapists and structuring the documentation of assessments and interventions . The ICF model facilitates the process of clinical reasoning and decision making but is not a measurement tool for the quality of physiotherapy care. The ICF facilitates ‘what to measure’ but not ‘how to measure the quality of care.’
Despite a general focus on clinical reasoning over the past decades in medicine and physiotherapy, a method of evaluating the clinical reasoning process that is both objective and comprehensive has limited ability to evaluate this process . The Script Concordance Test is one of the evolving tests which are considered to be valid and reliable tools for assessing clinical reasoning and judgment ,. To the best of our knowledge, the use of various clinical reasoning and decision models in physiotherapy is not linked to a set of QIs. This means, in effect, that (improvement of) the quality of the clinical reasoning process in physiotherapy care is barely measurable.
Incompleteness of clinical reasoning in randomized clinical trials
Randomized clinical trials (RCTs) are commonly conducted to estimate the effectiveness of physiotherapy interventions. However, there are many areas of physiotherapy care for which the level of evidence is limited or entirely lacking, especially within the setting of the primary care physiotherapy practice.
In this context, a study by Maissan et al.  is instructive as it provides insight into the completeness of the clinical reasoning process in RCTs. In most RCTs (n=122) involving patients with nonspecific neck pain, the clinical reasoning process was reportedly incomplete, specifically in the diagnostic aspect of the process, with only 6% of the RCTs including a complete diagnostic process . Similar findings were reported in the study of Smith and Bolton  who found that RCTs (n=30) included in the systematic review did not report diagnostic strategies and criteria for spinal manipulative therapy in patients with neck pain. These are important findings because it reveals how often the effectiveness of physiotherapy and manual therapy interventions is examined without a prior adequate diagnostic and decision-making process.
These findings suggest that it is necessary to use other methods to develop, implement, and evaluate the process of clinical reasoning, particularly the diagnostic steps of this process. The measurement of (the improvement of) the quality of physiotherapy care should be based on professional consensus and a complete process of clinical reasoning and decision making using a set of QIs from the perspectives of physiotherapists as stakeholder.
Routinely collected data as supplement
The use of routinely collected data (RCD) is one of the preferred methods to measure the (improvement of) quality of physiotherapy care. RCD are collected in practices for reasons unrelated to research or prior research questions and are increasingly used in retrospective research. Nevertheless, RCD are not a substitute for RCTs  but they are a necessary counterpart, allowing measurement of the quality of the clinical reasoning process using QIs. Data from daily practice are readily available (although accuracy and completeness may vary) and represent a potentially rich source of information on large numbers of patients with diverse conditions. Use of existing data is less demanding and has fewer ethical constraints than planning, funding, and executing long-term pragmatic or experimental studies. RCD are diverse, available worldwide in both hospitals and general practice, and include clinical information from electronic health records, disease registries, and epidemiologic surveillance studies. Examples of RCD in primary physiotherapy care are, nevertheless, scarce ,.
QIs derived from RCD may cover the steps of the clinical reasoning and decision-making process (e.g. from the objectives of an examination to a clinical examination or from treatment goals to physiotherapy modalities that agree with treatment goals) or outcome measures (e.g. pretreatment and posttreatment of pain and functioning). However, proper use of RCD may require certain challenges to be overcome . Accordingly, to improve the quality of reporting of studies that use RCD in physiotherapy, a checklist of items − guidelines for REporting of studies Conducted using Observational Routinely collected Data (RECORD) − has been developed  and adopted by journal editors (including the Journal of Orthopedic & Sports Physical Therapy) .
Despite the limitations of RCD studies, we expect that the results of studies using RCD could plausibly act as preliminary evidence regarding the completeness of the physiotherapy clinical reasoning and decision-making process and could be used to improve the design of future RCTs. In summary, although improvement of RCT quality is an important goal, broadening our focus to include the improved, accurate documentation of patient records is also a worthwhile goal.
Quality improvement has become a central tenet of health care, primarily in hospitals but increasingly also in primary care physiotherapy. A variety of methods can be used in processes of quality measurement and improvement. One of the most commonly used methods is the development and application of QIs as measurable elements of care. Information required to develop QIs is preferably derived, using systematic methods, by combining available systematic review-based scientific evidence and CPG recommendations with clinical evidence, professional expertise and standards, and patient perspectives.
The use of QIs derived from RCD represents a useful tool for understanding the quality of physiotherapy care, as many evidentiary gaps still exist in terms of diagnostics, prognostics, and treatment, and concerning patient-related outcome measurements in patient groups such as those with low back pain or neck pain.
The combination of different sources of evidence regarding physiotherapy management in patients with diverse conditions in primary care may provide a broader view of the clinical reasoning process, and a more comprehensive and realistic view of the (improvement of) quality of routine practice compared with data gathered exclusively during an RCT.
The recommended set of QIs embedded in the clinical reasoning process for patients with WAD can be used as a starting point for the development of a general set of QIs that measure the quality of primary care physiotherapy.
International consensus on a set of QIs embedded in the physiotherapy clinical reasoning process, and on performance targets and scoring procedures, would improve the comparability of studies of the quality of physiotherapy care.
The authors declare that they participated in drafting the work or revising it critically for important intellectual content, and gave final approval for the published version.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Braspenning J, Campbell S, Grol R. Measuring changes in patient care: development and use of indicators. In: Grol R, Wensing M, Eccles M, eds. Improving patient care. The implementation of change in clinical practice. Edinburgh: Elsevier Butterworth Heineman; 2005. 222–234
Lawrence M, Olesen F. Indicators of quality health care. Eur J Gen Pract 1997; 3:103–108.
Mainz J. Defining and classifying clinical indicators for quality improvement. Int J Qual Health Care 2003; 15:523–530.
Grimmer K, Dibden M. Clinical indicators for physiotherapists. Aust J Physiother 1993; 39:81–85.
Scholte M, Neeleman-van der Steen CWM, Hendriks HJM, Nijhuis-van der Sanden MWG, Braspenning J. Evaluating quality indicators for physical therapy in primary care, Int J Qual Health Care 2014; 26:261–270.
Campbell S, Braspenning J, Hutchinson A, Marshall M. Research methods used in developing and applying quality indicators in primary care. In: Grol R, Baker R, Moss F, eds. Quality improvement research. Understanding the science of change in health care. London: BMJ Publishing Group 2004. 6–28
Hoque DME, Kumari V, Hoque M, Ruseckaite R, Romero L, Evans SM. Impact of clinical registries on quality of patient care and clinical outcomes: a systematic review. PLoS One 2017; 12:e0183667.
Moore A, Jackson A, Jordan J, Hammersley S, Hill J, Mercer C et al.
Clinical guidelines for the physiotherapy management of Whiplash Associated Disorder (WAD). London: Chartered Society of Physiotherapy; 2005.
Koes BW, van Tulder MW, Ostelo R, Kim Burton A, Waddell G. Clinical guidelines for the management of low back pain in primary care: an international comparison. Spine (Phila Pa 1976) 2001; 26:2504–2513.
Hurwitz EL, Carragee EJ, van der Velde G, Carroll LJ, Nordin M, Guzman J et al.
Treatment of neck pain: noninvasive interventions: results of the Bone and Joint Decade 2000-2010 Task Force on Neck Pain and Its Associated Disorders. Spine (Phila Pa 1976) 2008; 33(4 Suppl):S123–S152.
Meerhoff G, Heijblom K, Knoop J. Methodology for developing/updating and implementing KNGF guidelines. Amersfoort, the Netherlands: The Royal Dutch Society for Physical Therapy (KNGF); 2006.
Staal JB, Hendriks EJM, Heijmans M, Kiers H, Lutgers-Boomsma AM, Rutten G et al.
KNGF Clinical practice guideline for physical therapy in patients with low back pain. KNGF Clinical Practice Guideline for Physical Therapy in patients with low back pain. Amersfoort, The Netherlands: Royal Dutch Society for Physical Therapy (KNGF); 2013.
Bier JD, Scholten-Peeters GGM, Staal JB, Pool J, van Tulder M, Beekman E et al.
KNGF Clinical Practice Guideline for Physical Therapy in patients with neck pain. Amersfoort, The Netherlands: Royal Dutch Society for Physical Therapy (KNGF); 2017.
Higgs J, Jones M, Loftus S, Christensen N, eds. Clinical reasoning in the health professions. Edinburgh: Butterworth Heinemann 2008. 3–18
Jones MA, Rivett DA, eds. Clinical reasoning for manual therapists. Edinburgh: Butterworth Heinemann 2004. 1–24
Chia-Chien H, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res Eval 2007; 12:1–8.
Jansen MJ, Hendriks EJ, Oostendorp RA, Dekker J, De Bie RA. Quality indicators indicate good adherence to the clinical practice guideline on ‘Osteoarthritis of the hip and knee’ and few prognostic factors influence outcome indicators: a prospective cohort study. Eur J Phys Rehabil Med 2010; 46:337–345.
Peter WF, Hurkmans EJ, van der Wees P, Hendriks E, van Bodegom-Vos L, Vliet Vlieland TP. Healthcare quality indicators for physiotherapy management in hip and knee osteoarthritis and rheumatoid arthritis: a Delphi study. Musculoskelet Care 2016; 14:219–232.
Gijsbers HJ, Lauret GJ, van Hofwegen A, van Dockum TA, Teijink JA, Hendriks HJ. Development of quality indicators for physiotherapy for patients with PAOD in the Netherlands: a Delphi study. Physiotherapy 2016; 102:196–201.
Rutten GM, Harting J, Bartholomew LK, Schlief A, Oostendorp RA, de Vries NK. Evaluation of the theory-based Quality Improvement in Physical Therapy (QUIP) programme: a one-group, pre-test post-test pilot study. BMC Health Serv Res 2013; 13:194.
Oostendorp RA, Rutten GM, Dommerholt J, Nijhuis-van der Sanden MW, Harting J. Guideline-based development and practice test of quality indicators for physiotherapy care in patients with neck pain. J Eval Clin Pract 2013; 19:1044–1053.
Oostendorp RA, Elvers H, van Trijffel E, Rutten GM, Scholten-Peeters GG, Heijmans M et al.
Has the quality of physiotherapy care in patients with Whiplash-associated disorders (WAD) improved over time? A retrospective study using routinely collected data and quality indicators. Patient Prefer Adherence 2018; 12:2291–2308.
Scholte M. Developing and implementing quality indicators for physical therapy: lessons learned [Academic Thesis]. Nijmegen, The Netherlands: Radboud University, 2014.
Scholten RJPM, Tuut MK, Kremer LCM, Assendelft WJJ. Beoordelen van de kwaliteit van medisch-wetenschappelijk onderzoek. In: Van Everdingen JJE, Burgers JS, Assendelft WJJ, Swinkels JA, Van Barneveld TA, Van de Klundert JLM, eds. Evidence-based Richtlijnontwikkeling. Houten: Bohn Stafleu Van Loghum; 2004. 158–171.
WHO. International classification of functioning, disability and health. Geneva: WHO; 2001.
Bekkering GE, Hendriks HJM, Lanser K, Oostendorp RAB, Peeters GGM, Verhagen AP, van der Windt DA. KNGF-richtlijn Whiplash. Ned Tijdschr Fysiother 2001; 111(3 Suppl):S1–S25.
Scholten-Peeters GG, Bekkering GE, Verhagen AP, van der Windt DA, Lanser K, Hendriks EJ, Oostendorp RA. Clinical practice guideline for the physiotherapy of patients with whiplash-associated disorders. Spine (Phila Pa 1976) 2002; 27:412–422.
Spitzer WO, Skovron ML, Salmi LR, Cassidy JD, Duranceau J, Suissa S, Zeiss E. Scientific monograph of the Quebec Task Force on Whiplash-Associated Disorders: redefining ‘whiplash’ and its management. Spine (Phila Pa 1976) 1995; 20(8 Suppl):1S–73S.
Rothstein JM, Echternach JL. Hypothesis-oriented algorithm for clinicians. A method for evaluation and treatment planning. Phys Ther 1986; 66:1388–1394.
Rothstein JM, Echternach JL, Riddle DL. The hypothesis-oriented algorithm for clinicians II (HOAC II): a guide for patient management. Phys Ther 2003; 83:455–470.
Allet L, Bürge E, Monnin D. ICF: clinical relevance for physiotherapy? A critical review. Adv Physiother 2008; 10:127–137.
Groves M, Dick ML, McColl G, Bilszta J. Analysing clinical reasoning characteristics using a combined methods approach. BMC Med Educ 2013; 13:144.
Nazim SM, Talati JJ, Pinjani S, Biyabani SR, Ather MH, Norcini JJ. Assessing clinical reasoning skills using Script Concordance Test (SCT) and extended matching questions (EMQs): a pilot for urology trainees. J Adv Med Educ Prof 2019; 7:7–13.
Cooke S, Lemay JF, Beran T. Evolutions in clinical reasoning assessment: The Evolving Script Concordance Test. Med Teach 2017; 39:828–835.
Maissan F, Pool J, de Raaij E, Mollema J, Ostelo R, Wittink H. The clinical reasoning process in randomized clinical trials with patients with non-specific neck pain is incomplete: A systematic review. Musculoskelet Sci Pract 2018; 35:8–17.
Smith J, Bolton PS. What are the clinical criteria justifying spinal manipulative therapy for neck pain? A systematic review of randomized controlled trials. Pain Med 2013; 14:460–468.
Gerstein HC, McMurray J, Holman RR. Real-world studies no substitute for RCTs in establishing efficacy. Lancet 2019; 393:210–211.
Scholte M, van Dulmen SA, Neeleman-Van der Steen CW, van der Wees PJ, Nijhuis-van der Sanden MW, Braspenning J. Data extraction from electronic health records (EHRs) for quality measurement of the physical therapy process: comparison between EHR-data and survey data. BMC Med Inform Decis Mak 2016; 16:141.
Jorm L. Routinely collected data as a strategic resource for research: priorities for methods and workforce. Public Health Res Pract 2015; 25:e2541540.
Benchimol EI, Smeeth L, Guttmann A et al.
The REporting of studies conducted using observational routinely-collected health data (RECORD) statement. PLoS Med 2015; 12:e 1001885.
Benchimol EI, Smeeth L, Guttmann A, Harron K, Moher D, Petersen I et al.
Improving the reporting of studies using routinely collected health data in physical therapy. J Orthop Sports Phys Ther 2016; 46:126–127.
[Table 1], [Table 2]