A Meta-Evaluation : The Role of Treatment Fidelity within Psychosocial Interventions during the Last Decade

Treatment fidelity is widely accepted as a critical component of successful psychological program implementation and treatment integrity. The current study is a critical interpretive synthesis of the review and evaluation literature on Treatment fidelity in psychosocial interventions in the last decade with an aim to explore the emphasis placed by researchers on: i) definitions of treatment fidelity; ii) different components of treatment fidelity; and iii) existing strategies to enhance treatment fidelity in clinical practice. This encompasses ongoing clinician training, mentoring and supervision; provision of robust, cost efficient and portable suite of complimentary-tools. The results of the metaevaluation highlighted researchers attached varying importance to strategies to enhance treatment fidelity in clinical practice. Several recommendations for treatment fidelity in clinical practice including: development of standardised definitions of treatment fidelity within an overarching treatment fidelity Model, incorporating measurement of clinician and client fidelity (through use of competence and adherence scales), and support for investment in professional development for clinicians. Areas of focus for future research include further examination of the linkage between clinician fidelity to client outcomes, and exploration of the value of measuring treatment fidelity overtime.


Introduction
Dissemination researchers have debated for some time the importance and merits of treatment fidelity versus clinical adaptation of treatments to different settings.This increased emphasis has cast a spotlight on the significance of including robust and holistic treatment fidelity measures within the design, training, and treatment delivery, treatment receipt and treatment enactment of researchprograms implemented in a mental health and clinical settings [1].Treatment receipt and treatment enactment are intrinsically linked.Treatment receipt focuses on whether the client has retained the taught information and has the confidence to implement changes within their daily lives Treatment enactment centres around the client's ability to actually implement the taught skills at the right time and within the right setting [2].These elements of client behaviour are also known as client treatment fidelity.Potentially powerful and highly effective programs with insufficient Treatment fidelity measures run the risk of premature rejection while conversely, ineffective programs may be wrongly accepted [3,4].Two further advantages of treatment fidelity are simplicity and cost effectiveness.On the other hand, definitions of Treatment fidelity vary and with limited empirical guidance available to guide implementation of treatment fidelity components [5] variance across and within specialist disciplines remains common.The metaevaluation adopted in this article allowed a critical assessment of pertinent literature reviews from the last decade, to better understand these mentioned gaps, and opportunities to strengthen implementation success.

Defining treatment fidelity
Treatment fidelity is commonly defined as the degree to which an intervention or program is delivered as intended [6][7][8][9][10].This brief yet broad definition can be applied universally across disciplines of Medicine, Psychiatry, Psychology, Nursing, Social Work, Occupational Therapists and Education.Although treatment fidelity is further defined within each discipline, it remains a term that is often used loosely.With the growing emphasis on treatment fidelity there is increasing awareness that the concept is multi-faceted and complex.Commonly identified elements included treatment adherence, therapist competence, differentiation (disparity in delivery), duration (of treatment sessions and overall program) and dose (frequency of treatment delivery) [7,11,12].Despite the varying definitions and conceptualisation of treatment fidelity, most of these link fidelity with outcomes.

Benefits of treatment fidelity
Treatment fidelity offers a suite of benefits to the designer, assessor, clinician, and client including measuring the level of fidelity with which an intervention is delivered within a community or clinical setting.Such understanding allows researchers to more accurately evaluate whether a failure to replicate research in clinical settings is due to a flaw in program design or inadequate delivery by the clinician [7,13].High levels of treatment fidelity best positions researchers to revise interventions, provide clear strategies to enhance reliability of program, promote study replication, complement delivery of manual based interventions, identify clinician deviation, understand study findings, and increase statistical power by reducing unintended variability [4,[14][15][16]; collectively promoting enhanced stakeholder confidence in the delivered treatment and potentially improving client outcomes [17].

Impediments to treatment fidelity
Several key barriers impact upon successful adoption of treatment fidelity Models by researchers and clinicians, including differing definitions of treatment fidelity, competing treatment fidelity Models and variation in the emphasis placed upon continued professional development for those involved in program delivery.A ten year review of psychiatry and psychology research studies concluded only 3.5% adequately addressed requirements of treatment fidelity [11,18].Seeking to address perceived barriers, the National Institute of Health's Behavioral Change Consortium added provider training to the five key requisites of an effective treatment fidelity Model (study design, training, delivery, receipt and enactment) [3].
While benefits of treatment fidelity have been well documented and impediments to treatment fidelity are well recognised there remain gaps in our understanding of the impact of this knowledge on practice in the field.This paper utilised a meta-evaluation for critical analysis of contemporary literature to highlight opportunities to enhance future treatment fidelity theoretical and contemporary frameworks.This study aims to further expand current understanding and knowledge among program designers, researchers, assessors and clinicians as to the key elements of treatment fidelity required to improve program delivery and/or implementation.The current study is a critical interpretive synthesis of the review and evaluation literature on treatment fidelity in psychosocial interventions in the last decade with an aim to explore the following questions: 1) How many authors included definitions of treatment fidelity?
2) What treatment fidelity components were discussed by authors?
3) What strategies aimed to enhance treatment fidelity within clinical practice were identified in the meta evaluations?

Study selection
Only reviews, meta-evaluations, and Special Guest articles published in English in peer-reviewed journals were included.Development of scales, clinical trials, qualitative research, debates, and commentaries articles were excluded.Prior to commencing the systematic review, a preliminary search of the Database of Abstracts of Reviews of Effects (DARE), and the Cochrane Database of Systematic Reviews (CDSR) was undertaken to identify similar reviews.The PRISMA Checklist Guidelines for systematic review and meta-analysis [19] were used to assist in the current synthesis.

Search strategy
A systematic review of English articles using PsychInfo, MEDLINE, Google Scholar, EBM Reviews, Cochrane Database of Systematic Reviews (CochraneDSR) and Dissertation Abstracts was completed.Search terms included treatment fidelity * , integrity * , intervention integrity, adherence * , competence * , and implement * , scale * , assessment * , monitor * and outcome measure * .Additional studies were indentified by reviewing reference lists of relevant articles.

Data extraction
Independent extraction of articles was undertaken by two research assessors (PP and Research Assistant Ted Graham) using predefined data fields including study quality indicators (Figure 1).Each article was assessed against inclusion criteria as follows: systematic reviews, meta-evaluations, meta-analysis, literature review, mental health treatments research, English, and full text articles.Exclusion criteria entailed: non-review articles, non-peer reviewed papers, trials, reviews not explicitly assessing treatment fidelity, reviews focused on trials of dosage adherence, debates, letters, opinions, development of scales articles and qualitative studies.If the assessors agreed the reviewed paper did not supply sufficient pre-requisite information about treatment fidelity, the paper was omitted from the review.Any disagreement was discussed using the inclusion and exclusion criteria until consensus was reached.Papers double reported were identified to preserve data integrity.The assessors were not blind to the names of authors, institutions, or journal of publication.

Design
Stage one: using meta-evaluation to defining scope: A metaevaluation framework was used to synthesise the included articles.Meta-evaluations have been described as systematic reviews of evaluations to determine the quality of the review.They vary in stance from that of a narrative critique to a simple audit [20].The current study undertook a detailed review of the existing academic peer review literature through a staged process.The author defined the scope of the meta-evaluation with key elements including: • Fidelity (does the author discuss fidelity interventions?), • Comparison (are the relative effects of different concepts discussed?), • Adjunction (are theories discussed according to best fit with the current evidence?) • Reality testing (is there discussion of translation of findings into clinical practice?)

Stage two: review of literature:
The review literature on psychosocial implementation over the last decade presented a challenge due to the lack of consensus regarding standardized vocabulary and implementation processes.Major themes contained within included reviews quickly became evident during the analysis phase of the literature; treatment fidelity definitions, the use of various treatment fidelity Models, measurements of treatment fidelity, clinicians' professional development during and post treatment, linking clinician fidelity with client outcomes, and measuring treatment fidelity overtime.There was marked variation in the emphasis placed by researchers on treatment fidelity definitions.The definitions varied in focus and specificity depending on the context in which the term was used [1,21] with some articles excluding a treatment fidelity definition [22,23].A second theme arose in relation to the existence of differing models of treatment fidelity.These can generate confusion and make drawing comparisons burdensome [9,17,21].Another area of variance is the manner in which researchers conceptualise and measure fidelity implementations [24].Theoretically, adherence, competence and differentiation in treatment fidelity should be able to be individually measured [11].Adherence scales have been developed which generally assess the frequency with which the clinician delivers components of treatment as intended.Competence scales generally relate to assessment of the skill the clinician brings to the delivery of key treatment components.Bernard and Goodyear [25] emphasised the importance of rigorous evaluation (including using adherence and competence scales) of clinicians during treatment implementation.Adherence is an essential component to measure clinician competence and to detect when a clinician may be drifting away from treatment protocols [26].Further suggested methods to improve treatment fidelity include the training, supervision and professional development [27].Training and supervision (pre, during and post of treatment implementation) provides a readily assessable feedback for the implementation process and promotes greater levels of consistency [28].Supervision with targeted professional development opportunities supports treatment fidelity implementation by helping ensure a consistent delivery with minimal treatment drift form designed intervention protocol [1,29,30].In addition, planned professional development opportunities must be regularly assessed to maximise quality and suitability for the target recipients.Due to the often-multidimensional nature of introduced programs, the provision of training without quality indicators [31] runs a risk of being misdirected or ineffective.A further recurring theme within the literature related to the appreciation of the linkage between clinician fidelity and client outcomes.This not only allows identification of the elements of success within a program but also the detection of external factors that may exist within the treatment setting.External factors such as between-site differences and client variables have been found to significantly influence treatment success [32].Despite much attention to treatment fidelity measures in different guises and settings, the sustainability of these interventions has received scant attention.There is limited research undertaken in the area of measuring Treatment fidelity over time.Such an approach may help researchers best understand the longevity, strengths and weaknesses of the intervention.Borrelli and colleagues [3] shared this view, advocating measuring fidelity over time would deliver a better appreciation of treatment fidelity.Ratings from 0 to 3 were applied according to whether each aim was critiqued in comprehensive detail (rating of 3), in minor detail (rating of 2), in general terms (rating of 1) or not critiqued at all (rating of 0).Each study was coded against the six themes.This information was entered into the Synthesis Matrix with ratings applied according to the quality and relevance of review article appraisal within the six key themes.The majority of articles provided some evidence within each theme.The two assessors independently entered coding for the 32 articles into the Synthesis Matrix Database, with an average score being then being calculated for each item.Meta-evaluation procedures were used to help describe processes, direct discussion of criteria and synthesise the data [33].

Defining treatment fidelity
While the majority of articles (65%) provided detailed definitions of treatment fidelity, more than one third (35%) of reviews provided only brief or general descriptions (Figure 2).

The use of various treatment fidelity models
Most authors (29%) provided only a broad model outline or critiqued treatment fidelity Models only in general terms whilst 19% did not focus on Treatment fidelity Models at all in their evaluations/ reviews (Figure 3).

Key elements of treatment fidelity
A minority of authors (45%) focused in detail on the role of adherence scales while even fewer (26%) comprehensively critiqued the inclusion of competence scales for measuring treatment fidelity (Figure 4).

Clinician professional development during and post treatment
While more than one third (36%) of reviewed studies critiqued practitioner training during treatment in detail, 29% did not critique the role of mentoring and supervision within their examined studies at all (Figure 5).

Focusing on clinician or client treatment fidelity outcomes
The majority of authors (74%) placed a strong emphasis on measuring clinician treatment fidelity outcome, compared to only 42% emphasising the importance of measuring client treatment fidelity (Figure 6).

Measuring Treatment Fidelity over Time
Only one review [3] advocated measurement of Treatment fidelity of clinicians and client post treatment, to gain a greater appreciation of fidelity compared to 52% that did not critique at all.

Discussion
The current meta-evaluation found that researchers placed limited emphasis on provision of treatment fidelity definition or an overarching treatment fidelity model.It also found that researchers attached varying importance to strategies to enhance treatment fidelity in clinical practice such as ongoing clinician training, measurement of clinician and client fidelity, measurement of treatment fidelity over time and attempting to better understand the linkage between clinician fidelity to client outcomes.

Treatment fidelity definitions and models
Using a matrix synthesis and meta-evaluation, it is evident that there continues to be disparity amongst researchers in terms of the importance allocated to defining treatment fidelity.Such disparity will inevitably limit collective understanding within stakeholders including program designers, clinicians, clients and assessors.One reason for this variation may be the broad range of disciplines presently employing treatment fidelity to maximise effectiveness of delivered programs.Given the variations in treatment fidelity definitions a lack of consensus in terms of the key components and elements of the model was inevitable.The lack of a consistent framework and treatment fidelity Model thus perpetuates the differences between studies and approaches across disciplines.

Critiqued in comprehesive detail
Critiqued in minor detail

Critiqued in general terms
Did  domains for an effective treatment fidelity Model [3].Our study shows that, while some elements of this framework are valued by evaluators others are not yet routinely included within their scope.One such omission is the inclusion of practitioner training and measures of clinician fidelity.

Measures of clinician fidelity and professional development
Whilst there is strong research support for the importance of embedding adherence and competence scales [34,35], the results of the meta-evaluation did not reflect this high rating, with many articles choosing to emphasise adherence scales not competence scales [27,30,36].A number of articles mention the use of clinician and client self-assessments [1,3]; however these techniques have limitations in terms of impartiality and accuracy.Saneti and Kratochwill [37] found in their review that studies that incorporated treatment integrity tools with adequate psychometric properties (such as competence scales) had a more direct relationship between treatment fidelity than those that used indirect fidelity tools such as check lists or observational protocols.
Similarly, the measurement of treatment fidelity over time was not accorded high importance in the reviewed studies.Borelli and colleagues [3] was one of few research teams attempting to provide a greater appreciation of treatment fidelity through measurement over a specified time continuum.These researchers hypothesised improved understanding would provide a more accurate predictor of the success of a clinician or effectiveness of a program in delivering a consistent service, and identify key periods where follow up evaluation would offer maximum benefit.In addition, it would support evaluation of practitioner deviation and provide increased statistical power.
Given the mixed findings linked with the use of specific scales to measure the quality of clinician performance, it is not surprising that there was also limited value placed on clinician training.Multiple studies promote the inclusion of structured and ongoing training to clinicians and assessors to generate high fidelity levels [21,23,37].However, 43% (3 of 7 studies) of evaluations of the current review showed no evidence or only slight evidence of valuing the inclusion of provider training.The inclusion of specific professional development was given a low priority within many assessed articles [29,36].
In a number of the assessed studies, the absence of emphasis upon professional development suggested an expectation that clinicians and assessors are already experienced and skilled in methods that promote treatment fidelity.Yet, for program designers to expect clinicians and assessors who have varying levels of experience, post graduate qualifications and exposure to treatment fidelity to share a common understanding poses significant risks.Clinicians are expected to deliver a program in accordance with the provided theoretical and practical elements established by the designer [38,39].However, with so many moving parts contained within a program this expectation in the absence of meaningful support may be viewed as ambitious.
One of the identified constraints to inclusion of ongoing training is that this has resource implications for a given program.The inclusion of structured and ongoing professional development would need to be carefully managed to ensure other competing program components are not compromised [40].For example, the delivery of professional development could impact on budget and existing time allocations, and may not always be practical or feasible.The lack of specific training may also be reflective of an over reliance by designers upon the use of treatment manuals to generate a shared appreciation and direction for treatment fidelity.Bhar and Beck [41] hypothesised that adherence was linked directly to a clinician's ability to utilise the procedures specified within a treatment manual.Other researchers argue, however, that the key elements of treatment fidelity are multi-dimensional and require the inclusion of adherence scales, competence scales, structured professional development [2,11].

Link between clinician fidelity and client outcomes
This review finds that while some studies focussed on the link between clinician's fidelity and client outcomes, others did not.The exclusion of articles which focussed on trials of dosage adherence may have contributed to this variation as these articles would have focussed specifically on client outcomes.Several researchers report strong associations been treatment fidelity and participant treatment outcomes in effectiveness trials [5].Better understanding of this linkage enhances researchers' knowledge of the impact of external factors upon the implementation process [21,23].The identification of a number of significant external factors separate from, but impacting upon treatment fidelity implementation were noted during the metaevaluation process.External factors including patient variables or varying clinician responses to between-site differences can significantly influence outcomes [32].Firstly, influences that are not directly related to treatment fidelity including client responsiveness to the program, severity of client symptomology and therapeutic alliance [39] can influence outcomes.Secondly, other impacting variables may include quality of clinician facilitation and program design in terms of complexity [7].

Treatment fidelity over time
The majority of evaluators did not appear to attach value to the measuring of clinicians' treatment fidelity.This limited attention is reinforced further by the review results that highlighted evaluators had given a low priority to the role of clinician professional development and supervision.Only one review Borelli and colleagues [3] advocated for the measurement of treatment fidelity for clinicians and clients post treatment to gain a greater appreciation.Whilst the current study found limited progress was undertaken in this research area, such studies may help researchers to best understand the longevity, strengths and weaknesses of treatment fidelity.

Recommendations
In summary, the study identifies a need for standardisation.The standardisation process, however, must not add complexity.It must promote flexibility and portability to promote higher levels of fidelity when establishing, assessing, and reporting integrity [10].
Standardisation would be supported by adoption of a suite of treatment fidelity tools.This would allow development of a gold standard against which future programs can measure effectiveness.A renewed focus by authors to include a widely accepted treatment fidelity definition with a defined treatment fidelity Model and guidelines for subsequent monitoring and follow up would be preferable [8,30].An addendum to an agreed treatment fidelity structure would be a comprehensive list of both core and elective fidelity tools that in unison support the attainment of high levels of treatment fidelity within community-based and private settings.
The measurement of treatment fidelity would benefit from the inclusion of a treatment manual, therapist certification, fidelity ratings of treatment, supervision [42], adherence scales, competence scales and structured professional development opportunities.This comprehensive tool kit would support the facilitation and delivery of programs locally, regionally and internationally.

Limitations
Results from this study should be interpreted in conjunction with acknowledgement of two main limitations.Primarily, the inclusion and exclusion criteria may have been too restrictive.Perhaps including studies prior to a decade previously may have provided a more holistic view of the evolving area of treatment fidelity.In addition, a review of only systematic reviews and meta-analyses may have limited the findings whereas inclusion of randomised control trials measuring treatment fidelity may have provided a more heterogeneous field of review.

Conclusion
Treatment fidelity is an essential part of conducting psychosocial intervention research and clinical practice.This article reviewed the evidence for perceived benefits of high treatment fidelity, explored the current emphasis of researchers in the field, and identified strategies to enhance treatment fidelity in clinical practice.The meta-evaluation allowed a critical interpretive synthesis of pertinent literature reviews from the last decade to better understand current directions, theoretical and conceptual frameworks set forth by experts in this growing and complex field.In summary, this article has identified several recommendations for treatment fidelity in clinical practice including: development of standardised definitions of treatment fidelity within an overarching treatment fidelity model, incorporating measurement of clinician and client fidelity (through use of competence and adherence scales), and support for investment in professional development for clinicians.Areas of focus for future research include further examination of the linkage between clinician fidelity to client outcomes, and exploration of the value of measuring treatment fidelity overtime.
Stage three: Synthesising information into Matrix: The author developed a Synthesis Matrix, which incorporated the six distinct themes which had arisen in the course of the literature review: defining treatment fidelity, the use of various treatment fidelity Models, measurements of treatment fidelity, clinicians' professional development during and post treatment, clinician and client outcomes and measuring Treatment fidelity overtime.
Database search identified 3186 records 667 records excluded, not in English format 2519 records screened 190 full text article considered for inclusion 76 records excluded due to qualitative research, debates, and commentaries articles were excluded 114 full text articles reviewed for quality 26 Doubled up records were excluded 88 Treatment Fidelity reviews 2329 Records excluded due to:, non-review articles, non-peer reviewed papers, clinical trials, reviews not explicitly assessing Treatment Fidelity, reviews focused on trials of dosage adherence, letters and development of scales articles not supply sufficient prerequisite information about Treatment Fidelity.

Figure 1 :
Figure 1: Vetting process employed using The PRISMA Systematic Review Flow Diagram There have been attempts, however, to establish consistency in the field.The treatment fidelity Framework published in 2011 by the National Institutes of Health's Behavioral Change Consortium viewed study design, training, delivery, receipt and enactment as the key

Figure 2 :Figure 3 :Figure 4 :
Figure 2: Discussion of treatment fidelity definitionCritiqued in comprehesive detailCritiqued in minor detailCritiqued in general termsDid not critiqued at all

Figure 5 :
Figure 5: Importance of practitioner training during and post treatment.

Figure 6 :
Figure 6: Measurement of clinician and client treatment fidelity post experiment