Include sources from attached paper Week 1 included in instructions
Order Description
The assignment includes a research paper and process mapping (Utube resource attached)
Include sources from attached paper Week 1 included in instructions
5 References – 2 of 5 attached
Original Article The Self-Assessment for Organizational Capacity Instrument for Evidence-Informed Health Policy: Preliminary Reliability and Validity of an Instrument Cristina Catallo, RN, PhD ?Souraya Sidani, RN, PhD
Keywords health policy, evidence-informed policy, policymakers, organizational capacity, instrument validation, reliability, validity
ABSTRACT Background: Health policymakers work in organizations that involve multiple competing de- mands and limited time to make decisions. In?uential international policy organizations continue to publish guidance and recommendations without the use of high-quality research evidence. Fewstudieshavedescribedtheprocesswithwhichgovernments,includinghealthministries,use evidencetosupporthealthpolicymakingdecisions.Researchisneededtobetterunderstandthe psychometric properties of instruments that assess health policy organizations? capacity to use research evidence. Aim: The purpose of this study was to assess the preliminary psychometric properties of an instrument which assesses organizational capacity for evidence use. Methods:TheinstrumentwasadministeredbytelephonesurveyfromJanuarytoJune2011using a purposeful sample of 57 Canadian health policymakers (policy analyst and senior management levels). Reliability of the instrument was assessed with Cronbach?s a coef?cient and item-to-total correlation for internal consistency; interitem coef?cients were used to identify particular item redundancy. Discriminant validity was assessed using the known-group comparison approach, with the independent sample t-test to assess subscale responses of policy analysts and senior managers. Findings: Cronbach?s a indicated acceptable internal consistency across its subscales. Discrim- inant validity analysis revealed a statistically signi?cant difference between policy analysts and senior managers for one subscale. Linking Evidence to Action: Our study provides a ?rst look at the Self-assessment for Orga- nizational Capacity Instrument?s psychometric properties and demonstrates that this instrument can be useful when evaluating government and other organizations? use of evidence to inform decision making. Further testing of this instrument is needed using large and varied samples of policymakers, from policy analysts to senior managers, across varied policymaking contexts. This instrumentcanbeastartingpointforgovernmentandrelatedorganizationstobetterunderstand how well it supports evidence use, including its acquisition, appraisal, and use in health policy decision making.
BACKGROUND The organizational environment of the health policymaker in- volves multiple competing demands and limited time to make decisionsthathavethepotentialtohavetremendousimpacton health care systems and practice. Because of these demands, policymakers need timely access to reliable, high-quality, syn- thesized research evidence (Lavis, 2006, 2009; Lavis, Lomas, Hamid, & Sewankambo, 2006). Policymakers and other deci- sion makers may recognize the need for research evidence to informhealthpolicy,butmaylackthetime,skills,orresources to?ndandapplytheevidencethroughoutthedecision-making
process (Jewell & Bero, 2008; Lavis, 2006; Lavis et al., 2006). International policy organizations, such as the World Health Organization,haverecommendedthatpolicymakingorganiza- tionsbuildsystemstosupportgreaterresearchuseinthedevel- opmentandreformofhealthpolicy(Lavis,Oxman,Moynihan, & Paulsen, 2008; World Health Organization, 2004). Health policymakers as well as other health care system decision makers, including nursing administrators, are expected to uti- lize research evidence to inform daily decisions. Considering two prominent international health policy organizations, the World Health Organization and the World Bank, authors have
Worldviews on Evidence-Based Nursing, 2014; 11:1, 35?45. 35 C 2013 Sigma Theta Tau International
Evidence-Informed Health Policy Instrument Assessment
identi?ed that high-quality research evidence is not consis- tently used in their published recommendations and guide- line statements (Hoffman, Lavis, & Bennet, 2009; Oxman, Lavis, & Fretheim, 2007). Making policy decisions with the absence of evidence can have deleterious effects, such as the implementation of programs and services that may not be effective in achieving desired results or distributed in a way that maximizes resources (Oxman, Vandvik, Lavis, Fretheim,&Lewin,2009).Theoreticalframeworksonresearch evidence use have started to consider the impact of organiza- tionalfeaturesontheadoptionofevidencefordecision.Organi- zationalstructure,valuesandbeliefs,resources,andleadership style are examples of organizational features (Beyer & Trice, 1982;Graham&Logan,2004;Kitson,Harvey,&McCormack, 1998). Despite these theories, little is known about the impact of the organization on evidence-informed health policy. Within governments and the health care organizations that support government, the capacity and resources needed for an organization to support evidence use is not clearly under- stood (Kothari, Edwards, Hamel, & Judd, 2009). Some orga- nizations have created tools to support capacity development for decision makers housed within the organization, such as the National Collaborating Centre?s Registry of Methods and Tools, which provides critically appraised methods and tools for public health managers and policymakers interested in implementing knowledge translation interventions (National Collaborating Centre for Methods and Tools, 2009; Peirson, Catallo, & Chera, 2013). Other organizations have focused on capacity development of an organization as a whole to uti- lizeevidence,suchastheCanadianFoundationforHealthcare Improvement?s (formerly the Canadian Health Services Re- search Foundation?s) self-assessment tool ?Is research work- ing for you?? (Canadian Foundation for Healthcare Improve- ment [CFHI], n.d.; Kothari et al., 2009; Thornhill, Judd, & Clements, 2009). While this tool examines how organizations acquire, assess, adapt, and apply evidence, it does not look at how the organizational environment supports the knowl- edge translation process, such as those activities that ?push? evidence out to users or ?pull? users to seek research from or- ganizations (Oxman et al., 2009). Likewise, an organization?s structures and processes can facilitate or inhibit the internal use of evidence. Speci?cally, organizational culture can im- pact how well individuals employed at these organizations can apply evidence to policy-relevant issues (Kothari et al., 2009; Oxmanetal.,2009).Whilesometoolsexisttohelpindividuals evaluate their ability to assess research utilization, few have looked at the processes and routines involving research evi- dence use within an organization (Kothari et al., 2009). One instrument that attempts to address these gaps is the Self- assessment of Organizational Capacity to Support the Use of Research Evidence to Inform Decisions Instrument (Oxman et al., 2009). This instrument was designed to aid organiza- tions in their assessment of capacity to use research evidence, and to help improve in areas where capacity is lacking. The content of the instrument is based on the CFHI?s (n.d.) tool
?Is research working for you?? and expanded upon to re?ect effortstolinkresearch-to-knowledgetranslationactivitiessuch as:push,userpull,exchange,andintegratedstrategies(Oxman etal.,2009).TheinstrumentispartoftheSUPPORTToolsfor evidence-informed health policymaking series by the Support- ing Policy relevant Reviews and Trials (SUPPORT) project, an international collaboration funded by the European Commis- sion?s 6th Framework (http://www.support-collaboration.org; Lavis,Oxman,Lewin,&Fretheim,2009).TheSelf-assessment ofOrganizationalCapacitytoSupporttheUseofResearchEvi- dencetoInformDecisionsInstrumentincludessevendomains related to supporting the use of evidence to inform decisions: organizational culture and values; setting priorities for obtain- ing research evidence; obtaining research evidence; assessing the quality and applicability of research evidence and inter- preting the results to inform priority decisions; using research evidence to inform recommendations and decisions; moni- toring and evaluating policies and programs; and supporting continuingprofessionaldevelopmentonevidence-basedtopics (Oxman et al., 2009). Limited research is available discussing the organizational capacity of ministries for evidence use in health policymaking. The instrument has been assessed for its content and revised accordingly after use in a variety of workshops with different groups; however, its psychometric properties have not been evaluated.
AIMS The purpose of this study was to provide a ?rst assessment of thepsychometricpropertiesoftheinstrumentSelf-assessment of Organizational Capacity to Support the Use of Research Evidence to Inform Decisions (thereafter referred to as the Self-assessment for Organizational Capacity Instrument). The speci?c objectives were to examine the internal consistency of the instrument?s subscales re?ecting the seven aspects of or- ganizational capacity for research evidence use, and to explore its discriminant validity using the known group comparison approach. Data were obtained from a sample (n=57) of Cana- dian health policymakers, including policy analysts and senior managers. It was hypothesized that policy analysts, who are responsible for incorporating evidence into documents, and senior decision makers who implement leadership mandates and are involved in strategic planning for the organization, would have different perspectives related to the organization?s capacity for evidence use. For each of the Self-assessment for OrganizationalCapacityInstrument?sdomains,weanticipated that senior managers would be more familiar with the or- ganization?s mission-supporting evidence use, processes for priority setting for priority use, the organization?s ability to use research evidence to inform decisions, and monitoring and evaluation of policies and programs when compared to policy analysts. Although very little is written about the func- tions of senior managers in government versus a more junior role such as the policy analyst, Oxman et al. (2007) indirectly describehowtheroleswithinagovernmentorganizationdiffer
36 Worldviews on Evidence-Based Nursing, 2014; 11:1, 35?45. C 2013 Sigma Theta Tau International
Original Article
with senior managers more involved with activities related to commissioning expert committees to review a health care is- sue and assimilate evidence to support recommendations. We hypothesized that policy analysts and senior managers would have similar perceptions of how well the organization obtains researchevidence,assessesresearchevidencequalityandappli- cability, and supports continuing education on evidence-based topics. Much has been written about policymakers? focus on the technical aspects of policymaking, often without the use of evidence. However, previous work has lacked the distinc- tion between the perceptions of senior management versus policy analysts. Jewell and Bero (2008) in their study of se- nior level policymakers and legislative of?cials identi?ed that there is limited ability to collect and evaluate research due to limited resources, a lack of skills to ?nd and appraise research evidence, and an overall incongruence between organizational mandates and the requirements of evidence-based practice.
METHODS Design A cross-sectional design was applied. The instrument was ad- ministered by a trained interviewer via telephone. Telephone interviews were conducted from January to June 2011 using a purposefulsampleofCanadianhealthpolicymakersrepresent- ing policy analysts and the senior managers.
Sampling and Procedures Toidentifypotentiallyeligibleparticipants,weadaptedthepro- cedures used by Wathen et al. (2009). A matrix was created to generate lists of potential policy analysts and senior decision makers in different health ministries in each province. The information was obtained from publicly available online or- ganizational charts and staff directories. Research assistants initiallycontactedpotentialparticipantsbye-mailtodetermine eligibility.Participantsweredeemedeligibleifthey:(a)worked in a health ministry policy unit or department or a related unit or department involved with health program or policy devel- opment, implementation, or analysis; (b) were employed in a role (e.g., policy analyst, consultant, epidemiologist) respon- sible for reviewing and using evidence to prepare reports for senior management, cabinet submissions to fund new pro- grams or policies, or summaries and analyses to guide policy decision making; (c) assumed the role of a senior manager or director responsible for a unit or department and the hir- ing of staff who would use evidence as part of their day-to-day functions; and (d) were able to commit to a 1-hour telephone interview. Individuals who were not eligible were asked to rec- ommendotherpersonsfromtheirdepartmentwhomightmeet the study criteria. Eligible participants were e-mailed a study invitation, consent form, and a copy of the Self-assessment for Organizational Capacity Instrument; they could use the copy to follow through during the telephone interview to facilitate the administration of the instrument. Research assistants con- tacted the respondents by e-mail or telephone to schedule an
interviewtime.Oneinterviewer(CC)conductedallinterviews. The interviewer used a standardized scripted version of the survey and read each question to the participant and recorded all responses. For any potentially eligible policymaker who didnotrespondtotheoriginale-mailinvitation,amultimethod approachinvolvinge-mailandtelephonecontactswasfollowed based on Dillman?s original Total Design method (Dillman, 1978). Intotal,57participantsmeteligibilitycriteria,enrolled,and completed the instrument. The obtained sample size was ad- equate for assessing the internal consistency of each subscale (containing four to six items) of the Self-assessment for Orga- nizationalCapacityInstrument,applyingtherecommendation ofhaving5?10peritem(Streiner&Norman,2003),andforde- tectingdifferencesingroups?meansofamediumsize,setting a at .05 and ? at .80 (Cohen, 1992).
INSTRUMENT The Self-assessment for Organizational Capacity Instrument contains seven subscales with items describing the organiza- tion?s: (a) culture and values supporting use of research evi- dence to inform decisions, (b) setting of priorities for obtain- ing research evidence, (c) ability to acquire research evidence to inform decisions, (d) capacity to assess quality and applica- bility of the research evidence and to interpret the results so thattheyinformprioritydecisions,(e)useofresearchevidence toinformrecommendationsanddecisions,(f)monitoringand evaluationofpoliciesandprograms,and(f)continuingprofes- sional development on evidence-based topics. The instrument uses a 6-point Likert scale with responses ranging from don?t know(0)tostronglyagree(5).Theinstrumentcontains36items for a total possible score ranging from 0 to 180. The number of items per subscale varies from 4 to 6, and the total subscale scores range between 20 and 30.
Analysis Descriptive statistics were conducted to examine the distribu- tion, measures of central tendency, and dispersion for individ- ual items, subscale scores, and total scale scores, as well as the participants? characteristics. Internal consistency reliability of the instrument?s subscales was assessed using the Cronbach?s a coef?cient and item-to-total correlation. Examination of in- teritem correlation coef?cients assisted in the identi?cation of item redundancy. For a newly developed instrument, the sub- scales are considered reliable if the Cronbach?s a coef?cient is >.70 and the item-to-total correlations are >.30 (de Vaus, 2002; Streiner & Norman, 2003). The correlations among the itemswithinasubscaleshouldrangebetween.30and.80;cor- relations greater than .80 imply item redundancy (Streiner & Norman, 2003). The known group comparison was applied to explore the discriminant validity of the instrument subscales. The mean subscale scores were compared for policy analysts and senior managers using the independent sample t test. Statistically
Worldviews on Evidence-Based Nursing, 2014; 11:1, 35?45. 37 C 2013 Sigma Theta Tau International
Evidence-Informed Health Policy Instrument Assessment
Table 1. Demographic Characteristics of Sample (N=57)
Civil Servant
Senior Manager
Total
Characteristics (n=36) (n=21) (N=57) Roleinorganization
Dataanalysis 10 1 11
Advicetoseniormanagement 4 2 6
Policyanalysis 19 0 19
Managementofstaff 1 17 18
Administrationofprograms 2 1 3
Yearsincurrentposition =1year 9 0 9 1?5years 18 18 36
6?10years 6 3 9
>10years 3 0 3
Education
Bachelor?sdegree 10 4 14
Master?sdegree 21 15 36
Doctorate 5 2 7
signi?cant differences provided evidence of validity. In addi- tion, the effect size was computed as the standardized groups? meandifferencetoexaminetheclinicalmeaningfulnessofthe difference between groups.
Ethics This study was approved by the research ethics board at Ryer- sonUniversity,Toronto,Canada.Eligibleparticipantsreceived a study information form and provided written, informed consent.
FINDINGS Demographics The sample consisted of 57 policymakers across 9 provinces; 36 participants worked in a policy analyst role and 21 as a senior manager or director. Table 1 outlines the main demo- graphic characteristics of participants. Most respondents were employed from 1 to 5 years in their current position and had obtained a graduate degree. There were no missing data, as participants responded to all items on the scale.
Descriptive The mean (SD) score for each item and subscale are presented inTable2.Forthesubscaleswithsixitems,thescoresobtained
ranged from 0 to 30 with a mid-point value of 18. Among the subscales with six items, the highest score was found for ?organizational culture and values that supports the use of re- searchevidencetoinformdecisions?(mean=22.81;SD=5.38) suggesting that on average, participants in their organization agreed that their organizational culture and values support the use of research evidence to inform decisions. Subscales with ?veitemshadscoresthatrangedfrom0to25withamid-point valueof15.Amongthesesubscales,thehighestscorewasfound fortheorganization?s?abilitytoobtainresearchevidencetoin- form decisions? (mean = 18.86; SD = 3.49) suggesting that on average, participants agreed that the organization is able to obtain research evidence for decision making. Finally for sub- scales with four items, the scores ranged from 0 to 20 with a mid-point of 12. The only question with four items was the or- ganization?sabilityto?assessthequalityandapplicabilityofthe researchevidenceandtointerprettheresultstoinformpriority decisions;? its mean score was 2.74 (SD = 3.99), which sug- gested that participants agreed slightly that their organization can carry out this function.
Internal Consistency Reliability Theitem-to-totalcorrelationsforitemswithineachsubscaleare found in Table 2. In general, they re?ected positive and mod- erateassociationsbetweentheitems?scoresandthecombined score on the remaining items comprising the respective sub- scales.Twoitemshadacorrelationcoef?cientlessthanthe.30 criterion. These items were: the organization?s access to pub- lications and to databases such as PubMed and the Cochrane Library (item-to-total correlation=.16); and the organization?s staffhasenoughtimeforcontinuingprofessionaldevelopment (item-to-total correlation=.26). A possible reason for the low item-to-total correlation is the low variability of scores across participants(mostselectedagreeorstronglyagree).Overall,the interitem correlations for items within their respective sub- scales were positive and of a moderate to high magnitude. The range of these correlation coef?cients was identi?ed for each subscale: organizational culture and values to support research evidence use (.33?.78); setting priorities to obtain re- search evidence (.50?.88); acquiring research evidence to in- form decisions (.32?.58); ability to assess quality and applica- bility of the research evidence and to interpret the results to inform priority decisions (.39?.75); use of evidence to inform decisions (.35?.66); monitoring and evaluation of policies and programs(.54?.85);andprofessionaldevelopmentonevidence- based topics (.36?.57). Three items had interitem correlations >.80 suggesting potential redundancy. To further determine redundancy,theCronbach?sacoef?cientfortherespectivesub- scales was computed with the redundant items included and then excluded from the analysis. When all of the items were included in the analysis, the subscales showed acceptable in- ternalconsistencyreliability.TheCronbach?s a coef?cientwas .81 for organizational culture and values to support research evidence use; .91 for setting priorities to obtain research evi- dence;.67forobtainingresearchevidencetoinformdecisions;
38 Worldviews on Evidence-Based Nursing, 2014; 11:1, 35?45. C 2013 Sigma Theta Tau International
Original Article
Table2.DescriptiveStatistics(Mean,StandardDeviation,andItem-TotalCorrelation)forEachSubscaleoftheInstrument(N=57)
StandardCorrectedItem-to SummaryofInstrumentSubscaleItemsMeanDeviationTotalCorrelation
Organizationalcultureandvaluessupporttheuseofresearch evidencetoinformdecisions(sixitems)
(a)Ourmissionorotherkeyorganizationaldocumentssupport evidence-informeddecisions
3.811.28.49
Totalpointsacrossitems=30 Totalscalescores:Mean=22.81, SD
=5.38
(b)Leadershipintheorganizationsupportsevidence-informed decisions
4.071.14.58
(c)Weareactivemembersinnetworksthatsupport evidence-informedpolicymakingoractivelyfollowthe developmentsandtheproductsofrelevantnetworks
3.681.36.31
(d)Wehaveregularmeetingswherehighlyrelevantresearch evidenceisdiscussedinrelationshiptodecisions
3.371.17.55
(e)Ourorganizationhascommittedresourcestoensurethat researchevidenceisusedtoinformdecisions
3.751.43.44
(f)Overall,ourorganizationalcultureandvaluessupporttheuseof researchevidencetoinformdecisions
4.12.98.67
Organization?sabilitytosetprioritiesforobtainingresearch evidence(?veitems)
(a)Wehaveexplicitcriteriaforsettingprioritiesforobtaining researchevidence
2.561.32.59
Totalpointsacrossitems=25 Totalscalescores:Mean=14.67, SD
=5.86
(b)Anappropriatemixofpeoplewithrelevanttypesofexpertise, responsibilities,andinterestsmakedecisionsaboutprioritiesfor obtainingresearch
3.531.21.69
(c)Wehaveanappropriateprocessforsettingprioritiesfor obtainingresearchevidencedynamically
2.811.44.62
(d)Wehaveappropriateprioritiesforobtainingresearchevidence2.861.42.52 (e)Overall,ourorganizationdoesagoodjobofsettingprioritiesfor obtainingresearchevidencetoinformdecisions 2.911.40.62
Organization?sabilitytoobtainresearchevidencetoinform decisions(?veitems)
(a)Wehaveskilledstafftosearchforandretrieveresearch evidence
4.09.96.40
Totalpointsacrossitems=25 Totalscalescores:Mean=18.86, SD
=3.49
(b)Ourstaffhaveenoughtime,incentiveandresourcesor arrangementswithexternalexpertsto?ndandobtainresearch evidence
3.001.21.43
Worldviews on Evidence-Based Nursing, 2014; 11:1, 35?45. 39 C 2013 Sigma Theta Tau International
Evidence-Informed Health Policy Instrument Assessment
Table2.Continued
StandardCorrecteditem-to SummaryofinstrumentsubscaleitemsMeandeviationtotalcorrelation
(c)WehavegoodaccesstodatabasessuchasPubMedandThe CochraneLibraryandpublicationsthatreportrelevantresearch
4.141.15.16
(d)Wehavegoodaccesstonational,provincialorlocalevidence thatweneedtoinformdecisions(e.g.,routinelycollecteddata, surveys,one-offstudies)
3.791.09.43
(e)Overall,ourorganizationdoesagoodjobofobtainingresearch evidencetoinformprioritydecisions
3.84.81.62
Organization?sabilitytoassessthequalityandapplicabilityofthe researchevidenceandtointerprettheresultstoinformpriority
(a)Wehaveskilledstafftoevaluatethequalityandapplicabilityof researchevidenceandinterprettheresults
3.581.17.56
decisions(fouritems) Totalpointsacrossitems=20 TotalScaleScores:Mean=12.74, SD
=3.99
(b)Ourstaffhaveenoughtime,incentive,andresourcesto evaluatethequalityandapplicabilityofresearchevidenceand interprettheresults
2.651.04.68
(c)Wehavearrangementswithexternalexpertstoevaluatethe qualityandapplicabilityofresearchevidenceandinterpretthe results
3.121.46.52
(d)Overall,ourorganizationdoesagoodjobofassessingthe qualityandapplicabilityofresearchevidenceandinterpreting theresultstoinformprioritydecisions
3.391.19.65
Organization?suseofresearchevidencetoinform recommendationsanddecisions(?veitems) Totalpointsacrossitems=25
(a)Ourstaffhavesuf?cienttime,expertise,andincentivetoensure appropriateuseofresearchevidencetoinform recommendationsanddecisions
3.181.08.49
Totalscalescores:Mean=17.39, SD
=4.06(b)Staffandappropriatestakeholdersknowhowandwhenthey cancontributeresearchevidencetoinformdecisionsandhow thatinformationwillbeused
3.211.61.74
(c)Ourorganizationensuresthatappropriatestakeholdersare involvedindecisionmakingandthattheyhaveaccessto relevantresearchevidence
3.681.07.44
(d)Whatevidencewasusedandhowitwasusedistransparentin ourdecisions
3.681.10.54
40 Worldviews on Evidence-Based Nursing, 2014; 11:1, 35?45. C 2013 Sigma Theta Tau International
Original Article
Table2.Continued
StandardCorrecteditem-to SummaryofinstrumentsubscaleitemsMeandeviationtotalcorrelation
(e)Overall,ourorganizationdoesagoodjobofusingresearch evidencetoinformrecommendationsanddecisions
3.631.08.62
Organization?sabilitytomonitorandevaluatepoliciesand(a)Weroutinelyconsidertheneedformonitoringandevaluation3.841.19.44 programs(?veitems) Totalpointsacrossitems=25 (b)Ourstaffhaveenoughexpertiseoradequatearrangementswith externalexpertsformonitoringandevaluation 3.211.26.49 Totalscalescores:Mean=16.61, SD =5.60(c)Ourstaffhavetheincentiveandresourcestoconductor commissionmonitoringandevaluation 3.091.29.54 (d)Ourorganizationensuresthatappropriatestakeholdersare involvedindecisionsaboutmonitoringandevaluation 3.321.35.54 (e)Overall,ourorganizationdoesagoodjobofmonitoringand evaluationofpoliciesandprograms 3.161.36.61 Organization?sabilitytosupportcontinuingprofessional developmentthataddressesevidence-basedtopics(sixitems) (a)Ourstaffhaveenoughtimeforcontinuingprofessional development 3.531.03.26 Totalpointsacrossitems=30 Totalscalescores:Mean=17.68, SD =5.24 (b)Wehaveroutinestoensurethatourstaffcontinuetodevelop appropriateskillsforobtaining,appraising,andapplying researchevidence 2.821.25.56 (c)Ourstaffprioritizecontinuingprofessionaldevelopment activitiesthatare?evidence-based?(i.e.,withcontentthatis basedonresearchevidenceandusingcontinuingprofessional developmentmethodsthatarebasedonresearchevidence) 2.771.53.45 (d)Wehaveappropriateroutinesforprioritizinginternal professionalcontinuingdevelopmentactivitiesthat accommodatetheneedsofbothnewandlong-termstaff 2.861.31.42 (e)Wehaveappropriateroutinesfordecidingwhethertosupport participationinexternalcontinuingprofessionaldevelopment activitiesthataccommodatetheneedsofbothnewand long-termstaff 2.611.36.50 (f)Overall,ourorganizationdoesagoodjobofsupporting continuingprofessionaldevelopmentthataddressesimportant topicsandisevidence-based 3.091.21.54
Worldviews on Evidence-Based Nursing, 2014; 11:1, 35?45. 41 C 2013 Sigma Theta Tau International
Evidence-Informed Health Policy Instrument Assessment
.82 for ability to assess quality and applicability of the research evidence and to interpret the results to inform priority deci- sions; .78 for use of evidence to inform decisions; .91 for mon- itoring and evaluation of policies and programs; and .76 for professional development on evidence-based topics. When po- tentially redundant items (i.e., those which had high interitem correlations) were removed from the respective subscales, the value of the Cronbach?s a coef?cient decreased slightly to .80 for ?setting priorities to obtain research evidence,? .78 for ?monitoring and evaluation of policies and programs,? and .60 for the ?organization?s ability to assess quality and appli- cability of the research evidence and to interpret the results to inform priority decisions? subscales. The reduced values of the Cronbach?s a coef?cient did not support exclusion of re- dundantitems.Further,omittingtheitemswoulddecreasethe number of items comprising the respective subscales, which could pose challenges in how well the subscale operational- izes the aspect of organizational capacity for research evidence use. Accordingly, the items were retained until their perfor- mance can be evaluated in future research with larger sample sizes.
Discriminant Validity Table 3 presents the results of the independent sample t test and the effect sizes comparing the responses of policy ana- lysts and senior managers. A statistically signi?cant difference was found for the Organization?s Ability to Monitor and Eval- uate Policies and Programs subscale. Policy analysts (mean= 16.83, SD = 5.74) had a lower mean score on this subscale than senior managers (mean=18.57, SD=3.29). Effect sizes ranged from -.10 to .16, indicating a small difference in the subscales? mean scores for the two groups. Unsolicited com- ments made by participants during the telephone interviews provided some explanation for these ?ndings. A few senior managers commented that they were not responsible for ac- cessingorcritiquingevidence;rather,theyindicatedthatthese activities fell under the skill set and responsibility of the pol- icy analysts. Some policy analysts commented that they were comfortable describing the culture and process for evidence use in their department, but struggled to describe whether or not their observations were similar at the broader organiza- tional level as they were not involved in higher level manage- ment discussion. Overall, the results did not show signi?cant differences in responses between senior managers and policy analysts.
DISCUSSION This study offers the ?rst examination of the psychometric properties of the Self-assessment for Organizational Capacity Instrument. Evidence-informed decision making is an impor- tant issue for consideration among health policymakers and we address a current gap by evaluating an instrument that assesses capacity for evidence use among health policymak- ers. By comparing policy analysts with senior managers, we
Table3.DiscriminantValidityResultsUsingIndependentSamplet-TestComparingPolicyAnalystsandSeniorManagersandEffectSize
PolicySeniorIndependentIndependentEffect InstrumentAnalystManagert-Testtt-TestSig.Size SubscaleMean(SD)Mean(SD)Value(df)(2-Tailed)Cohen?sd Organizationalcultureandvaluessupportstheuseofresearchevidencetoinformdecisions(sixitems)22.16(5.60)23.90(4.90)-1.18(55).24-.32 Organization?sabilitytosetprioritiesforobtainingresearchevidence(?veitems)14.27(6.42)15.33(4.83)-.65(55).51-.17 Organization?sabilitytoobtainresearchevidencetoinformdecisions(?veitems)18.72(3.96)19.09(2.54)-.38(55).70-.10 Organization?sabilitytoassessthequalityandapplicabilityoftheresearchevidenceandtointerpretthe resultstoinformprioritydecisions(fouritems) 12.94(3.94)12.38(4.12).51(55).61.14 Organization?suseofresearchevidencetoinformrecommendationsanddecisions(?veitems)17.63(4.22)16.95(3.81).61(55).54.16 Organization?sabilitytomonitorandevaluatepoliciesandprograms(?veitems)15.47(6.35)18.57(3.29)-2.42(54.45).01a-.56 Organization?sabilitytosupportcontinuingprofessionaldevelopmentthataddressesevidence-basedtopics (sixitems) 16.83(5.74)19.14(3.94)-1.62(55).10-.44
aIndicatesstatisticallysigni?cant.
42 Worldviews on Evidence-Based Nursing, 2014; 11:1, 35?45. C 2013 Sigma Theta Tau International
Original Article
sought to better understand the evidence needs that inform policy decisions between these two groups. Due to the com- plex nature of the policy environment, these results suggested that both policy analysts and senior managers may not only have different needs for information, but also varied ability to critiqueandexaminethequalityoftheevidence.Althoughfur- ther study is needed, these results suggest that it may be more importantforthepolicyanalysttohaveastrongunderstanding of how to search for and critically examine research to distin- guish good quality evidence from poor quality so that it can be better used to support policy decisions. However, what the policy analyst may struggle with is the application of research results to policy issues, including how the evidence impacts or serves the needs of the larger organization?the government. Seniormanagers,asopposedtopolicyanalysts,aremorelikely tobefamiliarwiththeperspectiveoftheorganization,andmay be an important link to integrating the evidence at a broader systems level. Hence, it may not be as important for senior managers working in government to know how to critically appraise a research article, but rather it is important for them to possess the skills which enable them to consider the im- pact of the evidence in relation to the larger workings of the government, a realm to which the policy analyst may not be exposed.
Limitations of This Study Despite its potential to inform evidence-informed decision making,thisstudyhaslimitationsthatwarrantfutureresearch. Limitations of this study include the small pool of potentially eligible participants and the apparent similarity in their expe- riences and perceptions, which resulted in restricted range in their responses. Due to the sample size, more sophisticated psychometric analysis, such as factor analysis, was not possi- ble. One challenge that we encountered was that participants had dif?culty securing time to participate in the study due to their rapidly changing work commitments. If an urgent is- sue came up for a senior manager, it would not always be possible to carry out a survey at the scheduled time. As a result, researchers need to be ?exible when scheduling in- terview times with those in government environments. De- spite these issues, our sample size was adequate to carry out this preliminary assessment of the instrument?s psychomet- ric properties. Ongoing evidence of reliability and validity us- ing a larger and more diverse sample of policymakers is re- quired so that more complex psychometric evaluation can be performed. The study results also contribute to an emergent under- standingofhoworganizationssupportresearchevidenceuseto informpolicydecisionmaking.The?ndingsprovideinitialevi- denceoftheinstrumentreliabilityandvalidity.Theitemscom- prising each subscale were internally consistent; they measure therespectiveaspectsoforganizationalcapacityforresearchev- idenceusewithminimalerror.Anareaforongoingevaluation is the assessment of potential item redundancy. Three items were found to be redundant but were retained because remov-
ingthemreducedthevalueoftheCronbach?s a coef?cientand the number of items comprising the subscales. Whether or not these items should be retained in the instrument requires additional investigation using larger, more varied samples of policymakers at the policy analyst and senior management levels. Results of the known group comparison showed a statis- tically signi?cant difference in the policy analysts? and senior managers? perceptions of how well the government organiza- tion evaluates and monitors policies. However, no other sub- scale?s scores differed between policy analysts and managers. Furthermore, the magnitude of the differences between the two groups of participants was small. These ?ndings did not support the hypotheses, and may be related to the characteris- ticsoftheinstrumentorthesample.Thecontentoftheinstru- mentmaynotbesensitiveenoughtodiscriminateresponsesof senior managers and policy analysts. The sample represented seniormanagersandpolicyanalystsworkingingovernmentor related organizations; these institutions may have comparable capacityforevidenceuseormayascribesimilarresponsibilities to the two groups of participants. Using an instrument to assess organizational capacity for evidenceuseamonghealthpolicymakerscanhaveuniquechal- lenges. One challenge includes the potential for bias among respondents. Because government is often scrutinized by the media and must remain accountable to the public, those who workingovernmentmaynotbeabletoanswertheinstrument questions without apprehension. While for this study, respon- dents felt that they could answer truthfully, they did express some uncertainty in responding for the entire organization. These preliminary ?ndings warrant further study. During the telephone survey, some participants expressed concerns that they did not want to make the ?government look bad,? and debated whether to select a more favorable response. Policy analysts,whosawthemselvesasthe?frontline?workersofpol- icy development, had a good understanding of the operations and activities within their departments, but sometimes found it dif?cult to answer instrument items when considering the operations of the entire ministry. These respondents indicated thatseniormanagerswerebetterabletoansweronbehalfofthe organization as a whole because they were involved in higher level decision making than the policy analysts. This ?nding is supported by Birken, Lee, and Weiner (2012), who found that managers in health care organizations oversee and have in?uence over implementation of innovations such as those that support research use within the organization. Likewise, senior managers have in?uence on frontline workers, and a lack of support for an innovation, such as evidence use, could impact whether or not it was carried out (Birken et al., 2012). These issues have implications for the instrument that po- larized responses can introduce error and impact the overall psychometric assessment of the instrument. Additional stud- ies with a larger and more varied sample can help evaluate the overallintegrityoftheitemsandassesswhetheranyadditional changes to the instrument need to be made.
Worldviews on Evidence-Based Nursing, 2014; 11:1, 35?45. 43 C 2013 Sigma Theta Tau International
Evidence-Informed Health Policy Instrument Assessment
IMPLICATIONS FOR RESEARCH AND POLICY The Self-assessment for Organizational Capacity Instrument is relevant for policymaking organizations, including health and related ministries, to use as a way to assess their organiza- tional ability to access, appraise, and utilize research evidence to inform decisions. With increasing government emphasis on demonstrating accountability for decisions, funding alloca- tions, and the distribution of services, the need to explore how well health ministries (and government as a whole) support evidence use is an important consideration for demonstrating accountability. More research is needed to better understand the organizational capacity for evidence use across health and related ministries. From this study, a next phase of research could examine the types of resources that health policy envi- ronmentsneedtosupportevidence-informeddecisionmaking, and the facilitators and barriers to evidence use in policy envi- ronments. By identifying areas of strength and for continued efforts, health ministries can better understand how research use occurs within the organization, and establish supports to facilitateimprovedevidence-informedhealthpolicy.Thisstudy provides a ?rst look at the Self-assessment for Organizational Capacity Instrument?s psychometric properties and demon- stratesthatthisinstrumentcanbeusefulwhenevaluatinggov- ernment and other organizations? use of evidence to inform decision making. Due to our sample size, we must consider these results cautiously and remember that ongoing testing of this instrument is needed. Future psychometric evaluation of thispromisinginstrumentrequiresalargerandvariedsample ofpolicymakersfromhealthandrelatedministries.Thesample should also include both policy analysts and senior managers across varied policymaking contexts so that a robust assess- ment can be made for how well this instrument distinguishes between different groups involved in policy decision making. Factoranalysisshouldalsobeincorporatedintothepsychome- tric evaluation of this instrument to identify item redundancy and consistency across the measure.
CONCLUSIONS The Self-assessment for Organizational Capacity Instrument demonstrates acceptable internal consistency and discrimi- nant validity in this preliminary assessment of its psychome- tric properties. Use of this instrument can be a starting point forgovernmentandrelatedorganizationstobetterunderstand how well it supports evidence use, including its acquisition, appraisal, and use in health policy decision making. WVN
LINKING EVIDENCE TO ACTION
? Policy statements need to be based on high- quality research evidence.
? TheSelf-AssessmentforOrganizationalCapacity InstrumentforEvidence-InformedHealthPolicy is a promising scale that can aid organizations in their assessment of capacity to use research evidence. ? Further research with this tool that includes larger samples is important to further establish its validity and reliability.
Author information CristinaCatallo,AssociateProfessor,DaphneCockwellSchool of Nursing, Ryerson University, Toronto, Ontario, Canada; Souraya Sidani, Professor, Daphne Cockwell School of Nurs- ing, Ryerson University, Toronto, Ontario, Canada; Canada Research Chair in Design and Evaluation of Health Interven- tions. Address correspondence to Dr. Cristina Catallo, Daphne Cockwell School of Nursing, Ryerson University, 350 Victo- ria Street, POD 458B, Toronto, Ontario, Canada M5B 2K3; ccatallo@ryerson.ca
Accepted 17 July 2013 Copyright C 2013, Sigma Theta Tau International
References Beyer, J. M., & Trice, H. M. (1982). The utilization process: A conceptual framework and synthesis of empirical ?ndings. Ad- ministrative Science Quarterly, 27(4), 591?622. Birken,S.A.,Lee,S.D.,&Weiner,B.J.(2012).Uncoveringmiddle managers?roleinhealthcareinnovationimplementation.Imple- mentation Science, 7(28). doi:10.1186/1748-5908-7-28. Canadian Foundation for Healthcare Improvement (formerly The Canadian Health Services Research Foundation ). (n.d.). Is researchworkingforyou?Aself-assessmenttoolanddiscussionguide for health services management and policy organizations. Retrieved from: http://www.cfhi-fcass.ca/PublicationsAndResources/ ResourcesandTools/SelfAssessmentTool.aspx Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155?159. de Vaus, D. (2002).Surveys in social research (5th ed., p.184), New South Wales, Australia: Routledge. Dillman, D. (1978). Mail and telephone surveys: The total design method. Hoboken, NJ: John Wiley & Sons, Inc. Graham,I.D.,&Logan,J.(2004).Innovationsinknowledgetrans- fer and continuity of care. Canadian Journal of Nursing Research, 36(2), 89?103. Hoffman, S. J., Lavis, J. N., & Bennett, S. (2009). The use of re- search evidence in two international organizations? recommen- dations about health systems. Healthcare Policy, 5(1), 66?86. Jewell, C. J., & Bero, L. A. (2008). Developing good taste in evidence: Facilitators of and hindrances to evidence-informed health policymaking in state government. The Milbank Quar- terly, 86(2), 177?208.
44 Worldviews on Evidence-Based Nursing, 2014; 11:1, 35?45. C 2013 Sigma Theta Tau International
Original Article
Kitson, A., Harvey, G., & McCormack, B. (1998). Enabling the implementation of evidence based practice: A con- ceptual framework. Quality in Health Care, 7, 149?158. doi:10.1136/qshc.7.3.149 Kothari,A.,Edwards,N.,Hamel,N.,&Judd,M.(2009).Isresearch working for you? Validating a tool to examine the capacity of health organizations to use research. Implementation Science, 4, 46, 1?9. doi:10.1186/1748-5908-4-46. Lavis,J.N.(2006).Research,publicpolicymaking,andknowledge- translationprocesses:Canadianeffortstobuildbridges.TheJour- nalofContinuingEducationintheHealthProfessions,26(1),37?45. Lavis, J. N. (2009). How can we support the use of systematic reviews in policymaking? PLOS Medicine, 6(11): e1000141. doi: 10.1371/journal.pmed.1000141 Lavis, J. N., Lomas, J., Hamid, M., & Sewankambo, N. K. (2006). Assessingcountryleveleffortstolinkresearchtoaction.Bulletin of the World Health Organization, 84, 620?628. Lavis, J. N., Oxman, A. D., Lewin, S., & Fretheim, A. (2009). SUPPORT Tools for evidence-informed health policymaking (STP). Health Research Policy and Systems, 7(Suppl. 1), I1, 1?7. doi:10.1186/1478-4505-7-S1-I1. Lavis, J. N., Oxman, A. D., Moynihan, R., & Paulsen, E. J. (2008). Evidence-informed health policy 1: Synthesis of ?ndings from a multi-method study of organizations that support the use of research evidence. Implementation Science, 3, 53, 1?7. doi: 10.1186/1748-5908-3-53 National Collaborating Centre for Methods and Tools. (2009). The Registry of methods and tools. Retrieved from: http://www.nccmt.ca/registry/index-eng.html Oxman, A. D., Lavis, J. N., & Fretheim, A. (2007). The use of evidence in WHO recommendations. Lancet, 369, 1883?1889.
Oxman, A. D., Vandvik, P. O., Lavis, J. N., Fretheim, A., & Lewin, S. (2009). SUPPORT Tools for evidence-informed health poli- cymaking (STP) 2: Improving how your organisation supports theuseofresearchevidencetoinformpolicymaking.Health Re- searchPolicyandSystems,7(Suppl.1),S2,1?10.doi:10.1186/1478- 4505-7-S1-S2 Peirson, L., Catallo, C., & Chera, S. (2013). The registry of knowledge translation methods and tools: A resource to support evidence-informed public health. International Jour- nal of Public Health, 58(4), 493?500. doi:10.1007/s00038-013- 0448-3 Streiner,D.L.,&Norman,G.R.(2003).Healthmeasurementscales: A practical guide to their development and use (3rd ed.). New York, NY: Oxford University Press. Thornhill,J.,Judd,M.,&Clements,D.(2009).CHSRFknowledge transfer: (Re)introducing the Self-Assessment Tool that is help- ing decision-makers assess their organization?s capacity to use research. Healthcare Quarterly, 12(1), 22?24. Wathen, C. N., Tanaka, M., Catallo, C., Lebne, A. C., Friedman, M. K., Hanson, M. D., … McMaster IPV Education Research Team. (2009). Are clinicians being prepared to care for abused women? A survey of helath professional education in Ontario, Canada. BMC Medical Education, 9, 34, 1?11. doi:10.1186/1472- 6920-9-34 World Health Organization. (2004). The Mexico statement on health research: knowledge for better health: Strengthening health systems. Geneva, Switzerland: Author.
doi 10.1111/wvn.12018 WVN 2014;11:35?45
Worldviews on Evidence-Based Nursing, 2014; 11:1, 35?45. 45 C 2013 Sigma Theta Tau International
Copyright of Worldviews on Evidence-Based Nursing is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder’s express written permission. However, users may print, download, or email articles for individual use.
PLACE THIS ORDER OR A SIMILAR ORDER WITH NURSING TERM PAPERS TODAY AND GET AN AMAZING DISCOUNT
