Evaluation: What is needed in the field, and how can it be done??

Scott Leischow, Ph.D.
National Cancer Institute

State of the field

• At present, there are few standards for the evaluation of tobacco treatment quitlines
• The PHS Guidelines and Cochrane Reports both indicate that variability of quitline intervention research makes evaluation of efficacy more difficult
• Lack of consistency in quitlines research leads to lack of consistency in practice and in outcomes

You make the call…or not

• Scenario: A company called "Quality Quitline Services" offers to provide quitline services to all residents of a state based on a claim that they have achieved 15% quit rates in other states at a cost of $250 per person. Your state’s current quit service achieves 10% quit rates at $320 per person. Should the state change providers?

Not Necessarily!!

• The principle of "devil in the details" is paramount, and in large measure evaluation information can or should provide the answer. If evaluation information CANNOT answer this question, then the evaluation is inadequate.

Areas of Evaluation Emphasis

• Formative and Process Evaluation Approaches
• Summative Evaluation Approaches

Formative and Process Evaluation Approaches

• needs assessment determines who needs the program, how great the need is, and what might work to meet the need
• evaluability assessment determines whether an evaluation is feasible and how stakeholders can help shape its usefulness
• structured conceptualization helps stakeholders define the program or technology, the target population, and the possible outcomes
• implementation evaluation monitors the fidelity of the program or technology delivery
• process evaluation investigates the process of delivering the program or technology, including alternative delivery procedures

Formative Evaluation Strategies (samples)

• Develop a logic model for the service to assure that critical structures and functions are addressed
• Develop and maintain initial and ongoing community input (structured and unstructured)
• Develop SOPs so that procedures can be standardized
• Develop and implement QA procedures to assure that interventions are provided as intended, and so that data are accurately collected and entered into a database

Details work flow and procedures for all client types.
Enhances productivity and ensures consistency of service
Source: Powers, 2001, http://www.tepp.org/presentations/right.html

  [D]

Summative Evaluation Approaches

• outcome evaluations investigate whether the program or technology caused demonstrable effects on specifically defined target outcomes
• impact evaluation is broader and assesses the overall or net effects -- intended or unintended -- of the program or technology as a whole
• cost-effectiveness and cost-benefit analysis address questions of efficiency by standardizing outcomes in terms of their dollar costs and values

Summative Evaluation, cont

• Secondary analysis reexamines existing data to address new questions or use methods not previously employed
• Meta-analysis integrates the outcome estimates from multiple studies to arrive at an overall or summary judgement on an evaluation question
• Utilization (reach) and integration assess the degree to which the population recognizes, identifies and uses a resource, and the degree to which the resource becomes a fundamental component of relevant systems

Given the general categories of evaluation that exist, what evaluation standards exist for quitlines?

PHS Meta-analysis inclusion criterion

• Reported the results of a randomized, placebo/comparison controlled trial of a tobacco-use treatment intervention randomized on the patient level;
• Provided followup results at a timepoint at least 5 months after the quit date;
• Report published in a peer-reviewed journal;
• Report published between January 1, 1975 and January 1, 1999; and
• Report published in English.

PHS Conclusion

• Recommendation: Proactive telephone counseling, and group and individual counseling formats are effective and should be used in smoking cessation interventions. (Strength of Evidence = A)

Efficacy of Treatment Delivery Format (n = 58 studies)
Format	Odds Ratio (95%) CI	Estimated Abstinence Rate
No format (reference group)	1.0	10.8%
Self-help	1.2 (1.02-1.3)	12.3%
Proactive phone counseling	1.2 (1.1-1.4)	13.1%
Group counseling	1.3 (1.1-1.6	13.9%
Individual counseling	1.7 (1.4-2.0)	16.8%

Cochrane inclusion criterion

• Types of studies
• Randomised or quasi-randomised controlled trials, with unit of allocation individual participants, group, intervention site or geographical area.
• Types of participants
• Smokers or recent quitters. The definition of recent quitters was that used by the trial recruitment protocols, or by the participants themselves.

Cochrane inclusion criterion

• Types of intervention
• Provision of proactive or reactive telephone counseling to assist smoking cessation, to any population.
• Studies were excluded if the contribution of the telephone component could not be evaluated independently of face to face counseling. Studies which combined telephone counseling with self-help materials were included since the effect of self-help materials alone is limited.
• Types of outcome measures
• Smoking cessation at least 6 months after the start of intervention.

Cochrane Conclusions

• Cochrane:
• proactive telephone counseling helps smokers to quit
• Reactive counseling via telephone helplines has not been evaluated in the same way but indirect evidence suggests that callers receiving counseling via a quitline also have an increased chance of successfully quitting
• Telephone quitlines provide an important route of access to support for smokers

FDA Guidelines for Evaluating Efficacy of Medications

• 28 day assessment of efficacy
• Typically assessed between weeks 2-6 post-quit
• 1 year follow-up to assess outcomes
• Biochemical verification (e.g. breath CO)
• Provision of behavioral treatment (unspecified)
• Standardized baseline and follow-up questions
• Data collected and maintained to assure data quality
• All randomized subjects included in analysis (intent to treat)
• Ethical protection of participants
• Post-marketing assessment

Evaluation Decisions

• Prequit and Postquit data collection
• demographics
• tobacco use
• health problems
• degree of dependence
• intention and self-efficacy
• Intent to treat vs selective sample (e.g. quit for at least 24 hours)

Evaluation Decisions

• Assessing quitting and relapse prevention
• short term (e.g. weeks 2-6) and long term (26 wks minimum, 52 wks optimal)
• Set a quit date?
• Continuous abstinence, point prevalence (7 day)
• Handling dropouts (treat as relapsed is most conservative)

Evaluation Decisions

• Biochemical verification
• Random sample? (probably not necessary)
• Cost Effectiveness
• Intervention types (counseling, info only, etc)
• Include cost of media/advert.
• Assess cost for all callers and cost of successful quit
• Surveillance data to assess outcome and impact

Recommendations

• Develop and implement a standardized ‘bare minimum’ evaluation protocol/plan across quitlines that includes specific questions, assessment time frames, analytic methodologies, etc. that address the range of summative evaluation domains
• Can be supplemented as needed

Recommendations

• Develop a plan for on-going linkage across quitlines to maximize collaboration, raise standards, and allow for pooling of data. This will also require communities/states to partner with scientists. Subgroup of the quitline consortium??
• Modify surveillance instruments (if necessary) in order to collect information on quitline effectiveness/impact

Recommendations

• Build experimental and quasi-experimental studies into the quitlines in order to advance the field, and use modeling methodologies to analyze data from these ‘natural experiments’ (e.g. CISNET).
• Continued research is critically important to advance treatment (i.e. what works best for whom).