Running head: VALUE-BASED APPROACH TO EVALUATION

A VALUE-BASED APPROACH TO SMOKING PREVENTION WITH IMMIGRANTS FROM LATIN AMERICA: PROGRAM EVALUATION

Isaac Prilleltensky, Geoffrey Nelson, and Laura Sanchez Valdes

Wilfrid Laurier University

Key words: smoking, prevention, values, children, youth, community-based approach, immigrants, outcome evaluation, process evaluation, qualitative methods, quantitative methods

Abstract

The article describes the findings of outcome and process evaluations of a value-based approach to smoking prevention. The program was conducted with a community of immigrants from Latin America living in Canada. The approach is based on participatory community planning, sensitivity to cultural diversity, and on a comprehensive philosophy of health. Qualitative and quantitative methods were used to assess outcomes and processes. Quantitative measures indicate that when compared to a control group, program children of ages 8 to 14 improved their knowledge about some aspects of tobacco use and effects, engaged in more community activism, and showed a decrease in intentions to smoke. While improvements in community activism and intention to smoke were statistically significant, gains in knowledge only approximated statistical significance. No statistical differences were noted between comparison and intervention groups in the areas of social skills and family communication. The qualitative evaluation indicated that parents, children, and staff were highly satisfied with the impact of the program as well as with the processes employed to deliver it. Implications for improving our program and for value-based evaluations in general are discussed.

A VALUE-BASED APPROACH TO SMOKING PREVENTION WITH IMMIGRANTS

FROM LATIN AMERICA: PROGRAM EVALUATION

INTRODUCTION

Without a clear articulation of the values guiding our social programs, we risk losing track of fundamental moral principles. The daily pressures of community programs are such that, without an explicit commitment to moral values, there is a high chance of sacrificing principles for expediency. While many practitioners and researchers hope for community involvement, for instance, time and financial pressures prevent them from using participatory approaches. For community programs to remain loyal to a set of principles, it is crucial that their value-base be made explicit to all concerned (Prilleltensky, 1997; Prilleltensky & Nelson, 1997; Prilleltensky, Peirson, & Nelson, 1997; Prilleltensky, Peirson, Gould, & Nelson, 1997). With that consideration in mind, we tried to make our values central to the planning and evaluation process.

The purpose of this article is to describe the findings of process and outcome evaluations of a value-based approach to smoking prevention. A review of existing prevention projects in the province of Ontario revealed a gap in services to multicultural communities in general, and to children in these groups in particular (North York Public Health Unit, 1994). While many projects to reduce or prevent tobacco use were available for the population at large, the group was not aware of programs designed to address the needs of children in the Latin American community. This prompted the Latin American Educational Group (LAEG) of Kitchener, Ontario, to submit a proposal to Health Canada for a smoking prevention project.

The program and evaluation were based on three principal values: participatory community planning, sensitivity to cultural diversity, and holistic philosophy of health. Table 1 summarizes the expression of these values in our program evaluation. In general, our planning and evaluation philosophy is informed by participatory action research. In this approach, which has its roots in Latin America (Yeich, 1996), participants are involved in formulating and conducting all aspects of the research, including developing the evaluation questions, designing research instruments, selecting a sample, collecting data, and analyzing the information (Serrano-Garcia, 1990). Control and ownership over the evaluation process is shared by researchers and community stakeholders (Barnsley & Ellis, 1992; Papineau & Kiely, 1996). Whitmore (1991) has argued that participatory action researchers assume that knowledge is a source of power. Thus, marginalized people can experience both personal and political empowerment in participatory research. In addition to the participatory nature of this approach, there is a high level of commitment to using the data to bring about social change (Patton, 1997). This approach to evaluation is designed to promote the empowerment of the participants and to foster community development at the same time (Kroeker, 1996). In our case, as can be seen in Table 1, members of the local community were involved in all the phases of the evaluation. Various stakeholder groups, including staff, volunteers, parents, and professionals from academic and public health settings, collaborated in formulating questions, gathering and analyzing information, and composing a final report. We were aware of the importance of mobilizing community resources in advancing the cause of smoking prevention. Like many authors in the literature, we support the view that community-based approaches are essential in reducing the use of tobacco among children and youth ((Biglan et al., 1996; Cummings & Coogan, 1992; Goodman & Wandersman, 1994; Harachi, Ayers, Hawkins, Catalano, & Cushing, 1996; Hernandez & Lucero, 1996; Johnson et al., 1990; Kaftarian & Hansen, 1994; Lichtenstein, Lopez, Glasgow, Gilbert-McRae, & Hall, 1996; Papineau & Kiely, 1996; Sowers, Garcia, & Seitz, 1996; St. Pierre, Kaltreider, Mark, & Saikin, 1992; Vicary, Doebler, Bridger, Gurgevich, & Deike, 1996).

The second value guiding our planning and evaluation was sensitivity to cultural diversity. It has been persuasively argued that prevention programs should address the specific circumstances of each ethno-cultural group (Botvin, Dusenbury, Baker, James-Ortiz, & Kerner, 1989; Botvin et al., 1992; Schinke, Moncher, Holden, Botvin, & Orlandi, 1989). Hernandez and Lucero (1996) claimed that it "seems imperative that prevention curricula be culturally adaptable to the target populations it services...Clients are more willing to fully participate when the program is designed and structured to incorporate their values, beliefs, and traditions" (p. 270).

Furthermore, Orlandi (1992) stated that "the development and evaluation of community-based programs that can reach ethnic/racial subgroups within our society require expertise in two broad areas: program evaluation competence and cultural competence" (p. 12). Our evaluation was meant to capture the specific experience of Latin American children and families participating in the program. Therefore, we had to meet Orlandi's requirements for both evaluation and cultural competence. These requirements were met by bringing together professionals and local residents from Latin America in a planning and evaluation committee. The committee consisted of a public health official and an academic community psychology researcher, both of whom with experience in planning and evaluation; and an educator and a person with background in management. Together, these four people came from three Latin American countries which would be represented in the program. The committee worked closely with the community in planning and revising the program and evaluation tools. The planning and evaluation committee availed itself to information from previous local research (Prilleltensky, 1993) and from other sources related to the stressors facing immigrant families (Casas, 1992; Padilla & Salgado de Snyder, 1992). Factors such as voluntary versus involuntary immigration, differential process of acculturation in children and parents, language barriers, and cultural shock were taken into account when planning the program as well as the evaluation. The challenge for us was to develop methods and instruments that would reflect the culture of the our community. With that goal in mind, we formulated procedures and tools that were conversational and non-threatening, two features that appealed to the community.

The third principle guiding our program and evaluation was a comprehensive philosophy of health. It is abundantly clear from the literature that smoking is the result of an interaction of multiple factors, ranging from peer pressure, to exposure in the family, to commercial exploitation (Bailey, Ennett, & Ringwalt, 1993; Hine, Summers, Tilleczek, & Lewko, 1997; Hooked on tobacco: The teen epidemic, 1995; Marin, Marin, Perez-Stable, Otero-Sabogal, & Sabogal, 1990; Oei & Baldwin, 1992; Miller & Slap, 1989; Stanton, Mahalski, McGee, & Silva, 1993; Wallack, 1985). A holistic philosophy of health requires that we intervene at multiple levels of influence (Federal, Provincial and Territorial Advisory Committee on Population Health, 1996; Hamilton & Bhatti, 1996, World Health Organization, 1986). Hence, we tried to have an impact on personal, family, and social factors. Our evaluation, accordingly, sought to assess changes not just in psychological dimensions of smoking, such as beliefs and attitudes, but also in family communication, media literacy, and community activism.

We believe that the value-based and participatory action nature of our planning and evaluation process are worth sharing. We describe briefly the main components of the program, followed by the findings of the process and outcome evaluations. We conclude with a discussion of the program's contributions to value-based evaluation and smoking prevention.

BRIEF PROGRAM DESCRIPTION

The smoking prevention program, funded by Health Canada, was carried out by the Latin American Educational Group of Kitchener, Ontario, in collaboration with Waterloo Region Community Health Department and Wilfrid Laurier University. The project took place between 1995 and 1997.

About fifty children of ages 4 to 14 attended the program, which took place during the evening at the local community centre. The program run in conjunction with a Spanish school established by the Latin American Educational Group. The program consisted of psychoeducational sessions for children and parents. The project had five specific objectives: (a) to expand knowledge on the effects of smoking, (b) to help teens resist the pressure to smoke, (c) to increase family communication and parental input regarding smoking, (d) to engage children and parents in community activism to prevent the use of tobacco, and (e) to reduce the intentions to smoke in the future. In order to meet these objectives we conducted twelve educational sessions with children and youth and 8 educational sessions with parents.

The sessions with children were organized as follows: Two sessions were devoted to learning factual information about smoking and its effects (objective #1); four sessions were devoted to the topics of assertiveness, decision making, stress and anxiety, and self-esteem (objective #2); two sessions were devoted to the topic of communication skills (objective #3); and four sessions were devoted to learning and engaging in social action (objective #4).

There were four sessions exclusively for parents. Another four sessions were spent working with children on objective 4: community activism. About seven parents attended sessions regularly. About 20 parents attended the final celebration session. In order to enhance parents' knowledge of tobacco effects (objective #1), one session was devoted to addictive behaviors and tobacco's harmful effects. Parents learned statistical information about morbidity and smoking prevalence rates in Canada in the different age groups. In order to enhance family communication (objective #3), 3 sessions dealt with parenting skills. Finally, in order to enhance community activism (objective #4), parents helped children design pamphlets and art work about the harmful effects of tobacco. A week later they accompanied children to a shopping mall to distribute the pamphlets and exhibit their work. The last session devoted to community action was a celebration at the end of the project.

PROGRAM EVALUATION

The program was subjected to outcome and process evaluations. The outcome evaluation consisted of both quantitative and qualitative data, while the process evaluation consisted of qualitative data only. We present first the outcome evaluation, followed by the process evaluation.

Outcome Evaluation

Quantitative Findings

In order to measure the impact of the tobacco intervention program we compared the intervention group on pre- and post-intervention measures with a comparison group from the Latin-American community. The comparison group completed the questionnaires but did not receive the intervention. Although children in the 4 - 7 year old age group and parents completed pre- and post-intervention measures, the former group did not have reliable data due to their young age and the sample size for the parents was too small for statistical analyses. Consequently, the analyses we present relate to the 8 - 14 year old age group.

Sample

There were 37 children in the comparison group (20 boys and 17 girls) who completed the pre-test questionnaire and 27 children in the comparison group (15 boys and 12 girls) who completed the post-test questionnaire. The average age of the children in the comparison group was 11.5 years at the pre-test. There were 23 children in the intervention group (14 boys and 9 girls) who completed the pre-test questionnaire and 26 children in the intervention group (19 boys and 7 girls) who completed the post-test questionnaire. The average age of the children in the intervention group was 10.4 years at the pre-test.

Measures

We developed questionnaires in Spanish and English to measure the impact the program had on the main objectives of the program. As there were no questionnaires readily available in Spanish to measure the objectives our program pursued, we had to construct our own. As will be shown below, not all the questionnaires exhibited sufficient internal consistency.

The same questionnaire was completed by the children prior to the intervention and soon after the last session of the program. The questionnaire included measures to tap each of the outcome goals of the project: (a) knowledge about smoking (nine items), (b) social skills related to peer pressure (nine items), (c) relationship with family (six items), (d) participation in the community (five items), and (e) intention to smoke when aged 17 (one item).

Findings

To analyse the data, mixed model analyses of variance (ANOVAs) were used. In each case, group was the between group independent variable with two levels (intervention vs. comparison) and time was the within group independent variable with two levels (pre vs. post). The dependent variables were the items and/or scales from the questionnaire. A significant group by time interaction would indicate positive changes in the outcome measures for the intervention group compared with the comparison group.

Since Cronbach=s alpha was found to be very low for items measuring knowledge about smoking (outcome objective #1), the items were analysed individually, rather than using a summary score. The scores for each item are presented in Table 2. The mean scores represent the proportion of children who answered each question correctly. As is shown in Table 2, responses to the nine items are highly variable. For instance, very few children know the correct answers to items 1 and 5 (either at pre-test or post-test), while most know the correct answers to items 3, 4, and 9 (either at pre-test or post-test). Only two items show a group by time interaction effect which approaches significance. Children in the intervention group improved marginally on items 1, F(1, 49) = 3.74, p < .06, and 3, F(1, 48) = 3.12, p < .10, relative to the comparison group. The low alpha and the high degree of variability in the items for this scale suggest the need to search for a more reliable measure of smoking knowledge. Also, the lack of significant effects on most of the knowledge items suggests that future interventions of this type should tie the curriculum more closely to the items in the outcome measure.

The mean scores on the measure of community participation (outcome objective #4) are presented in Table 3. There are five items, which are rated on a three-point scale from "never" (1) to "many times" (3). Cronbach=s alpha was computed and was found to be .84 at pre-test and .90 at post-test. In view of the high level of reliability of the items, a summary score was computed (ranging from 5 to 15). There was a significant group by time interaction on this scale, F(1, 46) = 5.26, p < .05. The intervention group showed an increase in their participation in the community, while the comparison group showed a decrease. The items reflected participation in anti-smoking activities and other altruistic activities in the community, either on an individual basis or with one=s parents.

Children were asked to rate their intention to smoke daily when they are about 17 years of age (outcome objective #5) on a four-point scale, ranging from "for sure yes" (1), "probably yes" (2), "probably no" (3), to "for sure no" (4). The mean scores for the two groups are displayed in Table 3. Both groups scored in the "no" range at both the pre-test and the post-test on this item. However, there was a significant group by time interaction, F(1, 41) = 4.58, p < .05. The intervention group showed an increase in their intention not to smoke, while the comparison group showed a slight decrease on this measure. Thus, the program had a positive impact on children=s self-reported intention not to smoke at age 17.

Cronbach=s alpha was computed for the nine items designed to measure social skills and peer pressure (outcome objective #2) and was found to be low. Therefore, each item was analysed separately. There were no significant group by time interactions on any of the items.

The mean scores on the measure of relationship with family (outcome objective #3) are also presented in Table 3. There are six items in this scale, each of which is rated on a three-point scale from "never" (1) to "many times" (3). Cronbach=s alpha was found to be .87 at pre-test and .78 at post-test. A summary score (ranging from six to 18) was computed in view of the high level of reliability. The group by time interaction effect was not significant, indicating that the program did not have an impact on children=s relationships with their families.

Qualitative Findings

We asked participants to what extent, in their view, the program met its goals of improving knowledge, social skills, family communication and community activism. This portion of the evaluation had three groups of respondents: children, parents, and program staff. We report children's and adults' perspectives separately.

Children's perspectives

Method. There were 12 program participants (6 boys and 6 girls; ages 6 to 12) who volunteered to answer questions in a semi-structured interview. The questions addressed the four main objectives of the program. In addition, the interview asked participants questions related to the process evaluation, which we report in the next section. In this section we concentrate on the perceived impacts of the program. Interviewers wrote children's answers verbatim as much as possible. Detailed notes were taken about their answers. The answers were later typed and analysed by two researchers. The main themes and illustrative quotes related to each program objective were identified and used for this analysis. Table 4 presents a summary of children's qualitative responses regarding program impact. The table shows children's typical responses according to program objectives.

Knowledge about smoking and its effects (objective #1). The 12 children interviewed indicated that they learned something about smoking. They mentioned that smoking is harmful to one's health and that it's difficult to say no when invited to smoke. Some of the children's responses are as follows:

If you smoke you can get a heart attack and you can't breath. If a woman is pregnant and smokes, the baby can die (7 year old girl).

Tobacco causes cancer, attacks your heart, damages your lungs, make your teeth yellow, causes breathing problems, bad breath. Reduces the taste and smell of foods (11 year old girl).

Life and social skills (objective #2). Ten out of the 12 children indicated that they learned something about themselves. The children talked about looking after their health, improving their self-esteem, and getting to know themselves better. The 12 children reported to have learned something about how to improve peer relationships. Their learning had to do with team work, listening skills, communication, participation, assertiveness, and tolerance for frustration. They also learned how to avoid fighting and how to choose good friends. Some of their comments were as follows:

I learned that if we win or lose it doesn't matter. The important thing is to play (7 year old girl).

I learned to be patient with my friends, I learned to work in a group (8 year old girl).

Family communication (objective #3). Nine of the twelve children interviewed reported talking to their parents about tobacco since the beginning of the program. The other three children said they did not talk with their parents about smoking. The main topics of conversation with their parents were tobacco's harmful effects to the body and the environment, and how to say no to smoking. This is what some children reported:

I talked to my dad and told him not to smoke because it's bad for his heart (6 year old boy).

We talked about the fact that smoking is not good, that we can die from it because it's not good for the body and because it pollutes the environment (11 year old girl).

Children expressed satisfaction with respect to their conversations with parents about smoking. They thought their parents liked the program and appreciated talking about health with their children.

Community activism (objective #4). Some of the community action initiatives were signing and delivering a petition to city hall, children giving talks about smoking in their own schools, doing a play for the community, and preparing and posting anti-smoking placards in a shopping mall. Most children enjoyed the community experience, learned from it, and let the community know of their efforts to fight smoking. Children expressed their experience of community activism as follows:

It's good to tell people smoking is bad for them (7 year old girl).

It was nice to participate in community projects. I'd like to do it again (11 year old girl).

Adults' Perspectives

Method. Parents and staff offered opinions about the program. Four mothers who participated in the program were interviewed for this component of the evaluation. A semi-structured interview was used to ask parents about the positive and/or negative impacts the program had on parents, children, families, and community. Interviewers wrote parents' responses verbatim as much as possible. The answers were later typed and analysed by two researchers. Seven facilitators (all female) and the four members of the planning and evaluation committee (1 female, 3 male) also participated in this portion of the evaluation. The program coordinator, who was a member of the planning and evaluation committee as well as a facilitator, participated in the planning and evaluation focus group. A semi-structured focus group guide was used in two separate focus groups. One focus group was with program facilitators. The second one was with the planning and evaluation committee. The guide asked about the positive and/or negative impacts of the program on staff, children, families, and community. The focus group guide was the same as the interview guide used with parents. The only difference was that the focus group guide asked about the positive or negative impact of the program on staff as opposed to parents.

Both focus groups were taped and transcribed verbatim. They were later analysed by two researchers. The main themes and illustrative quotes related to positive and negative impacts at each level of analysis (parent, child, family, community) were identified and used for this analysis. The findings are based on both the interviews and focus groups. Table 5 presents a summary of adult's perspectives on program impact.

Impact of program on parents. Three of the four parents interviewed reported personal gains. They reported having learned important facts about tobacco and having acquired tools to help their children to cope with the threat of smoking. The following statements indicate their satisfaction with the program:

Having scientific knowledge about the harmful effects of tobacco is helpful. It helps guide our children better. (objective #1)

I appreciated the importance of parents working together with community organizations to prevent the use of tobacco.(objective #4)

Impact of program on facilitators and staff. Program staff identified several gains to themselves as part of their participation in the project. Their responses fit nicely into the objectives of the program. The gains made relate to knowledge about smoking, life and social skills, family communication, and community activism.

Staff reported improved knowledge in a number of areas. The first area is factual information about smoking and its effects. A woman facilitator said: I have now more concrete facts. Now I know how to explain to my children. I had a lot of myths about smoking and tobacco.

Although most facilitators had previous experience as teachers or community organizers, they felt the training they received improved their facilitation skills. Facilitators reported improvement in life and social skills as well. They learned how to work well in a team, how to facilitate others' participation, how to tolerate differences, how to assert themselves, and how to communicate their ideas to adults and children. Facilitators and program personnel felt their participation enhanced their own family communication. They learned how to get closer to their children. A mother said: it was nice to work here because I became involved in a program my daughter attended. This way I got closer to her.

Staff could readily identify benefits to the community at large. A better understanding of the various Latin American cultures, as well as a sense of enhanced solidarity in the community were commonly reported. There is a sense of community, of solidarity. We share the work and the duties, said a mother facilitator.

Impact of program on children. The mothers thought the program had a positive impact on their children. According to them children learned important facts about the harmful effects of tobacco. In addition, they said the program reinforced knowledge imparted at home. These positive effects relate to objective #1 of the program, that is, to improve knowledge about tobacco and smoking. Parents also noted children's enhanced ability to cope with pressure to smoke (objective #2).

Staff reported many beneficial effects for children. Their comments fall into the areas of knowledge, life and social skills, family communication, and community appreciation. Several comments were made about children's enhanced appreciation of their community. Children learned how to work in groups and how to contribute to their own community. A staff person said that children are starting to feel proud as Spanish-Canadians. They also felt proud of the activities we carried out in the program.

Impact of program on family. The four parents agreed the program had a positive family impact. The program was seen as contributing to family communication related to smoking. Parents reported a more open attitude on their own behalf and on behalf of other parents who smoked. This is how one parent put it:

I think that at the family level the program had an impact in that children show their father the problems with smoking and he promised them to quit smoking, even though he hasn't done it yet.

Impact of program on the community. Parents appreciated the fact that their children's friends were also talking about smoking and its effects. A parent put it this way:

I like the fact that community children know about the harmful effects of smoking. This is important because they are my child's friends and they won't offer him cigarettes.

Program personnel could identify more community gains than the parents. Staff talked about the strengthening of the Latin American community in general. The project increased the visibility and helped establish the Latin American Educational Group, the group in charge of conducting the program. There was a sense of pride among members of the Spanish speaking community. The project helped to improve the sense of community among Spanish-speaking immigrants in the neighbourhood. Of similar importance was the fact that part-time employment was created for several community residents.

Process Evaluation

The goal of the process evaluation was to learn what factors and dynamics helped or hindered the successful implementation of the program. We conducted an ongoing as well as a final process evaluation. Table 6 presents a summary of stakeholders' views affecting the process.

Method

The ongoing evaluation consisted of special time allocated during project management team meetings to problem solve as we went along. A mid-program open questionnaire was administered to program staff and volunteers to express their opinions concerning the process undertaken to run the program. Nine people answered the questionnaire. Meetings of the planning and evaluation committee were also used to evaluate progress and to problem solve. A final process evaluation took the form of interviews and focus groups. The same groups consulted for the qualitative outcome evaluation were asked to comment on the process. These were a selected group of 12 children who participated in the program, four parents, and program staff. The same methods used for the qualitative outcome evaluation were utilized for this portion of the evaluation. We present below the results of the ongoing as well as the final process evaluation with respect to factors that facilitated and inhibited the process. Given their distinct opinions concerning the process, we present the answers given by the three stakeholder groups separately.

Children's Perspectives

Facilitative factors. Children liked three aspects of the program very much: (a) activities, (b) themes, and (c) snacks. The activities children liked were movies, role plays, games, colouring, and puppets. A nine year old boy said that he liked the movies, the snacks, the role playing and the colouring. A twelve year old boy said that he liked the movie about peer influence. The themes children liked were self-esteem, stress, communication, decision-making, health, and smoking myths and realities. Six of the twelve children interviewed said they liked everything about the program.

Inhibiting factors. Six children observed they did not like reading and some of the presentations by teachers. When the teacher talked too much I got bored said a six year old boy. Some children also mentioned they had to work hard and had to wait for the snack. Another limiting factor noted by some children was the small size of one of the classrooms where the program was run. Although most classes were adequate, there was one which was very small. A final impediment reported by children was the lack of discipline exhibited by some participants.

Parents' Perspectives

Facilitative factors. The main factor identified by parents was effective and dynamic teaching. Parents praised the facilitators' pedagogical techniques. The facilitators were thought to be dynamic and enthusiastic. The teachers knew how to motivate the parents said one parent.

A second factor contributing to the process was building on the strengths of the existing Spanish school. The Spanish school, which predated the tobacco program, was an important launching base for the prevention initiative. Parents knew and trusted the staff who run the Spanish school. Building the prevention program into the Spanish school was a key ingredient of success. Without this link, it would have been harder to recruit and retain participants.

A third facilitative factor was the staff's human relations skills. Program staff was considered highly skilled in human relations. This helped avert and resolve conflicts and contributed to a positive climate.

Inhibiting factors. Parents identified two factors as detracting from the main objectives of the program. The first factor was lack of parental involvement. Only very few parents attended the sessions for parents. On average, only seven parents attended regularly. More effort could have been invested in attracting parental participation. Parents suggested making it known that the program addressed not only tobacco issues but family communication and parenting issues as well. Some thought that parents who smoke may have felt defensive about attending such program. The second factor interfering with program success was lack of discipline. In the future, better discipline strategies would have to be considered.

Program Staff's Perspectives

Facilitative factors. Program staff identified four key facilitative factors which contributed to the perceived success of the program: (a) effective planning, (b) conflict resolution, (c) community participation, and (d) popular education methodology.

Effective planning of the program and the specific sessions was regarded as a key ingredient in the smooth operation of the project. Facilitators talked about the importance of the three day training workshop and about their weekly planning meetings. A person had this to say about the training:

I congratulate the people who facilitated the training workshop. It was a new experience for me because I didn't know this type of work...I think I did my job well.

Facilitators drew on each other's strengths to plan the program. A facilitator said:

We all planned and worked together. We were united and developed a common understanding of what needed to be done. We learned to communicate with ease.

The second facilitative factor was constructive conflict resolution. A source of conflict was running the tobacco prevention program in conjunction with the volunteer-run Spanish school. Whenever conflicts were identified, people brought them to the attention of the parties involved and efforts were made to resolve them quickly. Community participation in the planning and execution of the program was regarded as a key factor in obtaining community support. The community was consulted during the proposal stage about their needs and priorities. Another round of consultation occurred when funding came through. An effort was made to employ community members for the various positions of coordinator, assistant coordinator, facilitators, evaluators, and baby sitters. A staff member said:

We developed in the facilitators and in the community a sense of ownership. People could identify with the program because they had a chance to express their ideas.

Another staff person said it was good the community developed the program and based it on its own people....and it resorted to the local community experts.

Popular education methodology was viewed as an excellent tool in motivating children and parents to participate. The essence of this method is the valuing of personal experience in the learning process. Everyone's experiences are regarded as valuable for the group. Participants can relate the material learned to their personal lives. This experiential approach proved highly successful.

Inhibiting factors. Two main factors were thought to interfere with the achievement of program goals: (a) lack of parental participation, and (b) lack of discipline.

Pretty early in the implementation of the program it became obvious that fewer parents than anticipated would attend the parenting sessions. Although parental input was sought at the beginning of the program, and many parents liked the idea of a prevention project, when it came time to commit themselves only a handful of five to seven parents would attend the sessions. Efforts were intensified to attract more parents but with very limited success. Parents were used to sending their children to the Spanish school and there were no expectations for them to stay there. It seems as though parents continued in the same mode of sending the children and using the time for other things.

Some participants in the evaluation focus groups speculated that parents thought the program dealt exclusively with smoking, whereas in fact it had a much broader scope. Parents may not have liked the idea of committing themselves to weeks of discussions about smoking, but perhaps they would have attended if it was better explained to them that the focus was on parenting and family communication in general. It is possible that many of the parents who smoked felt defensive about the program. They may have felt comfortable sending their children but not attending themselves, lest they would be chastised for smoking.

Misbehavior on the part of some children presented a problem. With about 50 children attending the various groups, it was nearly inevitable that some behavior problems would be encountered. In response to discipline problems the facilitators developed clear guidelines for children and parents. Disruptive behavior subsided but was not completely eliminated. In retrospect, it would have been helpful to have involved parents from the outset in setting clear rules and expectations for acceptable behavior of children during the sessions.

DISCUSSION

In this section, we focus on (a) the process and outcome findings of this evaluation in the context of previous literature, and (b) our observations of the planning and evaluation process in terms of the values which guided our work.

The Evaluation Findings

"A number of evaluation researchers have argued that the best approach is one of 'methodological pluralism,' in which both qualitative and quantitative methods are employed in any evaluation." (Pancer, 1997, p. 64). As evaluators, we endorse the methodological pluralism discussed by Pancer. The program was evaluated by using multiple methods and by consulting multiple stakeholders. Quantitative and qualitative data, gathered before, during, and after the intervention, were employed for outcome and process evaluations. Open and closed questionnaires, interviews, and focus groups were used to obtain information about the process and the impact of the program. These multiple methods helped us assess in a comprehensive fashion the positive and negative aspects of the program. The combination of both evaluation techniques, quantitative and qualitative, helps us focus on potential deficiencies and build on strengths (Goering & Streiner, 1996; Pancer, 1997). On one hand, the quantitative evaluation tells us that we should take the enthusiastic praise from the qualitative evaluation with caution. On the other hand, the qualitative evaluation tells us that we should not be too despondent about not achieving statistically significant differences between intervention and control groups in all areas. The methodological pluralism employed in our program is not frequently reported in the smoking prevention literature. The different methods give us useful information to improve the program in the future.

The lack of statistical differences between the intervention and the control groups with regards to social skills and family communication suggest that the program needs to sharpen its focus in these areas. Another possibility is that the questionnaires were not sensitive enough to address the lessons taught in the program. In the future, it would be advisable to examine the questionnaires carefully to make sure that they measure what was taught during the intervention. We should compare more carefully our social skills intervention with others which have achieved significant results in this area (Botvin, Dusenbury, Baker, James-Ortiz, & Kerner, 1989; Botvin et al., 1992). We would need to refine both the intervention and the questionnaires in the field of social competence. The same observation applies to dissemination of information, an area in which only minor statistical gains were noted in our intervention group.

The positive quantitative and qualitative findings regarding levels of community activism following the program are very encouraging. There are several reports in the literature of successful community-wide efforts to prevent tobacco use in youth (Biglan et al., 1996; Cummings & Coogan, 1992; Feighery, Altman, & Shaffer; 1991; Harachi, Ayers, Hawkins, Catalano, & Cushing, 1996; Jason, Ji, Anes, & Birkhead, 1991; Johnson et al., 1990; Kaftarian & Hansen, 1994). However, there are only few family oriented programs (Hernandez & Lucero, 1996). We believe our program is quite unique in that it combines family with community interventions. Our program brought together children and parents in preparing and delivering anti-tobacco materials to politicians, merchants, and the public at large. Although our efforts in these areas were admittedly modest, we believe they bear replication at a larger scale. The positive evaluation results support this conclusion.

The Planning and Evaluation Process

We believe that one of novelties of this particular research and development project is the demonstration of a value-based approach to planning and evaluation. Prilleltensky and Nelson (1997) argue that values, implicit or explicit, always shape program planning and evaluation. In this project, we consciously focused on three primary values: (a) participatory community planning and evaluation, (b) sensitivity to cultural diversity, and (c) a comprehensive philosophy of health.

Consistent with a participatory action approach to planning and evaluation, we considered it important that all stakeholders be represented in the evaluation (Barnsley & Ellis, 1992; Papineau & Kiely, 1996, Serrano-Garcia, 1990). Each group that we consulted -- children, parents, staff, and volunteers -- offered its unique perspective on the program. This approach is congruent with our overall philosophy of participatory evaluation. Had we not consulted the children, we would not have become aware that some teachers were lecturing too much, or that kids had to wait too long sometimes to get their snack. Similarly, had we not consulted staff, we would have missed the rich experience facilitators took away with them.

Congruent with participatory action research, our evaluation process had an important action component (Kroeker, 1996). We are currently in the process of disseminating the project to other ethno-cultural communities. Evaluation results informed our suggestions to other groups who are interested in implementing similar initiatives.

The second value that guided this project was sensitivity to cultural diversity. In this regard, it was very important that the principal investigator of the project was born and raised in Latin America. The matching of the background of the investigator and the participants meant that they shared a common language and some common cultural experiences and values. A person from outside of this community would need to go through a much more prolonged period of entry to gain the trust of this community. The principal investigator was able to serve an important bridging function between the Latin American community, a federal granting agency, and academic resources for evaluation (Prilleltensky & Nelson, 1997).

The participatory nature of the planning and the evaluation that was employed is a familiar and comfortable one for the Latin American community. Also, project staff from the Latin American community were recruited, hired, and trained to plan, implement, and evaluate the project. The program materials and program sessions were conducted in the parents= and children=s native language, and the setting was a Spanish language class, all of which reinforced the cultural identity and pride of the Latin American community.

Orlandi (1992) stated that "the number of individuals who are bicompetent in the areas of program evaluation competence and cultural competence must be increased" (p. 18). We implemented a viable model of increasing bicultural competence by having a bicompetent investigator train local residents in various aspects of evaluation research. Members of the Latin American community learned about designing, conducting, and analysing program evaluation. The last value which anchored this project was a comprehensive philosophy of health. While many health promotion and prevention projects focus rather narrowly on the individual, sometimes in the context of his/her family and other micro settings (Prilleltensky & Nelson, 1997), we enlarged the scope of smoking prevention to address social and community factors as well. We believe that our approach to the prevention of smoking is novel in its emphasis on community and political action. Consistent with a critical approach to education, students and parents were encouraged to raise their awareness of the connection between health, smoking, advertising tobacco products, and the profit-driven motivation of cigarette companies. Interestingly, it was on this outcome indicator that the intervention showed the most clear impact. Making these connections between the personal choice to smoke (or not smoke) and the politics of the tobacco industry is an important practical component of a comprehensive philosophy of health. In this way, participants are not rendered impotent by global market forces and seductive advertising that glamorize smoking (Hooked on tobacco: The teen epidemic, 1995), but can become active agents of social change.

REFERENCES

Bailey, S. L., Ennett, S. T., & Ringwalt, C. (1993). Potential mediators, moderators, or independent effects in the relationship between parents' former and current cigarette use and their children's cigarette use. Addictive Behaviors, 18, 601-621.

Barnsley, J., & Ellis, D. (1992). Research for change: Participatory action research for community groups. Vancouver, B. C.: The Women=s Research Centre.

Biglan, A., Ary, D., Koehn, V., Levings, D., Smith, S., Wright, Z., James, L., & Henderson, J. (1996). Mobilizing positive reinforcement in communities to reduce youth access to tobacco. American Journal of Community Psychology, 24, 625-638.

Botvin, G. J., Dusenbury, L., Baker, E., James-Ortiz, S., Botvin, E. M., & Kerner, J. (1992). Smoking prevention among urban minority youth: Assessing effects on outcome and mediating variables. Health Psychology, 11, 290-299.

Botvin, G. J., Dusenbury, L., Baker, E., James-Ortiz, S., & Kerner, J. (1989). A skills training approach to smoking prevention among Hispanic youth. Journal of Behavioral Medicine, 12, 279-296.

Casas, J. M. (1992). A culturally sensitive model for evaluating alcohol and other drug abuse prevention programs: A Hispanic perspective. In M. A. Orlandi (Ed.), Cultural competence for evaluators: A guide for alcohol and other drug abuse prevention practitioners working with ethnic/racial communities (pp. 75-116). Rockville, MD: U.S. Department of Health and Human Services, Office of Substance Abuse Prevention.

Cummings, K. M., & Coogan, K. (1992). Organizing communities to prevent the sale of tobacco products to minors. International Quarterly of Community Health Education, 13(1), 77-86.

Federal, Provincial and Territorial Advisory Committee on Population Health (1996). Report on the health of Canadians. Ottawa: Publications, Health Canada.

Feighery, E., Altman, D. G., & Shaffer, G. (1991). The effects of combining education and enforcement to reduce tobacco sales to minors. Journal of the American Medical Association, 266, 3168-3171.

Freire, P. (1971). Pedagogia del oprimido [Pedagogy of the oppressed][Spanish]. Buenos Aires: Siglo XXI.

Freire, P. (1975). Cultural action for freedom. Harvard Educational Review Monograph, 1.

Goering, P. N., & Streiner, D. L. (1996). Reconcilable differences: The marriage of qualitative and quantitative methods. Canadian Journal of Psychiatry, 41, 491-497.

Goodman, R. M., & Wandersman, A. (1994). FORECAST: A formative approach to evaluating community coalitions and community-based initiatives. Journal of Community Psychology, CSAP special issue, 6-25.

Hamilton, N., & Bhatti, (1996). Population health promotion: An integrated model of population health and health promotion. Ottawa: Health Canada, Health Promotion Development Division.

Harachi, T. W., Ayers, C. D., Hawkins, D. J., Catalano, R. F., & Cushing, J. (1996). Empowering communities to prevent adolescent substance abuse: Process evaluation results from a risk- and protection focused community mobilization effort. Journal of Primary Prevention, 16, 233-254.

Hernandez, L. P., & Lucero, E. (1996). DAYS La Familia Community Drug and Alcohol Prevention Program: Family-Centered Model for working with Inner-City Hispanic Families. Journal of Primary Prevention, 16, 255-272.

Hine, D. W., Summers, C., Tilleczek, K., & Lewko, J. (1997). Expectancies and mental models as determinants of adolescents' smoking decisions. Journal of Social Issues, 53, 35-52.

Hooked on tobacco: The teen epidemic. (1995, March). Consumer Reports, pp. 142-147.

Jason, L. A., Ji, P. Y., Anes, M. D., & Birkhead, S. H. (1991). Active enforcement of cigarette control laws in the prevention of cigarette sales to minors. Journal of the American Medical Association, 266, 3159-3161.

Johnson, C. A., Pentz, M. A., Weber, M. D., Dwyer, J. H., Baer, N., MacKinnon, D. P., & Hansen, W. B. (1990). Relative effectiveness of comprehensive community programming for drug abuse prevention with high-risk and low-risk adolescents. Journal of Consulting and Clinical Psychology, 58(4), 447-456.

Kaftarian, S. J., & Hansen, W. B. (Eds.) (1994). Community partnership program -- Center for Substance Abuse Prevention [Special issue]. Journal of Community Psychology, CSAP special issue.

Kroeker, C. J. (1996). The cooperative movement in Nicaragua: Empowerment and accompaniment of severely disadvantaged peasants. Journal of Social Issues, 52, 123-138.

Lichtenstein, E., Lopez, K., Glasgow, R. E., Gilbert-McRae, S., & Hall, R. (1996). Effectiveness of a consultation intervention to promote tobacco control policies in Northwest Indian Tribes: Integrating experimental evaluation and service delivery. American Journal of Community Psychology, 24, 639-656.

Marin, B. V., Marin, G., Perez-Stable, E. J., Otero-Sabogal, R., Sabogal, F. (1990). Cultural differences in attitudes toward smoking: Developing messages using the theory of reasoned action. Journal of Applied Social Psychology, 20, 478-493.

Miller, S. K., & Slap, G. B. (1989). Adolescent smoking: A review of prevalence and prevention. Journal of Adolescent Health Care, 10, 129-135.

North York Public Health Department (1994). Directory of Tobacco Use Prevention Activities and Contacts Within Public Health Units in Ontario. North York, Ontario: Author.

Oei, T. P. S., & Baldwin, A. R. (1992). Smoking education and prevention: A developmental model. Journal of Drug Education, 22(2), 155-181.

Orlandi, M. (1992). The challenge of evaluating community-based prevention programs: A cross-cultural perspective. In M. A. Orlandi (Ed.), Cultural competence for evaluators: A guide for alcohol and other drug abuse prevention practitioners working with ethnic/racial communities (pp. 1-22). Rockville, MD: U.S. Department of Health and Human Services, Office of Substance Abuse Prevention.

Padilla, A. M., & Salgado de Snyder, N. (1992). Hispanics: What the culturally informed evaluator needs to know. In M. A. Orlandi (Ed.), Cultural competence for evaluators: A guide for alcohol and other drug abuse prevention practitioners working with ethnic/racial communities (pp. 117-146). Rockville, MD: U.S. Department of Health and Human Services, Office of Substance Abuse Prevention.

Pancer, S. M. (1997). Program evaluation. In J. W. Sadava and D. R. McCreary (Eds.), Applied social psychology (pp. 47-67). Englewood Cliffs, NJ: Prentice Hall.

Papineau, D., & Kiely, M. C. (1996). Participatory evaluation in a community organization: Fostering stakeholder empowerment and utilization. Evaluation and Program Planning, 19, 79-93.

Patton, M. Q. (1997). Utilization-focused evaluation (2nd ed.). Thousand Oaks, CA: Sage.

Prilleltensky, I. (1993). The immigration experience of Latin American Families: Research and action of perceived risk and protective factors. Canadian Journal of Community Mental Health, 12(2), 101-116.

Prilleltensky, I. (1997). Values, assumptions, and practices: Assessing the moral implications of psychological discourse and action. American Psychologist, 52, 517-535.

Prilleltensky, I., & Nelson, G. (1997). Community psychology: Reclaiming social justice. In D. Fox and I. Prilleltensky (Eds.). Critical psychology: An introduction (pp. 166-184). London: Sage.

Prilleltensky, I., Peirson, L., Gould, J., & Nelson, G. (1997). Planning mental health services for children and youth: Part I - A value-based approach. Evaluation and Program Planning, 20, 163-172.

Schinke, S. P., Monacher, M. S., Holden, G. W., Botvin, G. J., & Orlandi, M. A. (1989). American Indian youth and substance abuse: Tobacco use problems, risk factors and preventive interventions. Health Education Research, 4(1),137-144.

Serrano-Garcia, I. (1990). Implementing research: Putting our values to work. In P. Tolan, C. Keys, F. Chertok, & L. Jason (Eds.), Researching community psychology: Issues of theory and methods (pp. 171-182). Washington, D. C.: American Psychological Association.

Sowers, W. M., & Garcia, F. W., & Steitz, S. L. (1996). Introduction: Community-based prevention: An evolving paradigm. Journal of Primary Prevention, 16, 225-232.

Stanton, W. R., Mahalski, P. A., McGee, R., & Silva, P. A. (1993). Reasons for smoking or not smoking in early adolescence. Addictive Behaviors, 18, 321-329.

Vicary, J. R., Doebler, M. K., Bridger, J. C., Gurgevich, E. A., & Deike, R. C. (1996). A community systems approach to substance abuse prevention in a rural setting. Journal of Primary Prevention, 16, 303-318.

Wallack, L. (1985). Mass media, youth and the prevention of substance abuse: Towards an integrated approach. Journal of Children in Contemporary Society, 18,(1/2), 153-180.

Whitmore, E. (1991). Evaluation and empowerment: It=s the process that counts. Networking Bulletin: Empowerment and Family Support, 2(2), 1-7.

World Health Organization (1986). Ottawa Charter for Health Promotion. Canadian Journal of Public Health, 77, 6-14.

Yeich, S. (1996). Grassroots organizing with homeless people: A participatory research approach. Journal of Social Issues, 52, 111-121.

Table 1

Expression of Main Values in Program Evaluation

Values	Program Evaluation Phase I Phase II Phase III Phase IV Design Data Collection Analysis Final Report
Community Participation	Community members were involved in research design and in creation of questionnaires and interview guides	Following training, local residents administered questionnaires and conducted individual interviews and focus groups	Interpretation of qualitative findings was done in consultation with program staff and volunteers	Involvement of several staff and volunteers in composing final report
Cultural Sensitivity	Creation of tools that capture informal and conversational style of participants and that reflect local realities	Data gathering on intervention and control groups was conducted in Spanish	Analysis of findings was done in cultural context of Latin American immigrants and refugees in Canada	Sensitivity to own and other cultures in formulating suggestions for dissemination purposes
Comprehensive Vision of Health	Research design addressed multiple determinants of health	Data were gathered about personal, family, social, cultural, and community factors	Analysis of findings was done in ecological terms that avoid victim-blaming	Recommenda-tions for future research and action pertain to multiple levels of intervention

Table 2

Mean Scores for Children Ages 8 to 14 in the Intervention and Comparison Groups at Pre-Test and Post-Test on the Smoking Knowledge Items

Item	Time period	Intervention group	Comparison group
1. About 7 in 10 youth from 15 to 19 years of age smoke. (Correct answer = not true)	Pre	.08	.11
	Post	.21	.00
2. Smoking is only bad for you if you smoke a lot every day. (Correct answer = not true)	Pre	.41	.77
	Post	.41	.65
3. Breathing smoky air is dangerous even for people who don=t smoke. (Correct answer = true)	Pre	.83	.96
	Post	1.00	.96
4. It=s hard to stop smoking once you=ve started. (Correct answer = true)	Pre	.88	.74
	Post	.83	.74
5. About 30,000 people a year died because of smoking. (Correct answer = not true)	Pre	.08	.00
	Post	.13	.11
6. Companies that make cigarettes do not admit that smoking is dangerous. (Correct answer = true)	Pre	.75	.44
	Post	.54	.52
7. Companies that make cigarettes help people who get sick because of smoking. (Correct answer = not true)	Pre	.46	.59
	Post	.42	.70
8. In Canada, smoking among youth increased from 1990 to 1994. (Correct answer = true)	Pre	.46	.37
	Post	.63	.67
9. The majority of cases of lung cancer occur in people who smoke cigarettes. (Correct answer = true)	Pre	.96	.85
	Post	1.00	.96

Table 3

Mean Scores for Children Ages 8 to 14 in the Intervention and Comparison Groups at Pre-Test and Post-Test on the Relationship with Family, Community Participation, and Intention to Smoke Measures

Measure	Time period	Intervention group	Comparison group
Relationship with	Pre	14.75	14.04
family	Post	14.50	14.35
Community	Pre	9.54	8.21
participation	Post	10.21	6.92
Intention to	Pre	3.20	3.78
smoke	Post	3.70	3.52

Table 4

Summary of Children=s Qualitative Responses on Program Impact According to Program Objectives

Program Objective	Typical Answers	Representative Quote
Improve knowledge about smoking	- tobacco is harmful - tobacco is hard to resist	Tobacco causes cancer, attacks your heart, damages your lungs, make your teeth yellow, causes breathing problems, bad breath. Reduces the taste and smell of foods (11 year old girl).
Improve life and social skills	- program improved self esteem - program improved self-knowledge - program helped create better relations with peers	I learned how not to fight with my friends, how to talk with my friends, and how to play with them (9 year old boy).
Improve family communication	- family talked about "how to say no" - family discussed harmful effects of tobacco	We talked about the risks of smoking and about how to say no when somebody offers me a cigarette (12 year old boy).
Enhance community activism	- children delivered petition to city hall - children distributed flyers - children gave talks at school	It was nice for people to see the flyers we did. They know we're doing something good for the community (12 year old girl).

Table 5

Summary of Adults' Qualitative Responses on Program Impact

Program Impact On:	Typical Answers	Representative Quote
Parents	Parents acquired: - information - parenting skills	Having scientific knowledge about the harmful effects of tobacco is helpful. It helps guide our children better
Facilitators	Facilitators improved: - knowledge about tobacco - life/social skills - family communication - community development	There is a sense of community, of solidarity. We share the work and the duties.
Children	Children developed: - knowledge about tobacco - ability to cope with peer pressure - family communication - community appreciation	My boy already knows that smoking is bad for your health, that people who smoke die earlier, that smoking is bad for children, and that they can say no to whoever offers them cigarettes.
Family	Family developed: - communication - openness to talk about drugs	I think that at the family level the program had an impact in that children show their father the problems with smoking and he promised them to quit smoking, even though he hasn't done it yet.
Community	Community benefited from: - preventive impact	I like the fact that community children know about the harmful effects of smoking. This is important because they are my child's friends and they won't offer him cigarettes.

Table 6

Stakeholders' Views of Factors Affecting the Process

Stakeholder Group	Facilitative Factors	Inhibiting Factors
Children	- Activities (movies, role playing, puppets, games) - Themes (stress, self- esteem, smoking myths) - Snacks	- Activities (reading and "frontal" lessons) - Space problems - Discipline problems
Parents	- Effective and dynamic facilitation - Building on strengths of Spanish school - Human relations skills of staff	- Lack of parental involvement - Discipline problems
Staff and volunteers	- Effective planning - Conflict resolution - Community participation - Popular education methodology - Personal satisfaction - Material resources	- Lack of parental involvement - Discipline problems - Space problems

Authors Notes

This research was funded by Health Canada. The views expressed herein are solely those of the authors and do not necessarily represent the official policy of Health Canada. We wish to thank Health Canada as well as the members of the community who participated in the project. Correspondence concerning this article may be sent to Isaac Prilleltensky, Department of Psychology, Wilfrid Laurier University, Waterloo, Ontario, Canada, N2L 3C5. Electronic mail may be sent via Internet to iprillel@mach1.wlu.ca.