Program Evaluation Proposal

Class assignment for EDF 5461: Introduction to Program Evaluation

1. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 1









Evaluation Proposal for the Florida State University Transfer Student Services Program
EDF 5461: Introduction to Program Evaluation
Dr. Linda Schrader

Dawn Fae Adolfson
December 4, 2019

2. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 2
Table of Contents
Introduction.................................................................................................................................... 3
Program Background ............................................................................................................... 3
Logic Model ............................................................................................................................... 3
Evaluation Purpose & Questions ............................................................................................... 4
Research Design & Methodology .............................................................................................. 9
Data Collection Instrument ....................................................................................................... 11
Communication Plan.................................................................................................................. 12
Reflection ..................................................................................................................................... 13
References .................................................................................................................................. 13
Appendices.................................................................................................................................. 14
Appendix A: Abbreviated Syllabus of SLS 3140 ............................................................... 14
Appendix B: Logic Model ...................................................................................................... 16
Appendix C: Executive Summary ........................................................................................ 17

3. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 3
Introduction
Program Background
The Transfer Student Services (TSS) program at Florida State University (FSU)
was developed to provide academic services to transfer students to reduce the effects
of transfer shock, help students get out of academic probation, and graduate with their
bachelor’s degree. The purpose of the program is to offer an academic success
strategies course, SLS 3140: Academic Success Strategies for Transfer Students, for
students whose institutional GPA drops below a 2.0 and they are placed under
academic probation and could be dismissed from FSU if they do not improve their GPA
score. Therefore, TSS offers this 1-credit course to teach transfer students the
academic strategies and skills needed to be successful at a large, research university.
By offering this course, the program’s goals are to improve transfer student
retention, decrease the number of transfer students under academic probation, and
increase transfer student engagement and involvement on campus. To achieve these
goals, the program is encouraging new transfer students to take SLS 3140. Transfer
students under academic probation are mandated to take the course in the first
semester of their probation.
The population of transfer students is diverse in terms of their demographics and
goals. Many transfer students are of nontraditional college age with the majority being
between 25 and 35 years old. Also, many transfer students are also are parents and
veterans and perhaps are even working full-time while striving to complete their
bachelor’s degree. Their goals are diverse from completing a bachelor’s degree and
going straight to the workforce, to starting their own business, or they have plans to
apply to graduate school for higher education.

Logic Model
The logic model describes the program’s inputs, activities, and outputs. Using the
program’s goals, it also states the desired outcomes and standards to indicate the
outcomes were achieved and which sources or methods will be used to collect data to
support whether the goals were achieved or not. The complete logic model is in
Appendix B.

4. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 4
Offering a course such as SLS 3140 requires multiple inputs, but the primary
input is funding to pay the faculty coordinator and the graduate assistants that teach the
course. Each graduate assistant receives a tuition waiver and stipend for their teaching.
Also, since the course is offered face-to-face, campus facilities are needed such as
classrooms and office space in which to offer classes and student meetings.
Additionally, the program requires instructional materials to teach the courses which
have been developed by the team.
Using these inputs, the program’s main activity is teaching multiple sections of
the SLS 3140 class each semester. The course is taught both face-to-face and online to
be the most flexibility for students. In addition to teaching the course, other activities
include advertising the course to new students at their initial university orientation and
providing personal academic consultations to students enrolled in the course. These
consultations are one-on-one appointments between the student and their SLS 3140
teacher. In these consultations, students have a chance to reflect on past challenges,
describe current challenges, create goals, and review course assignments.
Two major outputs of this program are the number of students that take the SLS
3140 course and the number of consultations that are conducted with the students.
Other minor outputs include the student assignments.
As a result of this program, the expected outcomes are improved student
retention, decreased number of students under academic probation and increased
student involvement on campus. The specific indicators of these outcomes are that 80%
of students that take SLS 3140 are not under academic probation after that semester,
75% of transfer student are retained, or re-enroll in the following semester, and 75% of
students that take SLS 3140 report high rates of campus involvement.

Evaluation Purpose & Questions
The purpose of this evaluation is to determine the effectiveness of the TSS
Program, specifically the SLS 3140 class, to improve transfer student retention,
decrease the number of transfer students under academic probation, and increase
transfer student engagement and involvement on campus. The evaluation will use a
formative focus since the program only started one year ago and it is still being

5. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 5
improved and revised. The results of this evaluation could assist with the revisions so
that the goals can be achieved.
This evaluation will use a program evaluation approach to determine if the
program is achieving its objectives and goals. Specifically, this approach will collect data
to determine if the goals are being achieved and to what extent. This data is useful to
determine if any revisions or changes need to be made in the program to improve its
effectiveness. The research design will consist of quantitative methods such as program
reports and student survey analyses.
The major stakeholders are the transfer students that participate in the program,
the lead faculty coordinator of the TSS program, the graduate teaching assistants that
teach the SLS 3140 class, the administrative staff of the Department of Undergraduate
Studies that are interested in the success of the program, and the Dean of
Undergraduate Studies. The students are interested in the program to help them
achieve academic success and graduation. The direct team, the instructors and lead
coordinator, depend on this program for employment and career advancement. Also,
they would like to see the program be successful and achieve its goals. Higher
administrators in the Division of Undergraduate Studies that invest funding into the
program will be interested to see if the program works or not. This evaluation will
primarily focus on gathering data from the students that directly receive services from
the program.
Table 1 details the evaluation audience checklist and all the potential individuals
and groups that may benefit from the evaluation’s results. These individuals and groups
may be interested in the results of the evaluation to make policy or decisions, or
perhaps just to react to the findings. The program managers and delivers will be asked
to provide input to the evaluation and also may make operational decisions about the
program depending on the results of the evaluation. The direct clients of the program,
the transfer students, will be asked to provide valuable input into the evaluation and
provide data to answer the evaluation questions. Other higher individuals, such as the
funder of the program and top managers of the Department of Undergraduate Studies,
will be interested in the results of the evaluation to develop new policies or even expand
the program.

6. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 6
The research questions for this evaluation proposal are (1) How many transfer
students under academic probation that pass SLS 3140 are removed from probation the
following semester?, (2) how satisfied are students that take SLS 3140 with how the
class improved their academic study skills, time and task management, involvement on
campus, and overall performance in their other classes? and (3) how has transfer
student retention changed before and after the program was started?
These questions were selected as the most important to the evaluation because
they scored the highest in all the criteria for ranking evaluation questions. The criteria
used to rank potential evaluation questions was developed by Cronbach and developed
into a matrix by Fitzpatrick, Sanders, & Worthen (2011, p. 331). The selected evaluation
questions are most likely to have an impact on the course of events because they could
influence change in the program and SLS 3140 course. Also, they are critical to the
scope and comprehensiveness of the evaluation because they directly align with the
purpose of the evaluation to truly understand how the program is impacting transfer
students that are under academic probation. Finally, criteria can be developed for each
of these questions to determine success of the program.
Any program evaluation can experience ethical issues. A potential ethical issue
in this research study could be the potential for certain stakeholders to pressure the
evaluation to find positive results or alter the results to show positive findings of the
program to validate the funds invested into the program, or on the flip side, other
stakeholders may want the evaluation to find negative findings to validate closing the
program to invest the funds into other opportunities. Also, it is important to include all
relevant stakeholders to ensure that the diverse population of transfer students are
included in the evaluation.
As a result, to these potential ethical issues, two program evaluation standards
were selected to help address these issues. First, is standard U2: Attention to
Stakeholders, which states that “evaluations should devote attention to the full range of
individuals and groups invested in the program and affected by its evaluation”
(Fitzpatrick, Sanders, & Worthen, 2011, p. 499). This evaluation should gather data
from all types of transfer students that take the course to consider all diverse
backgrounds and needs.

7. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 7
The second program evaluation standard is standard A2: Valid Information,
which states “evaluation information should serve the intended purposes and support
valid interpretations” (Fitzpatrick, Sanders, & Worthen, 2011, p. 500). The data collected
in this evaluation should relate to the program objectives and provide information to
make decisions about the program. It will be important that the data is not used for any
other purposes other than to answer the evaluation questions and make decisions
about the program.

8. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 8
Table 1: Evaluation Audience Checklist
Individuals, Groups, or Agencies
Needing the Evaluation’s Findings
To Make
Policy
To Make
Operational
Decisions
To Provide
Input to
Evaluation
To React For
Interest
Only
Developer of the program


Funder of the program


Person/agency who identified the
local need



Boards/agencies who approved
delivery of the program at local
level



Local funder


Other providers of resources
(facilitates, supplies, in-kind
contributions)


Top managers of agencies
delivering the program

Program managers


Program deliverers


Sponsor of the evaluation


Direct clients of the program


Indirect beneficiaries of the
program (parents, children,
spouses, employers)


Potential adopters of the program


Agencies who manage other
programs for this client group



Groups excluded from the
program


Groups perceiving negative side
effects of the program or the
evaluation



Groups losing power as a result of
use of the program


Groups suffering from lost
opportunities as a result of the
program


Public/community members

(Fitzpatrick, Sanders & Worthen, 2011, p. 289)

9. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 9
Research Design & Methodology
This evaluation will use a program-oriented approach to determine whether or
not the program is achieving its goals and objectives. Each evaluation question uses
different quantitative methods to answer the questions. Figure 1 below provides an
overview of the evaluation’s research design, purpose, focus, and data collection
methods.
The first research question will use quantitative data to determine how many
students that are under academic probation and take SLS 3140 are removed from
probation the following semester. This is determined based on the students’ GPA
scores. A student is under probation if their institutional GPA falls below a 2.0 whereas
a student in good standing has a GPA of 2.0 or higher.
The second research question will use a quantitative research through student
surveys. The students will self-evaluate their academic skills and abilities before and
after taking SLS 3140. Then, each student’s post-course survey and pre-course survey
will be compared with each other to determine if there was any improvement. If there
was an improvement, it could be attributed to the course. Also, the results could be
analyzed to determine if any specific skills or abilities are not improved as much and
instruction could be improved in those areas.
Finally, the third research question will use institutional reports to collect
quantitative data on transfer student retention and how it has changed as a result of
offering the SLS 3140 course. Retention is determined by checking to see if the student
has enrolled in courses for the following semester, if so, they are considered retained.
Therefore, the evaluation will collect reports for various semesters before the program
started and during the program and compare these semesters to see if the transfer
student retention percentage as changed.

10. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 10
Figure 1: Research Design Graphic Organizer

Purpose of
Evaluation
•Determine if the program is achieving its goals reduce the number of transfer students
under academic probation, improve transfer student retention, and improve transfer
student engagement and involvement on campus.
Focus of
Evaluation
•The evaluation will be formative in nature since the program is still new and able to be
revised and updated based on the evaluation.
Evaluation
Questions
•How many (or what percent of) transfer students under academic probation that pass
SLS 3140 are removed from probation the following semester?
•How satisfied are students that take SLS 3140 with how the class improved their
academic study skills, time and task management, engagement and involvement on
campus, and overall performance in their other classes?
•How has transfer student retention changed before and after the program was started?
Research
Design
•The design will quantitative methods to collect program data and students self-
evaluations and satisifaction with the course.
Data Collection
Methods
•University reports will include the number of transfer students under academic probation,
number of transfer students off academic probation that took SLS 3140, and percentage
of transfer students retained at the university.
•Student surveys will be adminstered to students at the beginning and end of the SLS
3140 course for students to self-evaluate their academic skills and abilities

11. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 11
Data Collection Instrument
First, evaluation question 1 is: How many transfer students under academic
probation that pass SLS 3140 are removed from probation the following semester? To
answer this question, the evaluation will collect institutional reports such as class rosters
to determine how many students were removed from academic probation after taking
SLS 3140. If program reports are not available, then student surveys could be
administered at the beginning and end of the course. On the pre-survey, the question
will be In this semester, are you currently under academic probation due to your FSU
GPA being below 2.0? On the post-survey, the questions will be Did you pass SLS3140
with a C or better? and After the completion of this semester, were you removed from
academic probation?
The second evaluation question is How satisfied are students that take SLS 3140
with how the class improved their academic study skills, time and task management,
involvement on campus, and overall performance in their other classes? Data to answer
this question will be collected through student surveys given to students at the
beginning and end of the SLS 3140 course that asks the students to self-evaluate their
academic skills and abilities. Then the results of the pre- and post-surveys can be
compared to determine if the students self-evaluate that their skills and abilities have
improved from the beginning to the end of the semester in which they take SLS 3140.
Also, the survey can ask questions about the students’ satisfaction with the course’s
learning objectives, assignments, activities, instructional methods, etc.
The student survey will include items that ask the students to self-evaluate their
skills and abilities on a five point Likert scale. The scale would range from 1, strongly
disagree, to 5, strongly agree, to various statements regarding academic skills and
abilities. For example, some statements can be I am confident in my overall academic
abilities. I am confident in my reading comprehension skills. I am able to organize my
time and tasks effectively. I am confident in my study skills to prepare for exams. Since
this data would be self-reported by the students, the surveys at the end of the semester
will be compared to those at the beginning to determine if there is any improvement.
The third evaluation question is How has transfer student retention changed
before and after the program was started? To answer this question, the evaluation will

12. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 12
collect program reports to determine the number of transfer students enrolled each
semester that were previously enrolled at FSU and the number of transfer students that
graduate from FSU. Retention can be determined by checking to see which students
have enrolled in courses for the next semester, if yes, then they can be considered
retained. The evaluation will collect data to see if transfer student retention has changed
as a result of offering the SLS 3140 course. The reports should include: the number of
transfer students enrolled each semester, the number of transfer students that are
continuously enrolled (were also enrolled in courses at FSU the previous semester),
and the number of transfer students that graduate from FSU. These reports should be
gathered from various semesters beginning with two semesters prior to the program
start and up to the last completed semester.

Communication Plan
Throughout the evaluation process, the stakeholders will want regular updates on
the process. The team will remain in communication with important stakeholders such
as the faculty coordinator and administration in the Division of Undergraduate Studies
through regular emails with summarized data results. At the end of the evaluation
period, a brief presentation will be conducted so that these stakeholders can hear the
results and ask questions regarding the evaluation. Also, a brief written report will be
provided so that other stakeholders in the division can read the results during their own
time. An example of a brief report, or executive summary, is attached in Appendix C.
Formal reports with all the data represented will be provided to decision making
stakeholders.
The students are also important stakeholders in this process. The results of the
evaluation will be presented to the students in a visual and engaging format such as an
online infographic or brief video regarding the results. The students do not need to see
the entire report or details, but rather may be interested in how the course has effected
transfer students. Also, this kind of engaging presentation style would be beneficial to
include on the transfer student website so that new transfer students can see the results
of this program when making the decision to transfer to FSU.

13. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 13
Reflection
After writing this program evaluation proposal, I have learned more about how to
plan and implement the evaluation process. I think that it’s important to spend sufficient
time planning, understanding the program’s objectives and stakeholders, and designing
the evaluation questions. Next time I plan an evaluation, I would perhaps include more
diverse methods such as qualitative methods to collect data. From this process, I have
learned more about how to consider the stakeholders throughout the evaluation
process, even in the planning stage. Finally, I anticipate my biggest challenge in this
plan would be analyzing the data and drawing conclusions or recommendations. Also, I
may have challenges working with diverse stakeholders and communicating the results
clearly and effectively while maintaining and objective point of view.

References
Fitzpatrick, J.L., Sanders, J.R., & Worthen, B.R. (2011). Program evaluation: Alternative
approaches and practical guidelines (4th ed.). Upper Saddle River, NJ: Pearson.

14. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 14
Appendices
Appendix A: Abbreviated Syllabus of SLS 3140
The abbreviated course syllabus is detailed below. It explains the course purpose,
learning objectives, course assignments, and module topics.
Course Description: This course is especially designed for transfer students who
seek to improve their academic outcomes. It focuses on the development and
application of classroom, life, and engagement practices necessary for successfully
navigating the increased scholastic expectations of a research university. Students will
reflect on and adjust formerly unsuccessful strategies and choices.
Learning Objectives: By the end of this course, students should be able to:
1. Reflect critically on past assumptions, prior learning, prior thinking, and prior
behavior.
2. Set realistic academic and career goals and establish benchmarks to measure
progress toward those goals.
3. Apply appropriate learning strategies according to their own learning styles and
course content.
4. Monitor their own progress toward learning, identify areas of need, and utilize
appropriate campus resources to support and reach desired outcomes.
5. Develop a comprehensive calendar and strategy to prepare for the requirements of
any course, including exams, papers, projects, or presentations.
6. Select appropriate learning and networking experiences to address identified areas
of need, enhance the transfer experience, and prepare for future professional and
academic opportunities.
Pre-requisites or Co-requisites: Students should be undergraduate students who
have transferred into Florida State University from another institution with 45 or more
earned credit hours.
Required Texts, Readings, and/or other Resources: There is no assigned
textbook for this course. Any assigned readings will be distributed by the instructor or
will be available via Canvas. Students should have a separate spiral notebook or 3-ring
binder for each course on his/her schedule. Students may be asked to purchase index
cards, post-it notes, highlighters, or other study, time management, and organization

15. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 15
aids during the semester. Each student is expected to have access to a calendar or
organizer to record and keep track of important deadlines, assignments, and personal
appointments. Students may select any form of planner including a physical calendar,
notebook, smartphone, Google Calendar, or any other approved example.
Teaching Strategies: The methods of instruction for this course include lecture,
large and small group discussion, audio-visual, on-campus field experiences, and guest
speakers.
Field/Clinical Activities: Students will be required to utilize assigned campus
resources and reflect on those experiences, for example, visiting the academic advisor,
utilizing office hours, attending support workshops, or engaging in tutoring/supplemental
instruction sessions.
Topical Course Outline:
 Self-Regulated Learning, Goal Setting, & Motivation
 Managing Tasks & Using Time Wisely
 Active Reading, Active Listening, & Developing Notes and Graphic Organizers
 Preparing for Exams & Maximizing Office Hours
 Maximizing Campus Resources & Self-Care
 Academic Leadership & Initiative
 Scholarly Thinking & Analysis
 Career & Graduate School Planning

16. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 16
Appendix B: Logic Model

Inputs Activities Outputs Outcomes
Standards
Indicators Sources/Methods
 Money to pay
teachers and
staff
 Facilities in
which to offer
classes
 Instructional
materials such
as
PowerPoints,
assignment
materials and
rubrics,
Canvas course
setup, etc.


 Advertise
SLS3140 class
at orientation
 Instruct
SLS3140
classes
 Provide
personal
academic
consultations
to students
enrolled in
SLS 3140
 Number of
students
enrolled in
classes
 Number of
consultations
conducted with
students
 Improved
transfer
student
retention
 Decreased
number of
transfer
students under
academic
probation
 Increased
transfer
student
involvement
on campus
 80% of
students that
take SLS 3140
are not under
academic
probation after
that semester
 Transfer
student
retention is at
least 75%
 75% of
students report
high rates of
campus
involvement
 Institutional
reports with
student GPAs
and retention
rates
 Pre/Post SLS
3140 course
survey for
students to
self-evaluate
their academic
success skills
and strategies
 Questionnaires
to determine
transfer
students’
involvement on
campus

17. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 17
Appendix C: Executive Summary
Overview
The Transfer Student Services (TSS) Program at Florida State University (FSU)
provides academic services to transfer students, primarily a 1-credit course: SLS 3140:
Academic Success Strategies for Transfer Students. The purpose of this evaluation is to
determine the effectiveness of the Transfer Student Services Program to improve
transfer student retention, decrease the number of transfer students under academic
probation, and increase transfer student engagement and involvement on campus. This
evaluation will use a program evaluation approach to determine if the program is
achieving its objectives. The research design will consist of quantitative methods such
as program reports and student survey analyses.

Stakeholders & Rationale
The primary stakeholders are the transfer students that take the SLS 3140 course. Their
input in the evaluation is vital to ensure the program is achieving its goals to help
transfer students be successful at FSU. Also, the administrators that work for the FSU
Division of Undergraduate Studies will be interested in the results of this program since
they allocate funds to the program to hire staff and instructors.

Evaluation Questions
1. How many transfer students under academic probation that pass SLS 3140 are
removed from probation the following semester? Students under academic
probation, whose institutional GPA drops below a 2.0, are mandated to take the SLS
3140 course. The answer to this question will answer whether or not the course is
achieving this objective to help transfer students get off of academic probation and
return to good standing at FSU.

2. How satisfied are students that take SLS 3140 with how the class improved
their academic study skills, time and task management, involvement on
campus, and overall performance in their other classes? The students’
satisfaction is also important to understand to determine if the course is helping

18. EVALUATION PROPOSAL OF TRANSFER STUDENT SERVICES PROGRAM 18
them improve relevant skills and strategies that lead to academic success such as
study skills, time and task management, and involvement on campus.

3. How has transfer student retention changed before and after the program was
started? The administrators of the program, such as the Dean and Associate Deans
of Undergraduate Studies, support this program and would like to see it impact
transfer student retention. Therefore, if the evaluation can determine that retention
was positively impacted, then the program will continue to be funded and perhaps
additional funding could be allocated to expand the program.

Data Collection
This evaluation will collect program reports and student surveys. The program reports
will provide data about the students that have taken the course such as their probation
status, GPA, grades, and retention. Also, students that take the course will be asked to
complete a survey at the beginning and end of the semester that will ask them to self-
evaluate their academic skills with the anticipation that the students’ self-evaluation will
increase at the end of the course if the course achieved its objectives.

Standards
 U2: Attention to Stakeholders: Evaluations should devote attention to the full range
of individuals and groups invested in the program and affected by its evaluation. This
evaluation should gather data from all types of transfer students that take the course
to consider all diverse backgrounds and needs. Also, administrators that are
invested in the program must be involved in the evaluation process.
 A2: Valid Information: Evaluation information should serve the intended purposes
and support valid interpretations. The data collected in this evaluation should relate
to the program objectives and provide information to make decisions about the
program.