SUNY Oswego Starfish pilot-CIT presentation

Presentation regarding the outcomes, implementation and operation of Starfish as a potential retention intervention.

1. Closing the Loop: Improving Student Retention through Analytics A report on SUNY Oswego's Starfish Early Alert pilot Michelle Bandla Coordinator, First Year Programs Greg Ketcham Director, Academic Programs, Division of Extended Learning Rameen Mohammadi Associate Provost for Undergraduate and Special Programs Elizabeth Schmitt Professor of Economics SUNY CIT May 2014

2. Agenda Why Starfish? Introduction to Starfish Pilot Early Alert and Connect Project Goals Specific Expectations/Sample Communications Pilot Assessment Usage data Faculty Focus Groups and Changes in Spring 2014 How to assess? Comparing pilot and non-pilot groups Pilot Implementation Issues Angel integration Faculty training and participation Other issues Slide 2

3. Why Starfish? Problem: Retention and persistence rates below institution goals Institution vitality Student success in meeting educational and life goals Accountability Nationally SUNY comprehensives are in the 75-80 th percentile of 4-year publics in persistence measures But Oswego’s retention and graduation rates are lower than most SUNY comprehensives Slide 3

4. Why Starfish? Solution: IPAS Integrated Planning and Advisement Services “…leverage technology to enhance and streamline course advising ... tracking of student progress and targeting of support services, in order to improve student academic decision-making as well as institutional strategies to help students stay on track to degree attainment .” [Gates Foundation LRM Market Acceleration Project, February 2013] Slide 4

5. Why does an IPAS help? Metacognition Dunning & Kruger (1999), Dunning et al (2004) Early targeted communications offer more effective feedback to struggling students Earlier intervention gives us more options Communication to build connection Student satisfaction with their classroom experience ( Pascarella 2011) Connection to resources Connection to instructors

6. Why Starfish? Oswego had been evaluating IPAS vendors for over two years Constantly changing product Major financial commitment SUNY adoption? The (unsuccessful ) Gates grant opportunity pushed us to focus on a vendor and move forward with a pilot Slide 6

7. Why Starfish? Vendor choice: Starfish Integrates with Angel, Tutortrac , Noel Levitz CSI, Banner Early Alert and Connect products Starfish Pilot, 2013-14 967 students Freshmen and Transfers (Biology, Psychology, Public Justice, Business ) Criteria: Department interest, past success rates, good indicator of campus experience Slide 7

8. Introduction to Starfish Pilot Early Warning & Student Tracking System Student concerns can be manually flagged when observed or through periodic surveys. Automatic flags are raised based on online grade book data. Recommendations can be communicated to the student. Appropriate personnel are informed of flags raised and cleared. Close the loop with faculty when progress is made. Slide 8 Educational Support Networking System Each student has personal My Success Network of instructors and campus staff. Online scheduling for in-person meetings – both individual and group appointments. Appointment confirmation by email. Documentation of contact with students through private and shared notes. Service catalog of available campus support resources.

9. Introduction to Starfish Pilot Goals for the Pilot: Participation in Progress Surveys: Progress surveys provides instructors the opportunity to raise concerns or kudos for pilot students in their course sections. Notification to students of concerns: Email notifications sent to students that include comments provided. Notification to advisors and coordinated o utreach to flagged students: Email notifications sent to advisors. Advisors to outreach to pilot students and close the loop with instructors.

10. Specific Expectations Slide 10

11. Example Survey Announcement To:  {Instructor Name} From:  [email protected] Subject: [Starfish] Progress Survey for {Deployment Date} Dear {Instructor Name}, This spring is the second semester of our year-long Starfish Early Alert Pilot. You are receiving this email because you have one or more of the pilot students in your course(s). You have a critical role in this process as you best know how the student is doing in your course. Starfish provides us with the communication tools to identify the struggling students and intervene earlier in order to help them succeed.   Approximately 1000 first year and transfer students are in the PILOT cohort from the following majors: Biology/Zoology, Public Justice, Psychology and those in the School of Business.With over 3,600 performance flags and kudos generated in the fall, faculty participation exceeded expectations. We are hoping that you will continue to help us through this pilot and utilize Starfish to report on the academic progress of our students. There will be two Progress Surveys deployed this semester through Starfish prior to mid-term in order to receive early performance feedback about our PILOT students. Starfish system offers many other tools to aid in assisting our students that some faculty have already found useful.  For example, you have the ability to identify “To Do” items for students, use the Starfish appointment features that synchronize with Google Calendar, or take advantage of the “Zoom In” option that uses your Angel gradebook to raise flags or generate kudos for students. To learn more about the functionality of Starfish, please follow this link: Oswego.edu/starfish Here is a short YouTube video explaining how to submit a progress survey: http://www.youtube.com/watch?v=NMMPaZxDMP8&feature=youtu.be Thank you for your participation! We appreciate your dedication to student success. On behalf of the Starfish Implementation Team, Rameen Mohammadi Associate Provost

12. Example Student Email Notification From : [email protected] ReplyTo : [email protected] Subject : [Starfish College Early Alert] Progress Concern: CourseName Dear StudentName , You have been flagged for FlagName in CourseName . The professor in this class has indicated that you are not making adequate progress and she/he is concerned about your ability to successfully complete the class.   [if comments] Below are your instructor’s comments: These are my comments! Please contact your instructor to discuss these issues and to make a plan to improve your performance in the course.  Refer to the course syllabus for contact information for your professor. We also recommend that you contact your advisor to discuss this issue and any other issues you are experiencing this semester. The College has free tutoring services to help you improve your academic performance. In addition, advisement and personal counseling services are available to support academic and career planning, as well as to assist you with personal concerns. To seek tutoring, please contact the Office of Learning Services: Address: 173 Campus Center Email: [email protected] website: http://www.oswego.edu/academics/support/OLS.html To seek Major or Career Exploration contact The Compass Address: 145 Campus Center Email: [email protected] website: http://www.oswego.edu/student/success/compass.html To seek personal counseling, please contact the Counseling Center: Address: 113 Walker Health Center Phone:  [phone] website: http://www.oswego.edu/student/services/counseling_services.html Sincerely, Student Success Team

13. Usage Data 2013-2014 Fall 2013 Spring 2014* Students 967 977 Courses and labs 711 704 Instructors 360 383 Advisors 100 131 Courses /w Completed Surveys 429 (60%) 431 (61%) Academic Flags 1440 765 Kudos 1951 768 System Flags (3 flags) 199 55 System Flags (6 flags) 76 6 Slide 13 *Significant changes in the flag structure occurred between these semesters as a result of faculty feedback

14. Usage Data 2013-2014 Fall f ocus group feedback Fewer flag categories Better defined default options Clearing flags—what does it mean? Clarifying the advisor role Slide 14

15. Tracking Items Changes Fall 2013 to Spring 2014 Fall 2013 Flags Spring 2014 Flags Attendance Concern Excessive Absences In Danger of Failing In Danger of Failing Low Quiz/Test Scores Incomplete or Unsatisfactory Graded Work Low Participation Failure to complete Assignments Behavioral Concern General/Behavioral Concern General Concern

16. Tracking Items Changes Fall 2013 to Spring 2014 Fall 2013 Kudos Spring 2014 Kudos Keep Up the Good Work Outstanding Academic Performance Outstanding Academic Performance Showing Improvement Showing Improvement

17. Assessment Plan Performance Differences between Fall Cohorts (2012 and 2013) Note: selectivity matrix differs across these cohorts Course specific performance Fall 2013: Pilot vs. non-Pilot Compare to Fall 2012 Fall to Spring 2013-14 Retention Good standing Fall to Fall Retention (not yet available) GPAs for the 2013-14 academic year (not yet available) Student, faculty and staff user surveys (in progress) Slide 17

18. Some Early Data Overall (freshman and transfers), there were no significant differences (Fall-Spring) in Semester withdrawals Term GPA Instances of disqualification, academic warning or attainment of honors (Dean's list, President's list ). This is robust to subgroup comparisons Psychology Transfers, Psychology Freshmen, Biology Transfers, etc . Slide 18

19. Overall Comparison, Pilot Cohorts Fall Semesters Slide 19 Differences shown here are not statistically significant.

20. Other Grade Data Fall semester grade comparisons Fall 2013 Pilot vs. nonpilot cohorts in selected courses Fall 2012 vs. 2013 Compare the pilot-eligible cohorts (first year students in certain majors) Preliminary analysis finds little difference Slide 20

21. Fall 2013 Average GPA Slide 21 CHE 111 N GPA Pilot 181 2.45* Non-pilot 124 2.76 *significant at 5% level BIO 203 N GPA Pilot 22 2.11* Non-pilot 84 2.43 PSY 100 N GPA Pilot 114 2.62* Non-pilot 157 2.35 No difference : ANT 111, BIO 110, CHE 209, ECO 101, HIS 100, HIS 202, HIS 203, MAT 104, MAT 158, MAT 208, MAT 210, MUS 101, SPA 102, THT 110 Selected Courses, All Sections

22. Pilot-eligible comparison, Fall 2012 vs. 2013 Slide 22 MAT 208 N GPA 2012 47 1.88* 2013 42 2.83 *significant at 5% level ECO 101 N GPA 2012 216 2.42* 2013 287 2.65 No difference : BIO 110, BIO 203, CHE 111, CHE 209,HIS 100, HIS 202, HIS 203, MAT 104, MAT 158, MAT 210, PSY 100 Note: selectivity in the Fall 2012 cohort is greater than the Fall 2013 cohort Selected Courses, All Sections

23. Eco 101 800, Fall 2013 Principles of Microeconomics 410 students Mastery learning quiz system i <clicker sessions every class (attendance proxy) Slide 23 *significant at 5% level N Course Average Missed i <clicker sessions Missed module quizzes Pilot 249 71.46 3.35* 1.45* Non-pilot 161 70.30 4.45 2.22

24. What to make of the preliminary data? Is it reasonable to expect big differences in the first semester of the pilot? Faculty usage Limited integration Intervention options and follow-through Is the action at the center or at the tails of the distribution? Slide 24

25. Instructor Comments CSC 102 Business Information System and Computing Tools 249 students, multiple sections Mike Smith wrote: My experience with Starfish was a positive one, I have larger classes ( for fall '13 I had one of 88 and 3 of 56 ). On my initial flagging of items I was swarmed with students... I think prior to this even though I would stand in front of the class and warn them "Task 'X' is due tomorrow ,” it was easier for them to put off (procrastinate) the task until it was too late. Most of the students seemed to respond to Starfish as an official warning, and spoke to me about the task with an intended solution . Slide 25

26. Instructor Comments Eco 101 Principles of Microeconomics 420 students, large section John Kane Wrote: While I have for many years been providing students with current grade report and class averages in ANGEL, the alerts and kudos sent through Starfish seemed to have resulted in more focused attention from students. Students that were falling behind in class were more likely to stop by to discuss their performance after receiving alert notifications. Students performing well were more likely to visit my office after receiving "kudos" through Starfish. In a class of 420 students, students often feel a bit lost in the crowd. These notifications helped reduce this barrier. Slide 26

27. Pilot Implementation Issues Technical issues Angel integration Integration of Starfish and Google calendars Integration of Tutortrac data, NL CSI Flags within and across courses Usage issues Faculty buy-in Effective interventions--What does it mean to close the loop? Full integration with other software Decision-time comes before Fall 2014 firm enrollment Slide 27

28. Advice for Adopters Allow time for implementation, faculty training Put an assessment plan in place What data? When will you gather it? Analyze it? Get feedback early and often from end users Don’t be afraid to make changes mid-pilot based on feedback Keep communications open with your vendor Trouble shooting Requests for new features Successes from other user campuses Slide 28

29. Next Direction Starfish decision Discontinue? Continue pilot? Full implementation? What is a reasonable time frame to expect results? Slide 29