2020 Final Report Summary
Transformation Grants Round 14, 15, and 16
Highlights
Projects ending in Spring, Summer, or Fall 2020 were in a precarious situation due to the COVID-19 pandemic. While it may be difficult to determine the effects of a Transformation Grant project in 2020, we can infer the following from the 2020 Transformation Grants Final Report:
- A total of 57 reporting teams saved 20,000+ students over $3 million just within the time of the project, and mostly within only the final semester of instruction.
- The COVID-19 pandemic caused a set of complications with the implementation and measurement of grant projects, including drastic changes in instruction and learning during the emergency move to online instruction.
- An unexpected barrier due to the move to online instruction was an increased difficulty in gathering qualitative data from students, with survey response rates not meeting teams’ expectations and focus groups being unsustainable to conduct in an online environment.
-
Despite these complications, Transformation Grant projects were not likely to result in negative changes to student opinions, outcomes, or retention during the pandemic:
- Teams unanimously reported positive overall student opinions (100%) of the new materials.
- Teams reported largely positive (65.22%) or neutral (30.43%) comparative student learning outcomes compared to previous semesters, control groups, and/or instructor and departmental averages with a commercial textbook.
- Teams reported equally positive (42.55%) or neutral (42.55%) comparative course-level retention and completion rates compared to previous semesters, control groups, and/or instructor and departmental averages with a commercial textbook.
- Students who gave positive reviews of the new materials in courses often commented on the relevance of the materials (to the course and to their overall career goals/lives) as a quality indicator.
Background
Affordable Learning Georgia’s (ALG) Transformation Grants are intended to pilot different approaches in University System of Georgia (USG) courses in reducing the cost of course materials to students. Approaches include the adoption, adaptation, and creation of open educational resources (OER), the adoption of all-rights-reserved no-cost or low-cost materials, and the adoption of materials available through GALILEO and USG libraries at no additional cost to students. The grants support release time or salary/overload, extra staff assistance (instructional designers, librarians, etc.), materials, and professional development needed for faculty to transform their use of learning materials.
This report summarizes the findings of all grantees in grant rounds ending in calendar year 2020. Three rounds (spanning rounds 14 through 16) resulted in 57 projects from teams of USG faculty and staff at USG institutions.
This is a significant increase from calendar year 2019, where only 27 Transformation Grants projects ended. This is due to a change that happened in late 2018: ALG transitioned to a deadline maximum of no less than one year from the originating semester per each round of Transformation Grants, a more reasonable and consistent amount of time than in previous rounds where deadlines were confined to the end of fiscal years.
All grant projects, along with a collection of their proposals, syllabi, and final reports, are included in the lists of Rounds 14-16 grantees:
The summary below addresses the perceptions and efficacy of OER and no-cost/low-cost implementations to students in course sections affected by ALG Transformation Grant projects. Continuous Improvement Grants are not included in this report, as the goal of these grants is the creation and revision of open educational resources—those results are visible in OpenALG and GALILEO Open Learning Materials repositories as well as the individual final reports for Continuous Improvement Grants.
Transformation Grants project teams submitted final reports at the end of their final semester, during which all implemented materials were taught within the course. Final reports included quotes from students and professors, data on student performance, drop/fail/withdrawal rates, and measures of student perceptions of course materials. Including savings estimates, this data meets all four requirements of the Open Education Group’s COUP Framework, measuring cost, outcomes, usage, and perceptions in each implementation.
Savings
During the course of their initial implementation, the projects in these three grant rounds affected an enrolled 20,294 students, and projects saved those students an estimated $3.2 million in student textbook costs in relation to their commercial equivalent—the purchase of new commercial materials as previously required within the course.
With nearly all teams indicating that these materials or other affordable materials will be used in future semesters, a high sustainability of these student savings is anticipated over at least the next academic year.
Grantee Experiences
Reporting project teams had positive experiences overall in teaching with their new materials. These final reports continue to support the idea that grants for open, no-cost, and low-cost materials adoption activities are valuable tools in building affordable and sustainable learning materials practices among faculty.
“Perhaps more importantly, for me, it caused me to spend a considerable amount of time familiarizing myself with the truly impressive array of materials freely available online which can be effectively used in an introductory human geography course.”
– Dr. Uli Ingram, Kennesaw State University
Project Leads indicated in reports that the activity of redesigning a course with affordable resources enhanced their teaching and learning experiences, including instances of pedagogy that only open resources could enable (Open Pedagogy).
"We also found that we both truly valued being able to spend more time interacting with students, and working problems with them, instead of allowing them to passively take notes."
– Dr. Kimberly Shaw, Columbus State University
COVID-19 Pandemic Complications
While the experience of participating in a Transformation Grant project was overall positive to team participants, dealing with the complications caused by the COVID-19 pandemic was an unpleasant experience for teams with projects ending in 2020.
"I cannot stress enough the impact the pandemic has on everyone this semester – students, instructors, administrators, and all their families and communities."
– Dr. Jung Ha Kim, Georgia State University
34 out of 57 teams reported encountering substantial barriers to a successful project that were caused by COVID-19. These barriers included instruction, student success, collaborations, extra confounding variables, scholarship, and qualitative research barriers.
Instruction Affected
More reports had standout quotes about COVID-19 being a barrier to effective instruction than any other complication associated with the pandemic, from the more obvious issues in shifting to online instruction to lesser-known issues such as sudden changes in class sizes or section enrollment.
"We, along with our students, had to adapt to this new way of learning. Many students struggled with this change, but we all adapted and did the best we could to help the students get through this very difficult time."
– Dr. Daniel Mancill, East Georgia State College
Student Success Affected
While the Student Learning Outcomes and Course-Level Retention sections will point to projects ending in 2020 reporting largely better or the same student learning outcomes in comparison to previous semesters, teams still noticed the barriers to success that students were encountering in their courses due to the pandemic.
"The effect of students being forced into online learning or hybrid environments, along with other personal challenges associated with the pandemic, may not have been the best for individual student success."
– Dr. Shoshana Katzman, Georgia Gwinnett College
Collaborations Affected
Due to the emergency online shift, many teams found themselves using new online meeting software and grappling with hardware issues, hindering plans to meet and collaborate on a project.
"Obviously, due to Covid-19, team members were restricted on making the project a smooth collaborative effort."
– Dr. Darren Broome, Gordon State College
Extra Confounding Variables
COVID-19 and the emergency shift to online instruction caused the findings of many project teams to be less attributable to the introduction of affordable materials; determining whether or not the Transformation Grant affected student learning or retention was at times impossible.
"The main co-factor that influenced 2020 results is the sudden transition of the traditionally-taught course into a remote asynchronous [one]. In addition, both students and professors have been continuously encouraged to practice leniency and flexibility in assessment, activity scheduling, and grading."
– Dr. Aselia Urmanbetova, Georgia Institute of Technology
Scholarship Affected
In previous final report summaries, many teams commented on the potential for future scholarship brought on by the success of a Transformation Grant project. This year, some teams commented on the diminishing possibilities for scholarship, particularly presentations, during the pandemic.
"I do not have any current plans to publish papers or give presentations about this project or OER. This is partly due to the fact that many academic conferences have been canceled or turned virtual due to Covid 19…"
– Dr. Uli Ingram, Kennesaw State University
Qualitative Research Affected
An unexpected barrier to completing projects during the pandemic was an increased difficulty in doing qualitative research, such as low response rates on surveys or a sudden shift to online instruction preventing focus group research.
"Of the 124 students in these sections, 18 participated on [the] survey questionnaire. The low completion rate was due to [the] COVID-19 pandemic."
– Dr. Zephyrinus Okonkwo, Albany State University
Student Satisfaction
Teams reported that students were highly satisfied with the affordability and ease of access with open textbooks and affordable materials, with every team that measured student satisfaction reporting an overall positive perception of the new resources in comparison to a traditional commercial textbook.
This is an improvement over the previous year, 2019, when 81% of grant teams reported positive student perceptions, with 19% reporting neutral perceptions.
"I feel as though everything needed was provided. The supplements helped the most since you personally wrote them tailored to what we needed to know for each test."
– Student of Dr. Brian Ray, Abraham Baldwin Agricultural College
Student Learning Outcomes
Comparing student performance in a course before and after an Affordable Materials Grant always includes taking multiple confounding variables into account, but the complications and barriers encountered during the COVID-19 pandemic made student learning outcomes even more difficult to attribute to a grant project alone. However, we can determine from the 2020 Final Reports that the implementation of affordable materials largely did not adversely affect the shift to online instruction during COVID-19. Overall reported changes in student performance were even more positive than with projects ending in 2019.
30 teams (65.22%) that measured changes in student learning outcomes reported positive significant changes to student achievement in comparison to control groups, previous semesters, faculty averages, and/or departmental averages, while 14 teams (30.43%) saw no significant changes to outcomes with the new affordable materials. Only two teams (4.35%) saw negative changes. This is a combined 95% of teams who experienced either a significantly positive or neutral effect on learning outcomes while using free and/or affordable materials.
This is an improvement over the previous year, 2019, when 11 teams (53%) reported positive changes in outcomes, with 7 teams (33%) reporting neutral changes and 3 teams (14%) reporting negative changes.
"Many students reported that their learning improved over the course of the semester and that they had to think more deeply about concepts than they usually did."
– Dr. Rachel Epstein, Georgia College and State University
Course-Level Retention
Comparisons of course-level retention rates during the COVID-19 pandemic are almost impossible to attribute to a grant project alone. However, we can determine from the 2020 Final Reports that the implementation of affordable materials largely did not adversely affect retention rates in the shift to online instruction during COVID-19.
Retention rates vary by institution, but most report DFW delta rates as Drop/Fail/Withdraw (others measure Grade D/Grade F/Withdraw). 20 teams (42.55%) reported positive significant changes, 20 teams (42.55%) reported no changes, and 7 teams (14.89%) reported negative changes.
More teams reported neutral/no changes in DFW rates in 2020 as opposed to the previous year, 2019, where 13 teams (61.9%) reported positive changes, 5 teams (23.81%) reported no changes, and 3 teams (14.29%) reported negative changes.
"...we suspect that COVID-19 has impacted DFW rates in our OER courses, thus muting the positive impact of the new textbook."
– Dr. Shane Peterson, Kennesaw State University
Lessons Learned
Each final report contains a section within the narrative document for lessons learned. From these 2020 reports, Affordable Learning Georgia has reached the following overall conclusions when implementing open, no-cost, or low-cost materials to replace a commercial textbook in a course.
As with previous years, OER implementation takes substantial planning in order to achieve goals within the deadline: Many teams with completed projects in 2020 reported time management and planning as a barrier to project completion, especially due to the pandemic. Some of these issues were also attributed to a lack of foreknowledge of the time it would take to complete major tasks and reach milestones.
In response to this trend of time management being reported as a barrier to completion, Affordable Learning Georgia revised the application process in Spring 2020 to include more detailed time management within action plans, containing achievable milestones and an estimate of the time needed to complete major tasks. Projects which used this new application process will start submitting Final Reports in 2021.
"The amount of work required to coherently combine course materials can be daunting; therefore it is important to plan for course development early on."
– Dr. Sherly Abraham, Georgia Gwinnett College
Confirm departmental commitment before the application process starts: Affordable Learning Georgia has prioritized the scaling-up of the use of affordable materials department-wide at institutions as a way to expand materials affordability system-wide. These projects typically involve the entire department committing to the use of affordable materials for at least the final semester, and most Departmental Scaling projects have ended successfully with sustained use of the new materials. However, two projects ending in 2020 saw a department not commit to the use of these materials; one due to administrative turnover, and one due to an assumed, not confirmed, departmental commitment.
In response to these issues, Affordable Learning Georgia revised the administrative review process to further confirm departmental commitments before accepting a Departmental Scaling application.
"A central recommendation for future large-scale OER adoptions at any institution is to be sure that there is sufficient buy in across the department faculty and administration to support this initiative."
– Dr. Daniel Farr, Kennesaw State University
Plan on ongoing development after the project: This year, teams often reported a need for expanding the resources they adopted, adapted, or created, including rewriting, making content adjustments, and creating new ancillary materials once the planned materials had been implemented in the classroom for a semester. This is not an undesirable outcome; the continued improvement of materials and instruction should lead to improved instruction and student learning in future semesters.
"Some portions of the textbook have been rewritten based on the text’s use in the classroom. In particular, some students found the concepts of “high contrast” v “high key” lighting to be confusing, so this portion was rewritten… some students wanted more clarification on lens types and camera angles, so new diagrams were developed in response."
– Dr. Yelizaveta Moss, University of North Georgia"
When evaluating the quality of materials or planning on the creation or adaptation of new materials, focus on the relevance of these materials to your students. Faculty assessments of quality often diverge on what quality means, often because of differences in subject matter or pedagogical techniques. Quality aspects to faculty include comprehensiveness, design issues, technical issues, accuracy issues, writing styles, and opinions about using print versus digital formats. In contrast, student quotes from ALG final reports converged in 2020, focusing on relevance as the most important indicator of a course material’s quality outside of the affordable or free cost.
Future projects should consider the relation of potential course materials to students’ overall experience with the course; for example, the connection between a textbook and the corresponding tests and quizzes, the connection between materials and the typical career path within a course’s corresponding major, or the connection between materials and the lived experiences of the students taking the course.
"It is more in line directly with what is taught and required to understand, and [it] had none of the unnecessary material that usually fills paid-for books/texts.”
– Student of Dr. Lydia Ray, Columbus State University
"I feel as though everything needed was provided. The supplements helped the most since you personally wrote them tailored to what we needed to know for each test."
– Student of Dr. Brian Ray, Abraham Baldwin Agricultural College
"I think I found [the readings] interesting because they connected to what I cared about most.”
– Student of Dr. Gregory Gullette, Georgia Gwinnett College
"From the instructors’ perspectives, collecting and organizing the learning materials ourselves not only enable us to better respond to the dynamic nature of the information technology field, but also give us the flexibility to customize the course content to better serve our students.”
– Dr. April Abbott, Abraham Baldwin Agricultural College
Conclusions
The COVID-19 pandemic affected higher education as a whole in 2020, and as expected, grant teams reported substantial barriers due to the pandemic. Sudden changes to instruction, learning, collaboration, and the ability to conduct research all had an impact on the success of Transformation Grant projects ending in 2020. These changes not only hampered the progress of projects and the implementation of materials, but they also made determining the effects of implementing affordable materials even more difficult.
Despite these complications and barriers, it is clear from the 2020 Final Reports that the implementation of affordable materials did not adversely affect student perceptions, performance, or course-level retention. Grant teams funded in these three rounds still largely saw overall positive or neutral changes to student performance and to DFW course-level retention rates, all while saving over 20,000 students over $3 million just within the time of the project, and mostly within only the final semester of instruction.
Students who gave positive reviews of the new materials in courses often commented on the relevance of the materials (to the course and to their overall career goals/lives) as a quality indicator, and future grant teams should consider relevance when planning the creation or adaptation of materials. On top of projects still being largely successful and seeing comparable outcomes while saving students money, faculty often remarked that they enjoyed examining their pedagogical methods and the relevance of their materials to their students and desired learning outcomes.
The sustainability and success of these projects despite the COVID-19 pandemic’s complications not only speaks to the viability and efficacy of affordable materials projects under emergency circumstances; it speaks to the expertise, resilience, collaboration, and dedication shown by the USG faculty and staff responsible for the planning and execution of these projects.
"The major lesson learned is that even during a worldwide pandemic, our faculty [have] ‘what it takes’ to transform the courses to no-cost."
– Dr. Scott Jacques, Georgia State University
Note on Analysis Challenges
Variation will occur with the number of semesters of data used and whether averages calculated were department-wide or instructor-specific. Confounding factors when measuring learning outcomes and retention efficacy can include differences in student composition between semesters, enrollment shifts, and organizational complications due to institutional consolidations.
Three uniform questions about student perceptions, student learning outcomes, and drop/fail/withdraw rates are given to all project leads within the final report as supplementary to their summaries of all research conducted. The interpretation of qualitative and quantitative findings, evaluations of statistical significance, and analysis of confounding factors are the responsibility of the faculty teams conducting the research.
In 2020, the COVID-19 pandemic exacerbated these challenges; not every team was able to conduct measures to answer the three uniform questions asked in the Final Report. These reports have “Not Measured” marked as their response. “Not Measured” responses are not included in the overall measures discussed in this summary report.