Affordable Learning Georgia

Affordable Materials Grants
Research Grants Final Report

Description

Research Projects will end with a Research Report due at the end of the Final Semester of the project. This will be a manuscript that is well-developed, concise, and suitable for publication. Research reports will be shared in an ALG repository under a Creative Commons Attribution License 4.0. Supplementary files, such as data sets, will not be shared in the repository.

While there is no specific page limit, award recipients are strongly encouraged to produce succinct Research Reports; these should be written with a broad public audience in mind, including faculty and professional staff of all disciplines. A template will be provided, but an outline is provided here for planning purposes.

Note: Online Submission Form

Once you have completed this template, to submit your Final Report, go to the Final Report submission form.

Follow the instructions on the webpage for uploading your documents. Based on receipt of this report, ALG will process the final payment for your grant. ALG will follow up in the future with post-project grantee surveys and may also request your participation in a publication, presentation, or other event.

General Information

Date: 12-12-2024

RG Grant Number: R-001

Institution Name(s): Kennesaw State University

Project Lead: Dr. Jeanne Beatrix Law

Team Members (Name, Title, Department, Institutions if different, and email address for each): Dr. Tammy Powell, James Blakely (GRA)

Course Name(s) and Course Numbers if applicable: ENGL 1101

Semester Project Began: January 2024

Final Semester: December 2024

Narrative

Give a narrative summary of your project. Include:

• Major goals and objectives, research questions addressed
• Research design, methods, analytical and data analysis techniques
• Findings and implications

In three rounds of ALG course design funding, Tamara Powell and Jeanne Law led teams who were awarded ALG grant funding to transform Composition 1101. The projects focused on the creation of three course templates for Composition 1101, utilizing low and no-cost readings, assignments, and handbook. The project aimed to enhance student learning experiences, reduce costs, and improve course consistency and accessibility for students at a large state university with a first-year enrollment of more than 8,500 students. In addition, we designed and piloted a seven-week OER/no-cost course infused with AI in Fall 2023, which was designed to measure students’ input with OER/AI materials and also to generate completion of ENGL 1101 in a truncated format. To draw multiple data points together and present relevant trends and findings as they impact student success, we conducted a research meta-study that analyzed quantitative student success in no-cost courses . Given KSU’s diverse student population, which includes 40% first-generation students, and 51% minority students, we believe our findings will be generalizable to different institutions in multiple contexts.

Our research questions include both quantitative and qualitative aspects of the study and aim to provide insights into the impact of no-cost materials and course templates on student learning experiences and outcomes in Composition 1101 at a large state university.

1. What is the impact of implementing OER materials, including the handbook, interactive reading, and multimedia materials, on student learning experiences in Composition 1101 at a diverse state university?

We conducted two surveys with students enrolled in OER ENGL 1101 courses. Our findings are explained below, as well as a summary of suspected trends we observed. Survey 1 focused on the course structure, weekly modules, preferences for additional activities, and familiarity with Open Educational Resources (OER). The findings are as follows:

• 1. A significant portion of students (87.5%) reported challenges in navigating the course structure, suggesting potential areas for improvement in course design.
  2. While 37.5% of students found the online modules helpful in preparing for assignments, 50% indicated they were only somewhat useful, pointing to a need for greater alignment between module content and course objectives.
  3. Students expressed diverse preferences for supplementary activities. 20% each suggested incorporating ungraded quizzes and writing activities, while 40% preferred no additional resources, implying varied learning needs within the cohort.
  4. Awareness of OER was limited, with only 12.5% of students indicating familiarity. This highlights a potential opportunity to integrate OER more prominently into the curriculum to enhance resource accessibility.

Survey 2 concentrated on assignment clarity, the use of the D2L learning platform, and the benefits of assignment examples. Key findings include:

a. Most students (89.1%) found the expectations for assignments easy to understand, indicating strong communication of course requirements.

b. All respondents (100%) reported using the D2L platform in this course, and nearly all agreed that its effective use supported their understanding of assignments.

c. An overwhelming majority (97%) expressed that seeing examples before writing their own assignments would enhance understanding, underscoring the value of clear and practical guidance.

d. Students praised detailed rubrics and examples provided by instructors but suggested improvements to D2L navigation and specific instructional videos, particularly on technical aspects like MLA annotation.

e. Most students (89.1%) found the expectations for assignments easy to understand, indicating strong communication of course requirements.

f. All respondents (100%) reported using the D2L platform in this course, and nearly all agreed that its effective use supported their understanding of assignments.

g. An overwhelming majority (97%) expressed that seeing examples before writing their own assignments would enhance understanding, underscoring the value of clear and practical guidance.

h. Students praised detailed rubrics and examples provided by instructors but suggested improvements to D2L navigation and specific instructional videos, particularly on technical aspects like MLA annotation.

Trends and correlations between Survey 1 and Survey 2

Survey 1 highlighted challenges with course navigation and low OER awareness, balanced by some positive impressions of specific activities and the integration of discussions. Survey 2 was overwhelmingly positive, with high marks for assignment clarity and D2L use, though minor concerns emerged regarding platform navigation and rubric detail. Together, the surveys suggest improving course organization and providing clearer examples and platform instructions could enhance student experiences across the board. Survey 2 demonstrates overall more positive impressions compared to Survey 1, with clear assignment expectations and effective instructor support. However, both surveys underline areas for enhancement, particularly in navigation and resource integration. Efforts to address these issues could lead to improved student experiences in future iterations of the course.

1. How do the cost-saving measures influence student pass rates in Composition 1101? We found that OER-based ENGL 1101 courses is similar to non-OER courses, with a small significant difference. Pass rates: fall semester: 50% hybrid 18%; (17% overall) spring semester: 28% (30% overall). Note: the spring pass rates are a reason to continue looking at ENGL 1101 courses for any given Spring semester.
2. How do AI-infused OER assignments appeal to student learning?

The surveys provide valuable insights into how students perceive Open Educational Resources (OER) and related assignments, as well as the potential for integrating AI to enhance learning. Survey 1: Familiarity with OER is notably low (87.5% of students unfamiliar), indicating that its potential remains untapped for most learners. However, those aware of OER expressed no significant objections, suggesting that increasing awareness and accessibility could be beneficial. Survey 2: Students appreciated detailed rubrics and clear examples for assignments, both of which are hallmarks of high-quality OER. Incorporating such resources into AI-driven systems could further personalize and clarify learning experiences.

a. Survey 1 identified that 87.5% of students struggled with course navigation. AI could enhance OER by offering dynamic, user-friendly platforms that adapt to individual student preferences and provide clear pathways to resources and assignments.

b. Students in Survey 1 suggested adding ungraded quizzes and writing activities. AI could enhance OER with interactive tools like adaptive quizzes, automated feedback on writing, and virtual discussion boards, making learning more engaging.

c. In Survey 2, 97% of students emphasized the value of examples. AI-driven OER could provide tailored examples, explanations, and even automated assistance, bridging gaps in understanding and boosting confidence in completing assignments.

d. Survey 2 highlighted student diversity in preferences, with requests for more explicit guidance and rubrics. AI-infused OER could address these needs by tailoring content and feedback to individual learning styles and pacing

e. Survey 1 shows that many students are unfamiliar with OER, meaning institutions need to introduce these resources effectively and communicate their benefits.

f. Students in Survey 2 found D2L navigation challenging. For AI-infused OER to be successful, platforms must prioritize intuitive design and seamless integration

Overall, our analysis of this data concluded that AI-infused OER assignments have the potential to greatly appeal to students by improving clarity, engagement, and accessibility. By addressing challenges in navigation and awareness, these tools could make learning more personalized and effective while fostering a positive attitude toward digital and open resources. The number of students taking this survey represented almost 40% of the students who took OER-infused template courses in a regular semester format, making these results generalizable.

1. What are the potential long-term effects on student retention and progression rates in Composition 1101 when no-cost courses are deployed in a first-year writing program?

We did not collect viable data to answer this question.

1. How does the project impact the accessibility of course materials for all students, including those with diverse learning needs and different college experience?

We did not have enough available data to determine the answer to this question. We could not gain access to this data.

1. Are there significant differences in the quality of student work, as assessed through textual and discourse analysis, in a seven-week AI-infused course vs. a full-semester course?
   1. Overall, 50% of students surveyed in the seven-week ENGL: 1101 course found the pacing "challenging but unmanageable," and 42.86% found it "challenging but manageable." Only 7.14% rated it as "easy but unmanageable."
   2. Half of students mentioned that the course content, no matter that it was OER had as much impact on their success as the pacing and scheduling in a short course.
2. What are the perceived benefits and challenges of an OER course template for Composition 1101 among instructors and students, including AI-infused assignments?
   1. There was only one instructor who piloted the seven-week course, and the anecdotal data shows that the instructor (Jeanne) had a vested interest in delivering research-based best practices in regards to generative AI writing instruction.
   2. We did not have enough reliable data to conclude whether instructors using OER based AI-infused content were satisfied with their experiences. We had two responses to our instructor reflection requests, which might indicate an unwillingness to participate. We have experienced similar issues with surveys on generative AI use among general education instructors. The two reflections we received are attached to this report. They indicate overall positive responses, with a key negative point being that the instructors were using free versions of GPT and could not engage with generative AI as much as they wanted to influence their assignment development.
3. How do students perceive their learning experiences in the pilot seven-week course (AI-infused) for Composition 1101 compared to full semester courses taught with OER?
   1. 71.43% of students believed the no-cost materials were as effective as traditional textbooks. None of the students disagreed, which indicates a strong positive reception to OER materials.
   2. While most appreciated the cost-free nature, some students found the format less user-friendly or underutilized.
   3. About 50% of students found the pacing "challenging but unmanageable," and 42.86% found it "challenging but manageable." Only 7.14% rated it as "easy but unmanageable."
   4. Several students indicated the course length was too short to adequately cover material, which might hinder deep learning or retention.
   5. Many expressed concerns about insufficient time to grasp concepts, while others found the format feasible with strong personal motivation.
   6. Students overwhelmingly praised the instructor for fostering inclusion (100% agreement) and providing substantial support, including personalized guidance, prompt communication, and making course materials accessible.
   7. Some students suggested alternative scheduling (e.g., avoiding Fridays or offering fully online options) and a preference for a longer course duration to better accommodate learning needs.
   8. A majority (92.86%) stated that AI positively contributed to their learning. Students recognized AI as a tool for revision, generating ideas, and improving essays.
   9. Students valued learning ethical and practical AI use, highlighting its relevance for future careers, particularly in fields like marketing.
   10. While most appreciated AI integration, one student expressed lingering uncertainty about its application.
4. How do instructors perceive student learning experiences in the OER Composition 1101 compared to non-OER courses?

As noted above, we only collected two instructor reflections, even after multiple attempts via email and personal outreach. These instructors taught five sections (130 students). We categorized our analysis by OER vs non-OER and a summary of both.

a. Reflections on OER Courses

Both instructors expressed a largely positive attitude toward OER courses, emphasizing how tools like the FYC GPT Assistant streamlined their teaching practices. One instructor noted how the AI tool saved significant time by revising assignment guidelines, creating rubrics, and generating quizzes tailored to TILT principles. This allowed them to focus more on pedagogical outcomes rather than administrative tasks. Another instructor highlighted how the tool supported scaffolding by generating model essays for assignments, which students found helpful for understanding expectations.

The integration of institutional expectations into the AI tool was particularly appreciated. This ensured alignment with course outcomes, giving instructors confidence in using the tool to create assignments and provide support to students. Additionally, the instructors valued the opportunity to explore new technologies, finding the process of learning how to use these tools both engaging and beneficial to their teaching practices.

Despite these advantages, the instructors acknowledged several challenges. Technical limitations of the free version of the FYC GPT, such as timeouts and restrictions on generating specific outputs like images, were a recurring issue. Additionally, one instructor expressed concern about the potential for AI tools to homogenize students' writing styles. They emphasized the importance of teaching students to preserve their unique voice while using AI for revisions.

Lastly, both instructors expressed a desire for additional training to unlock the full potential of the tool, including exploring possibilities for grading and more advanced literary analysis.

b. Reflections on Non-OER Courses

While non-OER courses also had their strengths, the instructors noted that these courses often lacked the efficiency and customization offered by OER tools. For example, when adapting textbook-based materials, instructors had to rely on manual processes, such as creating presentations and quizzes from scratch. This not only required more time but also limited the ability to quickly tailor materials to student needs. Additionally, feedback from students in non-OER courses indicated a need for more detailed rubrics and examples, a gap that OER tools could address more effectively.

However, the instructors did appreciate the flexibility of creating materials independently of technological constraints. In traditional courses, they were able to design assignments based on established resources, though this often came at the cost of efficiency.

c. Key Trends and Future Potential

The instructors viewed OER courses as a significant improvement in terms of efficiency, adaptability, and alignment with pedagogical goals. Tools like the FYC GPT Assistant allowed them to focus on student learning by automating many of the time-consuming tasks involved in course design. Non-OER courses, by contrast, were seen as more labor-intensive and less responsive to student needs.

Looking ahead, the instructors expressed optimism about expanding their use of OER tools. They were particularly interested in exploring new functionalities, such as AI-assisted grading and more dynamic student interactions. However, they also highlighted the importance of robust training to fully realize the potential of these tools.

Overall, we found that the reflections revealed a strong preference for OER courses due to their efficiency and effectiveness, with a recognition of the need to address current limitations through better technology and training.

2. Resulting Practice Recommendations

In a bullet-point list, briefly list any recommendations the team has for future practices in teaching and learning with open and/or affordable materials as related to this research.

• The results from the seven-week course survey show us that students appreciate OER materials, but that they want more interactivity in those materials. More user-friendly formats or training on how to effectively use these materials might improve student outcomes. We have developed a custom GPT from a PDF OER text to address this need.
• The results of the seven-week course survey and the higher DFWI rate indicate that a seven-week ENGL 1101 face-to-face course might not be the best modality and schedule to support student success, regardless of OER integration. Extending the course might address concerns about pacing and knowledge retention.
• Continued emphasis on ethical AI use and prompt engineering can equip students with transferable skills for academic and professional contexts.
• Offering varied modalities or schedules could increase accessibility and satisfaction.

3. Future Plans

Give a description of any scholarship work involving this research planned for the future, including publications, presentations, and conference papers.

(Research Grants teams are encouraged to publish their findings in peer-reviewed journals after the end of the project. Open Access (OA) publications are preferred to increase access to research materials, but due to varied OA options and tenure/promotion procedures within different disciplines and institutions, OA publishing is not a requirement.)

1. Given the percentage of generative AI usage in first-year writing courses, along with the positive attitudes from first-year students towards AI literacy and learning ethical generative AI use cases, we plan to pilot and scale a custom GPT as an ENGL 1101 interactive textbook/tutor based on the OpenStax Writing Guide OER textbook.
2. We plan to continue this work with ENGL 1102, with a Lumen OER textbook
3. We plan to submit our work to Macmillan Education for publication in the Bits on Bots series.
4. Once the 1102 work is completed, we plan to submit our work to Computers & Composition and seek out book chapters in series dedicated to AI-infusions in general education courses.

4. Supplementary Files

Give a description of any supplementary files provided to ALG, such as data sets or figures. Indicate whether these files can be shared with the public.

1. Seven-week student survey (not to be shared with public)
2. Summary analysis of AI survey distributed to all first-year students at KSU (raw data is protected because the project is on-going) – summary can be shared with public audiences