PTA 130 PLAN OF EVALUATION

Improving Content Retention
An issue identified by both instructors and students is that there is a struggle with retaining/memorizing content from previous courses and the large amount of information within PTA130.  To take a proactive approach, instructors will create a document with tools, tips, and study techniques that students can utilize during their time in PTA130. It should be discussed in the Canvas Navigation video and be available on the main page and “PTA130 Resources” page of the related Canvas course.

Medical Terminology Test Out
PTA130 is an information-heavy didactic course with difficult medical terminology. Students who struggled with remembering medical terminology struggled in the course. Students entering the PTA program will be given a test to determine their existing body of terminology knowledge before entering PTA specific courses.

The test will primarily focus on the students’ knowledge of basic scientific and medical terms. It will consist of [ADD NUMBER and QUESTION TYPE] questions and will be conducted by the admissions department before the student starts term 3 of the Health Science portion of the PTA program. Students that score above [ADD PERCETANGE] on the test will be able to take PTA130 while those who score below will be entered into a remedial medical terminology course (most likely BIO112).  This remedial medical terminology course must be taken during term 3 of the Health Science portion of the PTA program.

Canvas Navigation
To acclimate student to how Canvas will be used in PTA130, the instructors will create a “tour” video. This video should include the layout of modules, how to find and submit assignments, how to use the discussion boards, where to find assessments and grades, etc. Canvas has been used sparingly, if at all in previous iterations of this course. Creating this video not only acclimates students to how it will be used during the course but creates accountability for the instructor to use the Canvas LMS as the main hub for all course content. This will eliminate the frustration students had when assignments, quizzes, and tests were sent through email.

Student Interview/Surveys
Since the PTA program population is small, all students will be candidates for either a survey or individual interview. The PTA director will provide the eLearning team with a contact list of the students and help to encourage student participation.

Students will be contacted via email and offered some P.I.T. swag and a chance to win a $50 gift card in a raffle for their participation. The eLearning team will select up to 5 interviewees. The team will also work with the PTA130 instructors to determine a set of questions that will help gauge the students’ existing body of knowledge, education experience prior to enrolling at P.I.T, and experience with the Canvas LMS.

Those who do not participate in an interview will be encouraged to take a survey with a similar set of questions to the interviews. While the interview questions will be very open-ended, the survey questions will be modified to revolve around 1 -5 Likert scale with some short-answers and room for comments.

Prior to the term starting the eLearning team suggested adding Canvas discussions, class activities, and case studies to integrate active learning into the course. The eLearning team also provided examples of each that could be used in the course.

After midterms, the eLearning team will schedule interviews with the PTA130 instructors. The focus of the interview(s) will be on current student grades, midterm grades, implementation of Canvas, and student focused activities. Instructors will be asked questions including:

• Are students utilizing the study techniques document made available at the start of the course?
  • If yes: Do you see improvement in grades compared to previous sessions of PTA130?
• The instructor’s comfortability with using Canvas at this point (i.e.: adding content, assignments, tests, grading, etc.)
• Implementation of Canvas discussions, class activities, and case studies during class
  • Number of each implemented during first half of the course
  • Reaction of their use (from student and instructor perspectives)
• Any concerns (with Canvas, student interaction, implementing activities, timing constraints, etc.)

Finally, the instructor will have the opportunity to discuss the positives and negatives they have seen through the term so far. This can include student grades, student engagement during class, if objectives are being met, and how to modify /adjust moving through the second half of the term for better student success.

Interview, Surveys, & Data Analysis

Student
The post instruction component will be a combination of previous information gathering methods including student interviews/surveys, instructor interviews, and reviewing student grades in relation to course objectives and outcomes.

Five students (preferably the same 5 from pre instruction) will be interviewed. Interview questions will focus on three main sections: the implementation of Canvas, unit goals and objectives, and the addition of active learning activities. Questions that will be presented on Canvas include how the student felt about:

• The organization and ease of finding materials on the Canvas page
• The ease of taking quizzes and exams and getting their grades back from instructor
• Their thoughts on the use of discussion questions on Canvas (beneficial or not?)

The next set of questions in this stage will be aligned more closely to the unit learning goals and objectives for the course. The eLearning team will work with PTA130 instructors to determine the specific questions and the format.

Questions about the implementation of active learning activities will focus on the students’ thoughts on if they were: helpful with reinforcing content, did not help, or have any suggestions to make the activities more meaningful. Once again, students will be rewarded with P.I.T. swag and a chance to win a $50 gift card in a raffle to encourage participation.

Instructors
PTA130 instructors will be interviewed like they were during midterm. The focus of the interview(s) will be on current student grades through the final, implementation of Canvas, and student focused activities. Instructors will have the opportunity to discuss observations and experiences that occurred through the end of the term. This can include student grades, student engagement during class, if objectives are being met, and how to modify /adjust to make the next iteration of the course more successful.

Course Evaluation
The eLearning team will use Watermark Course Evaluations & Surveys to distribute course evaluations to all PTA130 students. They will take the survey with a combination of different question categories that include:

• Student class experience
• Across the curriculum goals
• Administrative goals
• Student Services goals
• Course specific questions

The course specific questions are designated for each department head to determine if any additional questions are needed to gather information about learning goals/objectives. The eLearning team will work with the PTA department head to construct additional questions for the course evaluations.

Concluding the Process
All data from the pre-instruction and during instruction portions will be collected and analyzed by the eLearning team. The data will be presented in a well-designed PowerPoint. A meeting will be scheduled that includes the Dean of Academic Affairs, the Associate Dean, the college President, the Director of Student Services, and the Director of Admissions. The eLearning team will present the information using graphics that are clear and easy to understand. All participants will have the opportunity to provide feedback to the eLearning team after the presentation. Action items will be assigned to the appropriate party that are specific to their role.

Rapid Prototyping
The Learner Evaluation data collection methods will be combined with a Rapid Prototyping approach. Data from student interviews and surveys will be applied in conjunction with current instructional materials and learning objectives to create course related prototypes. Prototypes will be designed for a Canvas course design and interactive learning activities. Prototypes will be built using the web-based prototyping application Figma. The prototypes will consist of low, mid, and high-fidelity versions. The low-fidelity version will consist of content placement and information architecture. Mid-fidelity drafts include more interactions components like links, navigation, and buttons. The high-fidelity prototypes will add imagery, fonts, colors, etc. to present a near replica of the final products.

Active Learning Activities
A Plan of Intervention will be presented to the instructors of PTA130. The plan includes examples of learning activities that could be implemented during instruction.  The eLearning team will identify and build two activities to be tested in class.  After the activity is completed, students will be surveyed for opinions on the activity. The information will be used to refine the activities so that the instructors can use them when the course runs in other terms.

Usability Testing
Usability testing will be conducted throughout the Canvas course prototyping process. Instructors will test the low-fidelity (wireframe) prototype. Students and instructors will be requested to test the mid and hi-fidelity prototypes. Two instructors and three students will be selected to participate. Testing sessions will be recorded for future review and the participants will have to sign “permission to record” waivers. Recordings and notes will be reviewed to identify usability issues after each testing session.

Problems will be rated using the following 0-4 scale:

0 = I don't agree that this is a usability problem at all
1 = Cosmetic problem only: need not be fixed unless extra time is available on project
2 = Minor usability problem: fixing this should be given low priority
3 = Major usability problem: important to fix, so should be given high priority
4 = Usability catastrophe: imperative to fix this before product can be released (Nielsen, 1994).

During the low-fidelity stage, instructors will complete simple tasks that focus on text and content layout. The mid-fidelity stage will have the instructors and students complete tasks that focus on finding content on specific pages, click links and buttons to find any dead ends, and explore all navigation. The final stage of testing would involve more complex tasks that would mirror actual course assignments, quizzes, etc.

After all data has been reviewed, the eLearning team will address and solve any concerns from prototype testing. The eLearning team will then build out the course shell in the Canvas LMS. When finished, PTA130 instructors will explore the course and share any final edits they would like. The instructors will be given a final preview of the course. The eLearning team will continue to follow up with instructors to identify if further revisions are needed.

References
Nielsen, J. (1994). Severity Ratings for Usability Problems: Article by Jakob Nielsen. Nielsen Norman Group. https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/

The goals of the summative evaluation will focus on Canvas implementation and the use of active learning techniques in the course. The summative evaluation will determine if using Canvas as the main hub for course content resolved previous navigation frustrations. One other goal of the summative evaluation is to determine if the use of active learning strategies and interactive activities improved information retention for students.

Audience
The primary audience for the evaluation will be the PTA130 instructor(s) and the department head of the PTA program. Instructors should know if the changes they implement in the course are effective. A secondary audience for the evaluation will include the Dean and Associate Dean of Academic Affairs. These individuals will want to know if the implemented recommendations improved student grades and if they are achieving curriculum goals. Another audience group would include the Director of Student Services, Director of Admissions, and the college President. All groups will be concerned with program enrollment numbers and student dropout rate related to their own prospective department’s needs.

Evaluation Questions

• Were course materials consistent in design and easier to comprehend?
• Is Canvas being utilized as the main hub for all content, assignments, assessments, etc.?
• Did students have an easier time retaining PTA130 content with interactive activities being employed?
• Are lecture PowerPoints being used more as guide and not the main resource during instruction?
• Did the instructor use engaging, content related activities in their teaching?
• Were content specific discussions and case studies assigned during the course?

Data Collection Procedure
The majority of data collection procedures were discussed in detail within the Learner Evaluation and the Summative Evaluation sections. The procedures that were discussed include:

• Medical Terminology test out
• Student interviews
• Student surveys
• Watermark end of course surveys
• Instructor interviews
• Course grade reviews at the midterm and end of term
  

Analysis Procedure
The eLearning team will use comparative analysis of different data points. One analysis will focus on comparing grading data from the midterm and end of term instructor interviews. Another focus will be on student interviews/surveys from pre-instruction and end of term. This data will clarify potential student growth and the use of Canvas throughout the course. The final focus will be on analyzing the test-out data to demonstrate where students’ medical terminology knowledge was before enrolling in PTA130.

Evaluation Criteria and Judgement
Students must achieve a grade of 75% (C+) or higher to pass the PTA130 course. In addition, the layout of the PTA130 Canvas course will be reviewed. Module arrangement, assignments and tests, and resources will be examined to ensure Canvas is being used at the main hub for PTA130 content.