Quality assurance involves assessing the quality of the processes (the learning experiences) and the quality of the products (the graduates). You may want to refresh your memory on building effective learner support and the qualities required for support personnel that we covered in Module 2.

The two most familiar and most important tools for assessing your online course are surveys and evaluations completed by participants. Chances are that your institution already requires your learners to complete overall course evaluations at the end of term, and you are probably already accustomed to gleaning information from these evaluations for course improvement. In many instances, however, official evaluations may not be specifically designed with online courses in mind. Therefore, while they often provide important feedback, it can be helpful to supplement them to get a more accurate picture of how well your online design and teaching strategies are working.

This is where surveys for your online course come in. Nearly all Learning Management System (LMS) providers allow you to create your own surveys and polls within the system to get additional feedback from your learners. You may wish to survey your learners near the beginning of the term, or at some point in the middle, to see how things are going with the course design, especially if you think there may be things you are able to change on the fly in response to their feedback.

Regardless of whether you use in-term surveys, however, creating at least one end-of-term survey for your learners is highly recommended. For your end-of-term survey consider the following tips to focus on online-specific issues and questions, especially those that won’t be covered in your official course evaluations.

1. Ask about layout and information design in the course. Were things easy to find? Was the course intuitive to use?
2. Ask about third-party tools and other course technology if you are using them. Were there any technical problems or tool issues that learners ran into repeatedly? Can you use another tool or strategy, or improve your activity design to help reduce these issues going forward?
3. Ask about discussions and interactions to get a sense of what learners found valuable and why. Could you use better prompts or design different ways of building social presence in your course?
4. Ask about bottlenecks and problems that learners ran into with the content. Can some of your materials be made clearer? Can you spend more time during the next offer on certain difficult topics?
5. Ask about the accessibility of the content, in terms of if they were able to access all the required materials for learning in the course in a format that met their academic accommodation needs. Prompt for details regarding specific challenges or gaps.
6. Ask if the course provided a safe and accountable space for rigorous academic discussion. Prompt learners for positive experiences and situations that prevented them from full participation or where they may have felt uncomfortable. This can be a flag for further investigation.
7. Ask learners if they felt represented in the material and to justify their response. An open-ended question could illuminate areas where learners might feel alienated by the way the content is presented, or opportunities to incorporate more explicit diversity in learning materials (e.g., in content, images, readings, videos, questions posed, assignment tasks, or outputs).
8. Ask open-ended questions about what worked and didn’t work in the online environment. Give learners a chance to comment on anything that went well or poorly in the online learning environment. Much of the most valuable feedback you will receive comes from open-ended question fields.

In addition to text-based surveys, you may wish to consider having individual meetings with learners to get their perspective on the course or even using a focus group to get a more complete picture of what might be improved in your next offering. Note, however, that a focus group should only occur after final grades have been submitted.

Are the evaluative or assessment activities clear, feasible, relevant, accurate, and congruent with the objectives, content, and practical applications of the content?

1. Learners are given clear expectations and criteria for credit assignments. Instructors may require learners to submit an outline before learners proceed with their projects. Examples of assignments that meet the criteria may be included for students to review; however, this approach may limit creativity. If examples are provided, exemplars are provided that cover a range of achievements – what does an “A-rated” assignment look like and how does it differ from a “B-rated” assignment?
2. The number of assignments and their due dates are reasonable and take into account various holidays.
3. Evaluation and grading procedures are clear and explicit. Rubrics used in grading are provided.
4. Appropriate links to institutional policies on grading and evaluation are provided.
5. The relationships between course learning outcomes, evaluation strategies, course assignments, and the real world in which these skills will be used are evident to the learner.
6. The relationship between individual assignments and the final course grade is clearly specified.
7. A variety of feasible and content-relevant assignments or evaluative exercises are provided.
8. The evaluative exercises are relevant to the learners and the career or profession they may pursue.
9. Sufficiently detailed step-by-step instructions are provided for each evaluative exercise.
10. Guidelines for submitting assignments are provided.
11. Students are informed about the criteria that will be used to evaluate their participation in online activities such as discussion groups.
12. Learners are able to track and evaluate their own progress. Self-tests are similar to the final or formative evaluation instruments they will encounter.
13. Criteria and procedures for peer review and evaluation are clearly specified if these elements are included in the course.
14. Learners are informed about the consequences of plagiarism and the failure to properly cite copyrighted material.
15. Learners are told when they can expect to receive feedback from the instructor and that the feedback they receive will be timely, detailed, relevant, supportive, specific to the assignment, and will nurture reflection and critical thinking. The feedback should be motivationally effective.
16. Evaluation procedures are congruent with the objectives and reflect any priorities that have been established for the objectives.
17. Learners are given the opportunity to rate the module and provide feedback which can be used to improve the next iteration of the module.

This section covers checklists, rubrics, and guidelines for evaluating online programmes and course quality. The Selection/Application/Implementation Guide and Resources/Examples may be used to create a customised programme and course evaluation process for an institution’s specific needs. These artefacts may be used in a variety of contexts such as self-assessment, instructional designer review, peer review, institutional online programme effectiveness, and accreditation standards.

1. Identify fundamental, critical, and essential elements for your institution’s online courses, programmes, and institutional initiatives.
2. Use samples and resources to identify factors and elements related to the importance of your evaluation for high quality online course development and delivery.
3. List, categorise, and prioritise developed guidelines into personalised checklists, rubrics, or guidelines for your institutional needs, requirements, and implementation.

Considerations for Implementing Rubrics, Checklists, and Guidelines

Although all the listed criteria are important and should be highlighted, prioritising can assist faculty in their process for online content development. Once there is a comprehensive list of institution criteria, it is time to develop the rubrics, checklists, or guidelines. Consider how and when the criteria will be used. It is very probable that more than one of these rubrics, checklists, or guidelines will be created to meet your development needs. Ask yourself the following for context and usage:

• Will these rubrics, checklists, and guidelines be used to assess courses during the delivery of your professional development course?
• Will they be used for self-assessment, an instructional designer review, a peer review activity, or all of the above?
• Will you develop the rubrics, checklists, and guidelines as formal assessments that will prohibit or enhance an instructor’s ability to teach online?
• Will they be used more informally as self-assessment or peer review activities?
• Will elements be included to assess the course’s effectiveness after the delivery of the course online?

Although this section may seem an overwhelming list of considerations, all are important in the development of your criterion for online course effectiveness. It is critical/imperative to decide what is important at your institution and how it will be clarified to instructors in your professional development course.

As important as student engagement can be, student evaluations by themselves are not sufficient. Solicit peer review of specific resources, activities, or assessment strategies, your course structure, your communication strategies, or anything else about which you might have concerns. If you cannot find anyone in your school, department or college who is also teaching online you can ask school or district administrators, academic technology staff members, or faculty development centre staff members to identify prospective peer mentors for this type of feedback.

Peer Review Criteria

1. Comprehensiveness – covers all areas and ideas of the subject appropriately to meet course outcomes.
2. Content Accuracy – content, including supporting and supplementary material, is accurate, error-free, and unbiased.
3. Relevance and Longevity – content is up to date, but not in a way that will quickly make the course obsolete within a short period of time. The content is written and arranged in such a way that necessary updates will be relatively easy and straightforward to implement. The course uses OER where possible and does not overly rely on paywalled or hyperlinked content.
4. Clarity – uses accessible and clear language and provides adequate context for any jargon or technical terminology used.
5. Organisation and Structure – the topics are presented in a logical, clear fashion.
6. Diversity and Inclusion – reflects diversity and inclusion (e.g., shows people of different genders, races, abilities, and sexual orientations). It does not include insensitive or offensive language in these areas.
7. Recommendations.

Ensuring courses and online materials are fully accessible to all students is vital. It not only provides an equitable learning experience. All course content should be accessible to all students. Instead of waiting to create or reformat accessible content until you have a student with a documented need for it, create accessible content from the onset.

Accessible Tables

When creating a table, it is important to include identifiers, such as column and row headings and headers, where appropriate. Creating Accessible Tables covers the information you need to make tables accessible.

Accessible Images

Accessibility involves more than just providing alt text. Accessible Images discusses additional considerations such as colour, contrast, pixelation, and strobing.

Accessible Video, Audio, and Social Media

Review Create Accessible Video, Audio, and Social Media to learn about the steps needed to make video, audio, and social media accessible. Navigate to the links under the Video section to learn more about important considerations for this type of media.

Accessible PowerPoints

Microsoft PowerPoint can be used to create infographics as well as presentations. Explore Creating Accessible PowerPoint Presentations: How Do I Get Started? to find useful tips and instructions on how to make PowerPoint slides accessible.

Accessible Word Documents

Review Creating an Accessible Microsoft Word Document: Accessibility for MS Word to find useful tips and instructions on how to make Microsoft Word documents accessible. Be sure to pay attention to the sections on making Microsoft Word tables accessible, how to use the built-in accessibility checker in Microsoft Word, and how to convert a Microsoft Word document into an accessible PDF file.

Readability is the ease with which a reader can understand a written text. In natural language, the readability of text depends on its content (the complexity of its vocabulary and syntax) and its presentation (such as typographic aspects that affect legibility, like font size, line height, character spacing, and line length).

Why Is Readability Testing Important?

Web content authors should use the simplest language possible for the following reasons:

• Plain language will translate more easily for those who may wish to read the site in a different language.
• Plain language will be more accessible to those with lower levels of literacy.
• For a general audience, most readers will appreciate simpler language over the unnecessary use of complex words.

Though appropriate reading level is identified as a Level AAA requirement in Web Content Accessibility Guidelines, this is one Level AAA guideline that most public sites should aim to meet in order to reach the broadest possible audience.

The WCAG 2.1 guideline that is relevant to readability is:
When text requires reading ability more advanced than the lower secondary education level after removal of proper names and titles, supplemental content, or a version that does not require reading ability more advanced than the lower secondary education level, is available. (Level AAA)

Reading Level Test Tools

For any website intended for use by a general audience, the level of language used should not exceed that which could be understood by a “lower-level high school” reader. That is, language you might expect a grade 8 or 9 student, 12- to 15-years-old, to comprehend effectively on first reading. Some content developers may consider reading level an optional requirement, given WCAG 2.1, SC 3.1.5 Reading Level is a Level AAA success criterion. When writing for a general audience, however, it is always a good idea to write using the simplest language possible as a way to reach the broadest audience. Simpler, more readable language is appreciated by even the most educated of readers. And for those with cognitive- or reading-related disabilities, as well as those reading in a second language, simple language is often required to ensure they understand what is being said.

Key Point: Big words and complex language is not a sign of an intelligent writer. Writing with the simplest language possible, while getting the same message across, is a sign of a skilled writer.

There are many tools available on the Web for analysing readability.
Some examples of readability tools are:

• WebFX Readability Test Tool
• Test Document Readability
• Passive Voice Detector (also explore the other datayze writing tools)
• Readability Calculator
• Juicy Studio Readability Test

Often these tools will combine a variety of tests or formulas, to come up with a general grade-level score.

The 7 popular formulas that inform the level of readability include:

1. The Flesch Reading Ease formula
2. The Flesch-Kincaid Grade Level
3. The Fog Scale (Gunning FOG Formula)
4. The SMOG Index
5. The Coleman-Liau Index
6. Automated Readability Index
7. Linsear Write Formula

The content is designed with adaptability in mind, e.g., the course is not overly personalised with videos and PowerPoints that can’t be adapted without a lot of effort, nor does the course heavily rely on paywalled resources.

The course includes a variety of perspectives and reflects diversity and inclusion (e.g., gender, race, disability, sexual orientation). It does not include insensitive or offensive language in these areas.

Considerations have been made beyond accessibility: duration (not too long), quality (not hard to listen to), and relevance. The media used carries suitable permissions for reuse.

The presentation of the course is clear and easy to follow. Consistent terminology is used, and grammar and spelling are adequate. Links all seem to work; nothing is broken.

The Quality Assurance checklist items are intended to provide a measure of quality assurance for online courses in order to serve the needs of students. This checklist may be used to assess existing courses or those under development. It may also be used as a guide by faculty and online curriculum developers for planning purposes. You can explore the comprehensive checklist here.

Watch the video below to get a few tips on strategies for monitoring students’ progress. [Watchtime: 15.24 mins]