Skip to end of metadata
Go to start of metadata

Definition


Develop or acquire an online course evaluation system that provides the following features:

  • Automated rostering and setup involving minimal manual work and limited room for human error
  • Ample, highly visible, and effective alerts to students about evaluation start/end dates and incomplete evaluations
  • Alert systems (text, email, BB Learn) to keep students, instructors and administrators aware of upcoming, open, partially-completed, and completed evaluations
  • Complete anonymity for survey participants
  • Design for mobile devices (phones and tablets)
  • Incentive modules (e.g. gift card giveaways) to help boost response rates
  • Robust reporting options (longitudinal data; real-time reporting; filtering by program, subject, term; etc.) – detailed reporting needs to be provided
  • Customizable messages, views, and surveys

Our primary goal is to increase response rates, which are often < 30%, in order to increase the value and usefulness of our online course evaluations.

Stakeholders

BO
PO: Sue Cassidy, Program Coordinator, Div. of Continuing & Professional Education, UMass Amherst
TO:
SM:
Others:

Technical Needs: 

  • Assistance with rostering and setup of evaluations for online courses each term. This will include scheduling evaluations for (a) courses with standard start/end dates and (b) courses with non-standard schedules (need to have)   
  • Integration with Blackboard Learn to provide adequate alerts to students within Blackboard courses  about (a) evaluation start and end dates and (b) incomplete evaluations (need to have)
  • Automated email alerts to students about (a) evaluation start and end dates and (b) incomplete evaluations (need to have)
  • Automated email alerts to instructors about low response rates so that they can remind students to complete evaluations (nice to have)
  • Large menu of standardized reports (to be specified)
  • Query option for non-standard reports
  • Occasional tech support  for any problems with rostering, set-up, survey administration, reporting as needed

Hosting

  • Need technical input about best place/method to host

Training

  • If in Blackboard Learn: internal training of small group of people on UMass Amherst campus who will be using the survey tool
  • Full, user-friendly, current documentation of procedures for:  evaluation administrators, students, instructors, academic administrators
  • If outsourced: training and documentation to be provided by vendor and customized as necessary for UMass

End-user support

  • Standard Blackboard Learn help desk support
  • Vendor technical support for any product-related issues

Background

UMass Amherst has used the OWLET system developed by UMass Amherst's Center for Educational Software Devt (CESD) for online course evaluations for many years. This system provides a basic means of capturing SRTI (Student Response to Instruction) data as well as some additional feedback required by the College of Nursing.

However, among the major drawbacks of the current OWLET system are:

  • Highly manual setup process
  • High likelihood of human error
  • Not integrated into Blackboard Learn
  • No mobile formatting
  • After initial prompt upon BBL login, relies on occasional email reminders to prompt students to complete evals. As a result, many students fail to notice that evaluations are due, or fail to remember to complete the evaluations
  • Very limited reporting options and those that do exist are not automated enough
  • Difficult to customize surveys (manual process)
  • No incentive module to help boost completion rates

CESD is currently working with the Office of Academic Planning and Assessment (OAPA) at UMA to move campus-based (day school) courses from paper to online evaluations using the OWL system.  As part of this process, CESD will be inviting input from CPE about needs related to evaluations of online courses.  Most day school courses use Moodle, while courses delivered through CPE, AND all campus-based classes offered through the College of Nursing and the Isenberg School of Management, use Blackboard Learn.  OAPA and CPE share common needs for higher online evaluation response rates, reporting, ease of use, etc. and we will be discussing these needs in future meetings. 

UMass Amherst pilot of CoursEval:

In August 2012, UMass Amherst piloted an outside product, CoursEval (www.connectedu.org), with a very small sample size (13 classes, 243 students). The product was not integrated into Blackboard and, at the time, there was no single authentication/log-in for students. It was difficult to prompt students to do their evaluations and inconvenient for students to log in to complete the evals. 

The response rate for the CoursEval pilot were abysmal:  26.3%.  Nevertheless,  CoursEval offered many features that would meet our needs for administration, communication, incentives and reporting related to course evaluations.

 

 

System/Service Status


EVALUATING

Systems to be Considered


Online Course Evaluation Requirements