Evaluation Criteria for Peer Reviews

One of the goals of MERLOT is to develop and apply evaluation standards for its peer reviews of the learning resources in its catalog. These evaluation standards can also be used by faculty to:

• Select new materials to submit to MERLOT
• Review existing materials in MERLOT
• Provide requirements for the development of new materials for MERLOT.

1. Quality of Content
2. Potential Effectiveness as a Teaching-Learning Tool
3. Ease of Use

There are two general elements to quality of content:

1. Does the software present valid (correct) concepts, models, and skills?
2. Does the software present educationally significant concepts, models, and skills for the discipline?

• Content is core curriculum within the discipline. Core curriculum topics are typically covered to some degree in the introductory classes within the discipline and/or "Everyone teaches it" and/or it is identified as a core area by the discipline's professional organizations
• Content is difficult to teach and learn.
• Content is a pre-requisite for understanding more advanced material in the discipline

WARNING: This evaluation is the most difficult. Determining actual effectiveness requires actual use of the instructional software by real students and faculty. Evaluating POTENTIAL effectiveness is asking the Peer Reviewer to judge, based on his expertise as a teacher, whether the learning material is likely to improve teaching and learning given the ways the faculty and students could use the tool.

Sometimes the learning material being evaluated is "taken out of context", making it difficult to evaluate. The value of "modules" is their ability to be re-purposed for others’ use in different contexts. In evaluating the Potential Effectiveness for Teaching and Learning, it is CRITICAL to define the purpose of the learning materials. That is, one must contextualize the review. The MERLOT materials on Evaluation Standards and the Peer Review Reports emphasize this point. In performing a review, we use the following three questions to help define the pedagogical context.

1. What stage(s) in the learning process/cycle could the materials be used?
   1. Explanation or description of the topic/stating the problem
   2. Demonstration of the curriculum/exploration of the problem
   3. Practice using the curriculum/analysis of the outcomes from solving the problem
   4. Applying the curriculum to "new" problems/application of the outcomes to other problems
2. What is(are) the learning objective(s)? What should students be able to do after successfully learning with the materials?
3. What are the characteristics of the target learner(s)

An evaluation would be conditional on how the learning material was used. For example, "if the faculty using tool X in a supervised lab with freshman, reviewed the content beforehand, and had the students do tasks A and B, then the module should enhance students' learning. The reasons are..." All these issues and more are also represented in the Learning Assignment screens in MERLOT.

There are other general elements to effectiveness as a teaching-learning tool that MERLOT asks reviewers to consider:

1. Does the interactive/media-rich presentation of material improve faculty and students' abilities to teach and learn the materials?
2. Can the use of the software be readily integrated into current curriculum and pedagogy within the discipline?
3. Can the software be used in a variety of ways to achieve teaching and learning goals?
4. Are the teaching-learning goals easy to identify?
5. Can good learning assignments for using the software application be written easily?

The basic question underlying the ease of use standard is: how easy it is for teachers and students to use the software for the first time? Elements that affect ease of use include:

1. Are the labels, buttons, menus, text, and general layout of the computer interface consistent and visually distinct?
2. Does the user get trapped in the material?
3. Can the user get lost easily in the material?
4. Does the module provide feedback about the system status and the user's responses?
5. Does the module provide appropriate flexibility in its use?
6. Does the learning material require a lot of documentation, technical support, and/or instruction for most students to successfully use the software?
7. Does the material present information in ways that are familiar for students?
8. Does the material present information in ways that would be attractive to students?