MOOC Evaluation & Disruptive Technologies

The OLDS MOOC (Open Learning Design Studio MOOC) evaluation report by Simon Cross has just been published and I thought it was worth a quick look.  OLDS MOOC, based at The Open University, was a learning design MOOC that ran between January and March 2013. The course was structured to reflect a proposed process for design, and combined a number of design thinking methodologies such as the Ideo Design Thinking for Educators Toolkitinquiry based learning and educational design research.

As you’d expect, the report is thorough and systematic.  The data for the evaluation was a combination of more formal surveys as well as capturing information from the spaces being used for the course.  (Interestingly, some of the criteria for the evaluation were provided by those taking a pre- course survey.)  Here are some of the main points that I think come out of the report:

  • It’s hard to evaluate the experiences of those who mainly lurk or just passively learn from the materials, even though they seem to be a majority
  • Asking MOOC learners to use technologies with which they may not be familiar (in this case, Cloudworks) can present an additional challenge; this has implications for the use of open tools or technologies with which users may be less familiar
  • Only 50% of those who completed the post-course survey felt that their original learning goals were met; some participants seemed to have struggled with scheduling
  • Using traditional evaluation criteria, 63% said they were satisfied with the quality of the course and 72% were satisfied with the quality of facilitation
  • Between 1/3 and 1/2 of participants applied for badges that were awarded earlier in the course but the more advanced badges based in peer-review activity fared less well

In one respect, the main conclusion of the report is that there remain questions about how best to evaluate MOOC learning and in particular (i) how to form a balanced view of highly diverse experiences; and (ii) how to better understand and meet learner expectations and goals within MOOC.  Maybe we can summarise this with the view that there is a need for  MOOE (Massively Open Online Evaluation)…

You can download the report from

Another approach is found in Gilly Salmon’s recent suggestion that we should evaluate MOOCs “by their capacity and capability to create positive and successful change for universities of the future” where MOOCs are the ‘tugboats’ that can change the direction of the slow-moving university ‘supertankers’.  I’m not sure this is quite right, though: if the only measure is disruption then this can be achieved in a number of ways (not all of which are desirable).  Our own hypotheses tend to be framed in terms of the complementarity of open and traditional education models though of course the question remains open.  But perhaps the crises affecting education are such that the idea of disruption becomes appealing for its own sake.  Here’s what she has to say:

What might the qualities of this disruption be? Well, for me the constructive ones are pointing to the fuller development and deployment of open education resources, of the appreciation of the potential and reach for huge scale learning, addressing and solving challenges of very large numbers of participants, global reach, accessibility and participation and the enormous advantages of flexible, entirely digital learning provision. There’s also something about learners and participants determining their own choices and pathways, and following their own rather than the providers’ motivations, outcomes and determinants. But business cases and credentialing… well they are still in the murk, with only the odd lighthouse shining. Probably not clearly enough to turn the supertankers yet. The serious negative disruptor for me is the strange and mysterious view of the pedagogy of the many MOOCs based on face-to-face knowledge transmission. As others have pointed out, it’s not as if we don’t know some of the key qualities of learning online… truly we do.

My first thought in response was that MOOC designers are generally aware of good principles of design in online learning although perhaps I am being rather optimistic!  It seems to me that there is nonetheless a tension between the idea of the autonomous learner (described more fully in my paper with Marcus Deimann) and the case for credentialled learning which MOOC providers can monetize.  It’s not clear to me where the real potential of MOOCs lie here.  Perhaps it is this kind of confusion that makes MOOC so hard to evaluate.

6 thoughts on “MOOC Evaluation & Disruptive Technologies

  1. Pingback: MOOC, evaluación y tecnologías disruptivas! | juandon. Innovación y conocimiento

  2. I think we have come up with a scenario of emotional/affectionate responses to business/skills oriented, for profit/money making – fast track return on investment, cheap HE, and for effectiveness and efficiency model of education. There have been proven model of linear design in online education in closed LMS and structured education with articulated learning outcomes. However, when open education is now opened to the global environment, it seems that prescriptive design in MOOCs (as in many xMOOCs) trumps over emergent, adaptive and personalized design in MOOCs. Why? Surely, learners and educators from developing and under-developed countries are craving for world best quality education (via xMOOCs) as offered by elite HE institutions. There are lots of unknowns and assumptions in between, as regards to the intention, motivation and readiness of these participants in xMOOCs. I suppose the use of an reductionist approach towards evaluating MOOCs would only provide part of the complete picture of how MOOCs would aid in building HE and universities. As we have already seen surveys after surveys, with similar patterns of participation (i.e. 10 – 20 % active engagement at most, with majority being lurking or auditing, and a significant number of participants having reservations on the “instructivist approaches” especially for those who are more aligned to the innovators and connectivists/constuctivists. I like your metaphor of tug boats (for MOOCs). My concerns are: “if MOOCs (like a Massive online Train” are turning at high speed, what would happen?” See the latest train de-railing disaster in Spain. I am positive about MOOCs, only that each university MUST FIND their own vision and mission that would envision and grow HE, but they are now at the cross-roads of taking the track of MOOCs.

  3. Pingback: MOOC Evaluation & Disruptive Technologies | Web 2.0 PT

  4. Pingback: MOOC Evaluation and Disruptive Technologies | Flexible Learning

  5. Pingback: MOOCS, yo los quiero escalables y distribuidos y ¿tu? | juandon. Innovación y conocimiento

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s