In the formal education system "norm based" assessments are of paramount importance for "formative" and "summative" testing, as well as for initial diagnostic testing.

In the days when the Internet became fully established, however, a huge market for "informal" education came into existence: MOOC (Massive Open Online Course) offerings [1] are an innovative approach to education. MOOC providers include non-profits, like Khan Academy, and for-profits, e.g., Udacity, Coursera  or Udemy.

A MOOC radically changes the usual teacher-learner roles, and the forms of assessment change with respect to in-person types of education settings.

Next to auxiliary use by learners in the secondary up to the quaternary education system, MOOCs are widely applied in larger companies for staff training, so called "eLearning programs", and by indiviual learners for the acquisition of additional skills. MOOC "completion certificates" are often used in business networks, e.g., LinkedIn or Xing, to advertise an individuals readiness for keeping-up with market demand. Other learners use MOOCs for self-paced learning in addition, or as an alternative to, fully self-directed autodidactic learning styles.

This text is, in itself, an artifact for a "formative" "criterion based" "supply-response" "peer reviewed" and "rubric based" assessment, supplied by the author as an assignment in the MOOC "Assessment for Learning" as offered by University of Illinois at Urbana-Champaign in the Coursera platform [2].

The Coursera platform uses different forms of assessment and is, in itself, an interesting opportunity for studying standardized tests "in the wild". As a learner with a company-sponsored Coursera account (and an appetite for diverse topics) the author has been subject to a wide range of standardized tests [3].

The Coursera "SkillSets" were designed as a tool for the management of employer-defined curricula. The learner starts with a "SkillSets score" of zero in each "skill" category defined in a curriculum and works towards employer defined "proficiency levels" that range from 0 to 500, e.g.,

• Conversant (1-50)
• Beginner (51-150)
• Intermediate (151-350)
• Advanced (351 - 500)

Based on a selected "SkillSets" item the Coursera MOOC platform "recommends" courses that the learner can take to score "SkillSets score points".

Instead of taking courses the learner may get an opportunity to "leap ahead" by taking assessment tests in a range of topics ("skills"), e.g., "Data Visualization", "Computer Programming", "Probability & Statistics" or "Mathematics". Such tests are available for a limited number of "SkillSets skills" and can be taken only twice.

A "leap ahead SkillSets skill" assessment can be described as "criterion referenced" and "formative oriented" as it attempts to sort a learner into a "proficiency level" class. This assessment can be used by the learner for starting a "self-referenced" learning journey but, since the "proficiency level" is visible to the employer, depending on the employer's policy, the assessment may also partially replace a regular "norm referenced" "summative assessment".

Artifacts in the assessment are generally of the "select response" type (sometimes "supply response in disguise" [3]), which means that the assessee has a fair chance of scoring in the "Beginner proficiency level" class by guessing - especially if some common sense is applied.

For the assessee in a "SkillSets" test it's soon obvious that the automated assessment selects items from a range of "reference courses" that are considered applicable to assessment of a "SkillSets" defined "skill" (the name of the course is provided as an auxiliary information to the assessee).

It's not clear if the difficulty of a course ("beginner", "intermediate" or "advanced" level) or the success frequency of a particular question in the context of the "reference course" is subject to calibration. It can be observed, though, that depending on the formal character of a "SkillSets skill" ("Data Visualization" vs. "Mathematics") the chance of succeeding (or excelling) by applying common sense is wildly different.

Select-response items from assessments ("assignments" or "quizzes" in Coursera parlance) in these courses are, apparently, randomly selected (the author was given the same question twice in the same assessment). The use of advanced algorithms, e.g., on-the-fly calibration with "computer adaptive testing" is unlikely.

[1] "Massive Open Online Course". Wikipedia, Wikimedia Foundation, 19 Mar. 2022, https://en.wikipedia.org/wiki/Massive_open_online_course
[2] "Assessment for Learning", Coursera Inc, 27 Mar. 2022, https://www.coursera.org/learn/assessmentforlearning
[3] Goeppel, Thomas. "The Coursera MOOC platform as an example of applied standardized tests". (2022, March, 27). CG Scholar, Assessment for Learning MOOC’s Updates. https://cgscholar.com/community/community_profiles/assessment-for-learning/community_updates/151793