For an exam to meet rigorous testing industry standards, a passing score (cut score) cannot be arbitrarily selected. Scores must be based on a method that supports the overall testing goals of validity, reliability, and fairness. Ultimately, it must differentiate those who met the standards from those who did not.
There are a variety of ways to establish a passing score using standard-setting principles including beta, bookmarking, and Angoff. When setting the initial cut score, Scaled Agile, Inc. applies a modified-Angoff item-centric approach and then monitors exam performance to ensure reliability and validity in the established score.
How it works
After exam questions are written and reviewed by a group of subject matter experts (SMEs), each question (item) is scored based on the exam’s overall definition of a minimally qualified candidate. This typically comprises a combination of factors, including the targeted level of competency (Scaled Agile uses a 1 to 5 scale similar to the Dreyfus Model) and criteria for scoring (p-value = 0 – 100).
Competency levels, and the definition of a minimally qualified candidate, are critical to the Angoff process. A 75 percent cut score for a skill level 4 (advanced) is very different from a skill level 2 (novice). This skill level is determined before the first exam question is even written, ensuring consistency throughout the process.
SMEs are not given a predetermined cut score before the Angoff process takes place. The scores themselves allow the natural cut score to emerge. This is why some Scaled Agile exams have a cut score of 75 percent and others are at 78 percent.
SMEs are essential to the success of this scoring process. Without introducing their own subjectivity or bias about what the target candidate would know versus what they should know, they must keep the target candidate in mind. Should indicates a bias. SMEs score the items without any discussion, and the answers to the questions are not provided until after the scores are recorded. Providing the answers at the same time tends to bias the scores. Any score ranges that are greater than 30 are reviewed as a group, and any adjustments are made accordingly.
Sample data – partial
Item | SME #1 | SME #2 | SME #3 | SME #4 | SME #5 | SME #6 | SME #7 | Average Angoff | Max | Min | Delta | Status (>30, check) |
456.1.1.1 | 80 | 90 | 70 | 90 | 90 | 90 | 90 | 86 | 90 | 70 | 20 | - |
456.1.1.2 | 90 | 90 | 80 | 75 | 90 | 60 | 90 | 82 | 90 | 75 | 15 | - |
456.1.1.4 | 85 | 70 | 90 | 70 | 80 | 70 | 50 | 74 | 90 | 50 | 40 | CHECK |
456.1.1.5 | 60 | 60 | 70 | 60 | 60 | 60 | 70 | 63 | 70 | 60 | 10 | - |
456.1.1.7 | 50 | 70 | 50 | 45 | 60 | 50 | 70 | 56 | 70 | 45 | 25 | - |
456.1.1.8 | 90 | 90 | 95 | 80 | 80 | 80 | 80 | 85 | 95 | 80 | 15 | - |
456.1.2.1 | 95 | 70 | 70 | 60 | 90 | 80 | 70 | 76 | 95 | 60 | 35 | CHECK |
Average > | 79 | 77 | 75 | 69 | 79 | 70 | 74 | 75 |
|
|
|
|
After an exam is published, the overall pass rate should track with the established cut score. So, if the cut score is set at 75 percent, then the actual pass rate should be plus or minus a few percentage points of 75 percent. An in-service exam review is part of standard exam maintenance. During this review period, adjustments to non performing items are made.
So, if you’re wondering what the current pass rate of a Scaled Agile exam is, just look at the cut score!