# PWR-118 — Expert judgement: learner worksheet Release: Revision 7T · Full Tutorial Edition Estimated time: Estimated 11 min reading; practical time is provider-set Difficulty: Intermediate Equipment: Basic stationery or digital tools — Qualified domain instructor, 20–40 representative resolved and de-identified cases and outcome key.; Probability scale, rationale sheet, calculator or scoring tool and calibration chart.; Declared domain, case-mix, conflict, AI/version and appeal record. Space: Desk / seated Automated structural checklist: 10 of 10 structural checks present Evidence quality/context: G4; Deep research depth Automated method-quality band: Comprehensive Method-rating basis: Structure coverage, instruction/check specificity, alternative distinctness, source troubleshooting, comparison depth and whether purpose/check fields are authored rather than derived. Editorial review: Pending manual editorial sign off ## Before you begin - [ ] I read the tutorial authority and stop conditions. - [ ] I have the required equipment/space or a declared accessible alternative. ## Canonical source context Expert judgement is domain-specific performance in a feedback environment. A qualified domain expert selects representative resolved cases, protects sensitive information, defines the outcome and teaches probability forecasts. The learner commits a probability before feedback, then checks calibration—whether 70% forecasts occur about 70% of the time—and discrimination—whether higher probabilities went to events that happened. Credentials, confidence and eloquence are not substituted for resolved accuracy. Target outcome: Under a qualified provider, the learner completes at least 20 resolved domain cases and produces a Brier score (mean squared probability error), calibration bands, discrimination and an error review for the declared case mix. ### Authority and setup conditions - Provider verifies scope, confidentiality, case representativeness and legitimate use. - Define the binary or categorical outcome and the resolution date before any forecast. - Agree that every case receives a probability, not only memorable successes. ### Required or supplied materials - Qualified domain instructor, 20–40 representative resolved and de-identified cases and outcome key. - Probability scale, rationale sheet, calculator or scoring tool and calibration chart. - Declared domain, case-mix, conflict, AI/version and appeal record. ## 1. Lock domain and case mix Name the exact task, population, time horizon and exclusions; the provider samples resolved cases from that frame. Why: Expertise can disappear outside its domain or on a selective case set. Success check: The case list matches the written inclusion rule. Accessible alternative: Ask the qualified provider to adapt positioning, cue modality, pace, interface or rest while preserving this step’s declared check: “The case list matches the written inclusion rule.” If an adaptation changes the measured task or configuration, record it as a different configuration rather than an equivalent attempt. Alternative relationship: Conditional equivalence requires target and configuration check Learner notes: ________________________________________________________________________________ Completed: [ ] ## 2. Forecast independently For each case, record a probability from 0 to 100 before seeing the outcome, peer view or AI advice. Why: Independent commitment preserves the learner’s judgement for calibration. Success check: A timestamped probability exists for every case. Accessible alternative: A spoken or tactile version of “For each case, record a probability from 0 to 100 before seeing the outcome, peer view or AI advice.” can rehearse the response sequence, but it removes the declared visual stimulus and is therefore a related activity, not an equivalent attempt at Expert judgement. To preserve the target, keep the authored visual stimulus and adapt only non-target access such as response input, instructions, breaks or support; retain this success check: “A timestamped probability exists for every case.” Alternative relationship: Related alternative changes trained or measured target Learner notes: ________________________________________________________________________________ Completed: [ ] ## 3. Write diagnostic reasons List the base rate, two case-specific cues and one reason the forecast could fail. Why: Reasons expose whether confidence comes from relevant information or persuasive story. Success check: Each forecast has base rate, cues and counterargument. Accessible alternative: Ask the qualified provider to adapt positioning, cue modality, pace, interface or rest while preserving this step’s declared check: “Each forecast has base rate, cues and counterargument.” If an adaptation changes the measured task or configuration, record it as a different configuration rather than an equivalent attempt. Alternative relationship: Conditional equivalence requires target and configuration check Learner notes: ________________________________________________________________________________ Completed: [ ] ## 4. Reveal and score After the block, reveal outcomes. Convert a forecast q% to p=q/100 and code the outcome o as 1 when it occurred or 0 when it did not. For each case calculate (p−o)²; the Brier score is the sum of those errors divided by the number of resolved cases. Why: A proper score rewards probability accuracy across all outcomes. Success check: Every valid case contributes to the declared score. Accessible alternative: Ask the qualified provider to adapt positioning, cue modality, pace, interface or rest while preserving this step’s declared check: “Every valid case contributes to the declared score.” If an adaptation changes the measured task or configuration, record it as a different configuration rather than an equivalent attempt. Alternative relationship: Conditional equivalence requires target and configuration check Learner notes: ________________________________________________________________________________ Completed: [ ] ## 5. Compare with a simple benchmark Score the same cases using the historical base rate or another provider-approved simple rule, then compare it with the learner without changing the case set. Why: Expert judgement adds value only if it can be separated from an appropriate baseline, not merely from guessing. Success check: The record shows learner and benchmark scores on identical cases and names where the learner helped or hurt. Accessible alternative: Ask the qualified provider to adapt positioning, cue modality, pace, interface or rest while preserving this step’s declared check: “The record shows learner and benchmark scores on identical cases and names where the learner helped or hurt.” If an adaptation changes the measured task or configuration, record it as a different configuration rather than an equivalent attempt. Alternative relationship: Conditional equivalence requires target and configuration check Learner notes: ________________________________________________________________________________ Completed: [ ] ## 6. Check calibration and discrimination Use the predeclared probability bands. In each band, average the forecast probabilities and separately divide occurred outcomes by resolved cases; their gap shows calibration. For discrimination, compare the mean forecast assigned to occurred cases with the mean assigned to non-occurred cases. Why: Calibration and discrimination are distinct qualities. Success check: The chart shows both reliability by bin and case separation. Accessible alternative: Ask the qualified provider to adapt positioning, cue modality, pace, interface or rest while preserving this step’s declared check: “The chart shows both reliability by bin and case separation.” If an adaptation changes the measured task or configuration, record it as a different configuration rather than an equivalent attempt. Alternative relationship: Conditional equivalence requires target and configuration check Learner notes: ________________________________________________________________________________ Completed: [ ] ## 7. Repair and retest The provider classifies base-rate neglect, cue misuse, overconfidence or data leakage, teaches one correction and supplies a fresh matched case block. Why: Specific feedback can improve a component without claiming generic expertise. Success check: The retest preserves case mix and documents one targeted repair. Accessible alternative: Ask the qualified provider to adapt positioning, cue modality, pace, interface or rest while preserving this step’s declared check: “The retest preserves case mix and documents one targeted repair.” If an adaptation changes the measured task or configuration, record it as a different configuration rather than an equivalent attempt. Alternative relationship: Conditional equivalence requires target and configuration check Learner notes: ________________________________________________________________________________ Completed: [ ] ## Reflection What changed? ________________________________________________________________________________ What remains difficult? ________________________________________________________________________________ What will I repeat, adapt, ask for help with, or stop? ________________________________________________________________________________ --- Completion of this worksheet demonstrates tutorial participation only. It does not establish capability, qualification, safety clearance, diagnosis, treatment or independent validation.