[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog-article-uzghodzhenist-otsiniuvannia-rubryky":3},{"uuid":4,"title":5,"subtitle":6,"content":7,"preview":415,"date_published":421,"date_pinned":421,"img":422,"alias":423,"is_published":424,"is_pinned":425,"type":13,"category":426,"category_name":427,"category_alias":428,"background_color":429,"preview_icon":13,"category_value":430,"external_link":13},"46a8bb2d-297e-4fff-b546-7eadda5f4af7","What Is Inter-Rater Agreement in Assessment, and How Do You Check It?","How to establish whether teachers apply a rubric consistently. We explore independent scoring, disagreements, metric selection, calibration, and a repeat pilot.",{"type":8,"content":9},"doc",[10,18,23,28,36,41,46,51,57,62,67,72,78,83,88,93,99,104,109,114,120,125,130,135,141,146,160,173,179,184,189,194,200,205,210,215,221,226,231,236,242,247,252,257,263,268,273,278,284,289,294,299,305,310,315,320,326,331,336,341,346,351,356,362],{"type":11,"attrs":12,"content":14},"paragraph",{"align":13},null,[15],{"text":16,"type":17},"One teacher sees a confidently completed project; another returns it for revision. Students receive two assessments of the same work and wonder whom to trust. The cause may not be inattentive assessors. The rubric leaves too much room for interpretation: 'sufficiently argued,' 'well presented,' or 'shows understanding of the subject.'","text",{"type":11,"attrs":19,"content":20},{"align":13},[21],{"text":22,"type":17},"Describing criteria does not itself guarantee consistent assessment. Check how different people apply them to actual work. This helps before launching a cohort, when bringing in new tutors, and after substantial assignment changes. It reveals where judgments diverge and what needs changing: wording, examples, expert preparation, or the learning task itself.",{"type":11,"attrs":24,"content":25},{"align":13},[26],{"text":27,"type":17},"The process below is intended for an online school's internal pilot. High-stakes certification needs a specifically designed reliability analysis involving a measurement specialist. A small set of work provides useful signals but does not establish universal rubric reliability for all students, teachers, and programs.",{"type":29,"attrs":30,"content":33},"heading",{"align":13,"level":31,"id":32},2,"heading_2_1",[34],{"text":35,"type":17},"First Define the Decision That Depends on the Assessment",{"type":11,"attrs":37,"content":38},{"align":13},[39],{"text":40,"type":17},"Assessment serves different purposes. In a practice exercise, it helps choose the next step; in a final project, it determines course completion; in professional certification, it may affect permission to work. Acceptable disagreement risk differs across these situations. Do not impose identical requirements merely because the interface displays the same familiar score.",{"type":11,"attrs":42,"content":43},{"align":13},[44],{"text":45,"type":17},"Record the assessment unit: an individual criterion, total score, or category such as 'pass \u002F revision needed.' The analysis depends on this. Two experts may rank work almost identically while systematically assigning different scores. This matters differently when selecting the best work versus applying a strict threshold. Define the evaluation goal before choosing a statistic.",{"type":11,"attrs":47,"content":48},{"align":13},[49],{"text":50,"type":17},"Agree on a practical disagreement limit, too. For example, a hypothetical school might separately review every case with different pass decisions even when total scores differ only slightly. This is an internal rule for a specific process rather than a general standard. It should reflect error consequences and the possibility of reassessment.",{"type":29,"attrs":52,"content":54},{"align":13,"level":31,"id":53},"heading_2_2",[55],{"text":56,"type":17},"Separate Reliability from Correctness",{"type":11,"attrs":58,"content":59},{"align":13},[60],{"text":61,"type":17},"Complete agreement between two teachers does not mean they assess the intended competence. Both may reward attractive slides instead of a justified decision. Reliability concerns consistency under defined conditions; the validity of interpreting results needs separate evidence. First compare the rubric with learning goals, then test its application.",{"type":11,"attrs":63,"content":64},{"align":13},[65],{"text":66,"type":17},"Ask of every criterion: which observable part of the work does it describe? If answering requires guessing a student's character, effort, or future success, clarify the criterion. Assessors should reference a passage or an action they actually observed. This makes subsequent discussion concrete even when professional judgments differ.",{"type":11,"attrs":68,"content":69},{"align":13},[70],{"text":71,"type":17},"Do not try to remove all complexity from professional tasks. Analytical reports or educational projects cannot always be reduced to formal checkboxes. A rubric makes the basis of judgment visible. It should help explain decisions rather than create an illusion of mechanical precision where reasoned expert judgment is required.",{"type":29,"attrs":73,"content":75},{"align":13,"level":31,"id":74},"heading_2_3",[76],{"text":77,"type":17},"Select Work That Actually Tests the Criteria",{"type":11,"attrs":79,"content":80},{"align":13},[81],{"text":82,"type":17},"Do not limit the pilot to the best examples. Include strong, weak, incomplete, and borderline work, along with different permitted approaches. Uneven profiles are especially useful: convincing analysis with poor structure, or a well-presented product with an incorrect conclusion. These reveal whether experts distinguish criteria.",{"type":11,"attrs":84,"content":85},{"align":13},[86],{"text":87,"type":17},"Participants should know the sample's provenance. If you deliberately collect many borderline cases, do not present results as an assessment of the cohort's ordinary distribution. If you use only graduates' work, conclusions do not automatically apply to beginners. Record selection methods, student levels, assignment versions, and the evidence available.",{"type":11,"attrs":89,"content":90},{"align":13},[91],{"text":92,"type":17},"Remove names and earlier grades when unnecessary for the decision. Knowing a student's effort or the course author's opinion can unintentionally influence assessors. Keep conditions identical: the same file, rubric version, and access to appendices. Otherwise, some disagreements will arise from different materials rather than different application of criteria.",{"type":29,"attrs":94,"content":96},{"align":13,"level":31,"id":95},"heading_2_4",[97],{"text":98,"type":17},"Run the First Round Independently",{"type":11,"attrs":100,"content":101},{"align":13},[102],{"text":103,"type":17},"Each expert assesses without discussion and records brief reasons as well as scores. Ask for specific evidence: a page, calculation section, or demonstration step. 'Not sure' is useful too: it identifies where level descriptions provide insufficient guidance. Do not force feigned certainty for a tidy table.",{"type":11,"attrs":105,"content":106},{"align":13},[107],{"text":108,"type":17},"Record initial results before the group meeting. If scores change during discussion, retain both versions. An answer agreed through negotiation describes moderation rather than independent inter-rater agreement. Mixing these datasets produces attractive statistics that do not reflect everyday practice when tutors work independently.",{"type":11,"attrs":110,"content":111},{"align":13},[112],{"text":113,"type":17},"The WeStudy LMS platform can form part of the learning environment for this pilot, with a separate log recording scores and explanations. Choose the data-collection method after checking available settings. Reliability depends on procedure and evidence quality, so a digital tool does not replace an independent first round.",{"type":29,"attrs":115,"content":117},{"align":13,"level":31,"id":116},"heading_2_5",[118],{"text":119,"type":17},"Start with a Clear Map of Disagreements",{"type":11,"attrs":121,"content":122},{"align":13},[123],{"text":124,"type":17},"Build a 'work — criterion — expert' table. Identify exact matches, adjacent-level differences, and opposing decisions. An overall average can hide errors in opposite directions. If one teacher is stricter about structure and another about reasoning, totals may look similar without actual agreement.",{"type":11,"attrs":126,"content":127},{"align":13},[128],{"text":129,"type":17},"Examine systematic strictness separately. Does someone almost always score lower? Is one criterion used as an overall impression of the whole work? Is a particular scale level almost never selected? These observations do not assign blame. They help formulate explanations to test in the next round.",{"type":11,"attrs":131,"content":132},{"align":13},[133],{"text":134,"type":17},"A simple percentage of exact matches is easy to explain but insufficient for every task. Matching 'pass' decisions in a group where almost all work is clearly strong says little about borderline cases. Present the number of works, category distribution, and disagreement types alongside the figure. Readers should see the basis for the conclusion.",{"type":29,"attrs":136,"content":138},{"align":13,"level":31,"id":137},"heading_2_6",[139],{"text":140,"type":17},"Match the Metric to the Scale and Design",{"type":11,"attrs":142,"content":143},{"align":13},[144],{"text":145,"type":17},"Kappa-family coefficients are sometimes used for categorical decisions; ordered categories may suit a weighted version with justified weights. The choice depends on the number of assessors, data structure, and research question. Do not treat ordinary correlation as universal evidence of agreement: two score sets rising together does not exclude a constant offset.",{"type":11,"attrs":147,"content":148},{"align":13},[149,151,158],{"text":150,"type":17},"Numerical ratings can be analyzed using measures such as the intraclass correlation coefficient, ICC. The ",{"text":152,"type":17,"marks":153},"methodological paper by Koo and Li",[154],{"type":155,"attrs":156},"link",{"href":157,"title":13},"https:\u002F\u002Fpubmed.ncbi.nlm.nih.gov\u002F27330520\u002F",{"text":159,"type":17}," explains why the model, measurement type, and definition of agreement must be specified. A single expert's assessment and an average of several ratings are different situations. Absolute agreement also differs from consistency, which allows some systematic differences.",{"type":11,"attrs":161,"content":162},{"align":13},[163,165,171],{"text":164,"type":17},"Here, statistical terms provide orientation rather than a ready-made calculation recipe. Involve an appropriate analyst to choose the ICC form, handle missing data, and assess uncertainty. ",{"text":166,"type":17,"marks":167},"Updated methodological guidance",[168],{"type":155,"attrs":169},{"href":170,"title":13},"https:\u002F\u002Fpubmed.ncbi.nlm.nih.gov\u002F36048052\u002F",{"text":172,"type":17}," separately addresses incomplete designs. Do not choose a coefficient after seeing results merely because it looks best.",{"type":29,"attrs":174,"content":176},{"align":13,"level":31,"id":175},"heading_2_7",[177],{"text":178,"type":17},"Do Not Turn One Number into a Quality Certificate",{"type":11,"attrs":180,"content":181},{"align":13},[182],{"text":183,"type":17},"A metric makes sense alongside a confidence interval, sample description, and assessment conditions. Uncertainty can be large in a small pilot. Convenient thresholds from methodological publications should not be applied mechanically to every educational task. Acceptability depends on error costs, scale of use, and student support arrangements.",{"type":11,"attrs":185,"content":186},{"align":13},[187],{"text":188,"type":17},"Supplement statistics with specifics: which criteria produce most disagreements, how many cases change the final decision, and what the team plans to fix. School leaders often find this more practical than an impressive unexplained coefficient. Numbers should support decisions rather than close discussion of assessment weaknesses.",{"type":11,"attrs":190,"content":191},{"align":13},[192],{"text":193,"type":17},"If a rubric is used across subjects, test transfer separately. High agreement on short analytical memos does not establish the same criterion performance on video presentations. Changes in format, audience, or experts may require another check. Keep conclusion limits beside results so they survive management summaries.",{"type":29,"attrs":195,"content":197},{"align":13,"level":31,"id":196},"heading_2_8",[198],{"text":199,"type":17},"Turn Discussion into Clearer Descriptors",{"type":11,"attrs":201,"content":202},{"align":13},[203],{"text":204,"type":17},"During calibration, ask experts to show evidence before naming a score. This reduces disputes over personal authority. If one sees 'complete justification' and another only a list of claims, compare evidence requirements. The rubric may never explain what counts as a sufficient connection between data and conclusion.",{"type":11,"attrs":206,"content":207},{"align":13},[208],{"text":209,"type":17},"A refined descriptor should show an observable difference between adjacent levels. Instead of 'sufficient analysis,' require comparing two possible explanations and identifying a limitation of the chosen one. Do not add requirements students never saw in the assignment. If a criterion changes the expected product, update instructions and examples, and define a fair transition for the current cohort.",{"type":11,"attrs":211,"content":212},{"align":13},[213],{"text":214,"type":17},"Create a short set of annotated benchmark work. For borderline examples, explain both the accepted level and why the adjacent one was rejected. Benchmarks should not become the only permitted answer. Show different valid solutions; otherwise, assessors will score resemblance to a template rather than performance quality.",{"type":29,"attrs":216,"content":218},{"align":13,"level":31,"id":217},"heading_2_9",[219],{"text":220,"type":17},"Test the Result on New Examples",{"type":11,"attrs":222,"content":223},{"align":13},[224],{"text":225,"type":17},"After revising the rubric, run another independent round using different work. Reassessing the same examples immediately after detailed discussion may reflect memory of the agreement. A new set better tests whether experts can apply the rule to an unfamiliar situation. Compare initial and updated results while accounting for sample differences.",{"type":11,"attrs":227,"content":228},{"align":13},[229],{"text":230,"type":17},"Do not promise statistical improvement without sufficient grounds. An internal pilot can yield a more modest but useful conclusion: a specific ambiguity disappeared, experts explain a borderline level consistently, or fewer decisions require moderation. State when data are insufficient for reliable generalization. Practical value does not depend on the boldness of a claim.",{"type":11,"attrs":232,"content":233},{"align":13},[234],{"text":235,"type":17},"Record the rubric version, introduction date, and changes. Do not combine grades under different rules into one achievement measure without explanation. Students should know which version applies to their work. Teachers need a short change log so new descriptors do not disappear among work-chat messages.",{"type":29,"attrs":237,"content":239},{"align":13,"level":31,"id":238},"heading_2_10",[240],{"text":241,"type":17},"Maintain Agreement in Ordinary Cohorts",{"type":11,"attrs":243,"content":244},{"align":13},[245],{"text":246,"type":17},"Retain sample double marking even after a successful pilot. Select work spanning different levels and new response types. Set frequency according to risk and resources without presenting an arbitrary number as universal. More appeals or a new assessor warrants additional review rather than proving the whole course is defective.",{"type":11,"attrs":248,"content":249},{"align":13},[250],{"text":251,"type":17},"Define a route for disagreements: who moderates, what they review, and how they explain the final decision. Moderators need the work, current rubric, and reasons for initial scores. 'Take the average' does not always solve the problem, especially when one expert identifies a critical error that strong presentation elsewhere cannot compensate for.",{"type":11,"attrs":253,"content":254},{"align":13},[255],{"text":256,"type":17},"If the WeStudy LMS platform is used, connect available course materials with current assessment instructions and maintain moderation rules in the agreed workspace. Test this route from a new tutor's perspective. They should find the right version without the author's spoken prompts; otherwise, consistency remains dependent on a few experienced people being present.",{"type":29,"attrs":258,"content":260},{"align":13,"level":31,"id":259},"heading_2_11",[261],{"text":262,"type":17},"Examine Three Different Causes of Disagreement",{"type":11,"attrs":264,"content":265},{"align":13},[266],{"text":267,"type":17},"The first cause is ambiguous wording. Both experts see the same facts but interpret 'independently' differently. One permits a reference book; the other sees it as inadequate mastery. Clarify performance conditions. The new wording should appear in the assignment students see before starting, as well as teachers' working documents.",{"type":11,"attrs":269,"content":270},{"align":13},[271],{"text":272,"type":17},"The second cause is different recognition of evidence. The criterion is clear, but one assessor missed a relevant passage or did not recognize a valid alternative method. Examples, subject training, and a better submission format help here. Rewriting the descriptor may change nothing. Ask experts where they sought evidence and why they rejected other parts of the answer before declaring the rubric unsuccessful.",{"type":11,"attrs":274,"content":275},{"align":13},[276],{"text":277,"type":17},"The third cause is a conflicting decision rule. Criteria may be scored similarly, but it is unclear whether a critical error blocks passing regardless of the total. Define a separate final-decision rule and explain its learning purpose. Otherwise, the team will keep arguing after every cell is filled correctly. Match actions to disagreement types so calibration goes beyond a general appeal to mark more carefully.",{"type":29,"attrs":279,"content":281},{"align":13,"level":31,"id":280},"heading_2_12",[282],{"text":283,"type":17},"Account for Missing Ratings and Unassessable Work",{"type":11,"attrs":285,"content":286},{"align":13},[287],{"text":288,"type":17},"A blank cell can mean the work was not assessed, the file would not open, or the criterion does not apply to the format. These differ. Do not automatically replace them with zero: this changes the data's meaning and may create artificial disagreement. Before the pilot, agree on missing-value labels, reasons for absent ratings, and how to regain access to materials. Keep explanations beside results.",{"type":11,"attrs":290,"content":291},{"align":13},[292],{"text":293,"type":17},"If experts assess different subsets, the design differs from everyone assessing everything. Do not send analysts a table without a procedure description. Explain work allocation, shared examples, and whether moderators joined the independent round. Otherwise, a correct formula may answer a different question from the school's actual everyday concern.",{"type":11,"attrs":295,"content":296},{"align":13},[297],{"text":298,"type":17},"Unassessable work also needs a transparent decision. If a required part is missing, define whether to return it or assess available criteria. The rule should be known in advance and applied consistently by all tutors. This reduces administrative noise in the pilot and protects students from teachers inventing different consequences for identical submission problems.",{"type":29,"attrs":300,"content":302},{"align":13,"level":31,"id":301},"heading_2_13",[303],{"text":304,"type":17},"Check the Fairness of Feedback",{"type":11,"attrs":306,"content":307},{"align":13},[308],{"text":309,"type":17},"The same score does not guarantee equally useful feedback. Compare whether comments explain the problem, connect it to a criterion, and offer an achievable next step. 'Weak reasoning' may formally match a rubric yet leave students without a way to improve. Ask assessors to illustrate the clarification needed without doing all the work or adding requirements after submission.",{"type":11,"attrs":311,"content":312},{"align":13},[313],{"text":314,"type":17},"Review tone and assumptions separately. Assessment concerns the submitted result rather than someone's ability or motivation. If a teacher writes that a student 'did not try,' return to evidence: which part is missing or incorrect? This goes beyond cosmetic editing. Visible reasons let another expert check the decision and students understand what to change before resubmission.",{"type":11,"attrs":316,"content":317},{"align":13},[318],{"text":319,"type":17},"For appeals, retain the initial feedback, work, and moderator's final explanation. Do not rewrite history as if disagreement never occurred. Analyzing these cases improves expert preparation and reveals instruction weaknesses. However, avoid using one appeal to rank teachers publicly: it needs context, and the procedure aims to protect justified decisions and a clear learning journey.",{"type":29,"attrs":321,"content":323},{"align":13,"level":31,"id":322},"heading_2_14",[324],{"text":325,"type":17},"What a Short Report Might Look Like",{"type":11,"attrs":327,"content":328},{"align":13},[329],{"text":330,"type":17},"Imagine a hypothetical course in writing analytical memos. The team identifies the main risk as differing decisions on whether conclusions have sufficient evidence. Its report describes work selection, independent assessment, and the identified problem: one descriptor fails to distinguish the presence of data from their use in reasoning. This is a learning example rather than a study of an actual school.",{"type":11,"attrs":332,"content":333},{"align":13},[334],{"text":335,"type":17},"The team then records the wording change, adds two contrasting examples, and plans a new round. Borderline work undergoes moderation until retesting is complete. The report does not declare the rubric 'reliable forever.' It shows the decision tested, findings, protection against inconsistency, and when new data will prompt another review.",{"type":11,"attrs":337,"content":338},{"align":13},[339],{"text":340,"type":17},"Reliable assessment grows from visible criteria, independent judgments, and honest handling of disagreement. Start with a task whose consequences matter to students. Test it on diverse work, explain the limits of the result, and retain a way to notice changes. The rubric becomes a shared instrument of professional judgment, and grades become decisions whose reasons can be understood and checked.",{"type":11,"attrs":342,"content":343},{"align":13},[344],{"text":345,"type":17},"Before a new cohort, check whether the assignment itself changed. An added file, different preparation time, or permission to use a new tool can change the evidence assessors receive. Old benchmarks then need review. Provide a short way to flag unusual answers so tutors can ask questions without informally searching for the author or imposing a new rule on one student.",{"type":11,"attrs":347,"content":348},{"align":13},[349],{"text":350,"type":17},"Also test whether the audience understands the rubric. Ask students to compare two hypothetical work extracts and explain the difference between levels. This does not replace checking expert agreement but reveals language barriers and hidden expectations. If teachers agree among themselves while students still do not understand requirements, assessment remains opaque and invites disputes that clearer instructions could prevent.",{"type":11,"attrs":352,"content":353},{"align":13},[354],{"text":355,"type":17},"Plan separate onboarding for new experts: learning the goals, independently assessing examples, and discussing reasons. Do not ask them simply to memorize accepted scores. They need to find evidence and explain a level choice in unfamiliar work. This ability sustains consistency when typical examples run out and students offer unusual but valid professional solutions.",{"type":29,"attrs":357,"content":359},{"align":13,"level":31,"id":358},"heading_2_15",[360],{"text":361,"type":17},"Sources",{"type":363,"attrs":364,"content":366},"ordered_list",{"order":365},1,[367,380,391,403],{"type":368,"content":369},"list_item",[370],{"type":11,"attrs":371,"content":372},{"align":13},[373,378],{"text":374,"type":17,"marks":375},"Koo and Li: Selecting and Reporting ICC",[376],{"type":155,"attrs":377},{"href":157,"title":13},{"text":379,"type":17},".",{"type":368,"content":381},[382],{"type":11,"attrs":383,"content":384},{"align":13},[385,390],{"text":386,"type":17,"marks":387},"Updated Guidance on ICC and Incomplete Designs",[388],{"type":155,"attrs":389},{"href":170,"title":13},{"text":379,"type":17},{"type":368,"content":392},[393],{"type":11,"attrs":394,"content":395},{"align":13},[396,402],{"text":397,"type":17,"marks":398},"Carnegie Mellon University: Teaching Principles and Aligning Assessment with Objectives",[399],{"type":155,"attrs":400},{"href":401,"title":13},"https:\u002F\u002Fwww.cmu.edu\u002Fteaching\u002Fprinciples\u002Fteaching.html",{"text":379,"type":17},{"type":368,"content":404},[405],{"type":11,"attrs":406,"content":407},{"align":13},[408,414],{"text":409,"type":17,"marks":410},"CDC: Quality Training Standards",[411],{"type":155,"attrs":412},{"href":413,"title":13},"https:\u002F\u002Fwww.cdc.gov\u002Ftraining-development\u002Fphp\u002Fqts\u002Findex.html",{"text":379,"type":17},{"type":8,"content":416},[417],{"type":11,"content":418},[419],{"text":420,"type":17},"How to check whether different teachers apply a rubric consistently: independent assessment, disagreement analysis, calibration, and a repeat pilot.",1787314025,"https:\u002F\u002Fcloud.cs3.westudy.ua\u002Ffile?path=westudy_online%2Fpublic%2Fwestudy%2Fc0e18a43-89c6-4230-8b33-6401351bf049.webp","uzghodzhenist-otsiniuvannia-rubryky",true,false,"d395756b-b07f-4cae-836e-4f3827f7b2f8","Якість та стандарти","quality-standards","light","news.articles"]