[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog-article-lms-course-improvement-experiments":3},{"uuid":4,"title":5,"subtitle":6,"content":7,"preview":411,"date_published":416,"date_pinned":416,"img":417,"alias":418,"is_published":86,"is_pinned":419,"type":84,"category":420,"category_name":421,"category_alias":422,"background_color":423,"preview_icon":84,"category_value":424,"external_link":84},"fd9fa2ce-46ea-49de-af61-4fd2f6ead7be","How to test online course improvement: From idea to informed decision","A practical approach to testing course changes: from concrete hypothesis and student observation to careful comparison of results and implementation decisions.",{"type":8,"content":9},"doc",[10,16,17,21,22,26,27,35,39,40,44,45,49,50,56,60,61,72,73,77,78,87,93,97,98,102,103,107,108,114,118,119,123,124,128,129,134,140,144,145,155,156,160,161,167,171,172,176,177,181,182,188,192,193,197,198,202,203,209,213,214,218,219,224,230,234,235,239,240,246,250,251,255,256,260,261,265,266,272,276,277,281,282,286,287,291,292,298,302,303,307,308,312,313,317,318,324,328,329,333,334,338,339,343,344,348,349,355,359,360,366,373,377,378,384,388,389,395,399,400,406,410],{"type":11,"content":12},"paragraph",[13],{"text":14,"type":15},"The team rewrote the instructions for the task, added an example, and changed the reminder. In the next stream, more students submitted their papers on time. We would like to conclude that the new instruction worked. However, several elements changed at the same time, and the new group could have different experiences and more free time. The result should be investigated before being attributed to a single cause.","text",{"type":11},{"type":11,"content":18},[19],{"text":20,"type":15},"Checking for improvements helps the school make decisions based on the evidence and its limits. It does not always require a complex experiment. Sometimes it is enough to see how a student is misunderstanding a sentence and correct the obvious ambiguity. In other situations, you need to compare the options and assess whether the change really affects the learning outcome.",{"type":11},{"type":11,"content":23},[24],{"text":25,"type":15},"It is important to choose the method of verification according to the question, the available audience and the consequences of the decision. A small school can qualitatively explore the difficulties of participants without pretending that every difference between two small groups proves a universal advantage. An honest, limited conclusion is more useful than a confident statement that is not supported by the data.",{"type":11},{"type":28,"attrs":29,"content":32},"heading",{"level":30,"id":31},2,"heading_2_1",[33],{"text":34,"type":15},"Formulate the problem before choosing a solution",{"type":11,"content":36},[37],{"text":38,"type":15},"\"Add video\" is a decision that still needs justification. The problem may be that students confuse the two stages of the calculation. Then possible responses would be a brief explanation, a change to the example, additional practice, or clarification of the terms. First, find out exactly what happens in jobs and how people explain their actions.",{"type":11},{"type":11,"content":41},[42],{"text":43,"type":15},"Describe the observation specifically: participants choose the data correctly, but use the indicator from the previous period in the final conclusion. This setup allows you to check the target error. The general \"students are not engaged\" combines various reasons and easily leads to random design changes.",{"type":11},{"type":11,"content":46},[47],{"text":48,"type":15},"After that, formulate a hypothesis about the mechanism. For example, the period marking in the example is not clear enough, so people transfer the wrong number. This is still a hypothesis. You can first check it by observing the performance, and only then create a new version of the material.",{"type":11},{"type":28,"attrs":51,"content":53},{"level":30,"id":52},"heading_2_2",[54],{"text":55,"type":15},"Turn an idea into a testable hypothesis",{"type":11,"content":57},[58],{"text":59,"type":15},"A hypothesis links a change, an expected action, and an outcome. \"If you add a short period check before the calculation, students are less likely to use the wrong exponent in a new problem.\" Here it is clear what is changing and what sign to look for. There is no need to immediately promise a certain percentage of improvement without reason.",{"type":11},{"type":11,"content":62},[63,65],{"text":64,"type":15},"In the practice of digital products, experiments are used to test hypotheses and measure the actual impact of changes. This is how Microsoft Research describes the purpose of its experiment platform. For an online school, this is a useful general principle, but the scale of a product service cannot automatically be transferred to a small study group. ",{"text":66,"type":15,"marks":67},"Source: Microsoft Research - Experimentation Platform",[68],{"type":69,"attrs":70},"link",{"href":71},"https:\u002F\u002Fwww.microsoft.com\u002Fen-us\u002Fresearch\u002Fgroup\u002Fexperimentation-platform-exp\u002F",{"type":11},{"type":11,"content":74},[75],{"text":76,"type":15},"Also write down alternative explanations. If the new material is longer, the result may depend on the additional time. If the teacher separately draws attention to an error in only one group, not only the page is different. Such details help to determine honestly which set of changes is being tested.",{"type":11},{"type":79,"attrs":80},"image_block",{"id":81,"alt":82,"url":83,"type":79,"float":84,"width":85,"preview":86},"614c6f23-3429-402d-b2d5-b8d4b915cefb","Table: Turn an idea into a testable hypothesis","https:\u002F\u002Fcloud.cs1.westudy.ua\u002Ffile?path=westudy_online%2Fpublic%2Fwestudy%2F6065d74e-3360-4d3f-b917-5807faff830f.png",null,"100%",true,{"type":28,"attrs":88,"content":90},{"level":30,"id":89},"heading_2_3",[91],{"text":92,"type":15},"Choose an indicator that is close to the goal",{"type":11,"content":94},[95],{"text":96,"type":15},"Opening a page or pressing a button helps to understand the use of the material. However, if the goal is to properly justify the decision, an analysis of the answer is required. A person can open an example and not transfer anything from it. Can also read a short explanation without many transitions and successfully complete the task.",{"type":11},{"type":11,"content":99},[100],{"text":101,"type":15},"Before launching, define the main result and how it will be measured. For a submission instruction, this could be the proportion of correctly completed submissions among those to whom the task was available. For an educational explanation — the performance of a new task according to an agreed criterion. The denominator and the time period must be described, otherwise the numbers easily lose comparability.",{"type":11},{"type":11,"content":104},[105],{"text":106,"type":15},"Next to the main indicator, predict signs of undesirable consequences. The new format may improve accuracy, but make the work disproportionately long or inconvenient from the phone. This does not mean automatic rejection of the change. The team gets a reason to finalize it and weigh a real compromise.",{"type":11},{"type":28,"attrs":109,"content":111},{"level":30,"id":110},"heading_2_4",[112],{"text":113,"type":15},"Select the verification method according to the question",{"type":11,"content":115},[116],{"text":117,"type":15},"When you need to figure out why someone is getting lost on a page, tracking performance is often more useful than simply comparing percentages. Ask the student to perform a specific action and explain what he is looking for. Don't suggest a route before you see an obstacle. Then clarify the meaning of unclear words or button expectations.",{"type":11},{"type":11,"content":120},[121],{"text":122,"type":15},"Comparing successive cohort can show a change in results, but does not isolate the impact of an update. It can be influenced by the composition of the group, the teacher, the season, the method of recruitment and other differences. Such data are suitable for operational monitoring and formation of subsequent questions, if the limits of the conclusion are clearly described.",{"type":11},{"type":11,"content":125},[126],{"text":127,"type":15},"A side-by-side comparison with a random allocation can provide a stronger basis for causal inference when the design and analysis are done properly. However, it requires sufficient data, stable conditions and a well-thought-out method of distribution. The mere fact of having two versions of the page does not make the test a reliable experiment.",{"type":11},{"type":79,"attrs":130},{"id":131,"alt":132,"url":133,"type":79,"float":84,"width":85,"preview":86},"05e4f26e-64c8-4f12-9c02-c43cbaf36828","Table: Select the verification method according to the question","https:\u002F\u002Fcloud.cs1.westudy.ua\u002Ffile?path=westudy_online%2Fpublic%2Fwestudy%2F424fd8e7-3de1-4cca-aa01-a61960abd6c6.png",{"type":28,"attrs":135,"content":137},{"level":30,"id":136},"heading_2_5",[138],{"text":139,"type":15},"Commit the plan before viewing the results",{"type":11,"content":141},[142],{"text":143,"type":15},"A short protocol protects against changing the rules to a desired conclusion. It records the hypothesis, material versions, audience, observation period, main indicator and decision criteria. If the plan involves statistical analysis, the method of evaluation and the amount of data should be agreed with the specialist before the launch.",{"type":11},{"type":11,"content":146},[147,149],{"text":148,"type":15},"It is not necessary to review the difference every day and complete the check at the moment when it looks attractive. This approach can make the conclusion unreliable. If intermediate solutions are necessary, they are included in the corresponding design in advance. For a regular editorial check, you can choose a certain period and not announce the victory earlier. ",{"text":150,"type":15,"marks":151},"Source: Ham et al.—Design-Based Confidence Sequences",[152],{"type":69,"attrs":153},{"href":154},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2210.08639",{"type":11},{"type":11,"content":157},[158],{"text":159,"type":15},"The plan does not prohibit the failure scenario to be stopped. If the material has become unavailable or the instruction blocks submission, the problem is corrected. After that, they honestly point out that the initial comparison is broken. A learning process should not support an obvious malfunction for the sake of a beautiful report.",{"type":11},{"type":28,"attrs":162,"content":164},{"level":30,"id":163},"heading_2_6",[165],{"text":166,"type":15},"Keep conditions comparable",{"type":11,"content":168},[169],{"text":170,"type":15},"If one group receives the new instruction and additional counseling and the other only the old instruction, the entire set of differences is tested. Sometimes that's what a school needs. Then formulate a decision like this: is it worth implementing a support package. Don't attribute the result to a single sentence on the page.",{"type":11},{"type":11,"content":173},[174],{"text":175,"type":15},"In group training, participants can exchange materials. Because of this, individual distribution does not always provide segregated conditions. The distribution of entire groups also has limitations: teachers and interaction within the group differ, and analysis needs to take this design into account. This is a reason to involve methodological help, if a significant decision depends on the result.",{"type":11},{"type":11,"content":178},[179],{"text":180,"type":15},"Maintain material versions and change log during review. If the teacher corrected the example in the middle of the period, this should be reflected in the notes. Otherwise, different learning conditions will appear under the same option name, and the team will not be able to explain what exactly the participants saw.",{"type":11},{"type":28,"attrs":183,"content":185},{"level":30,"id":184},"heading_2_7",[186],{"text":187,"type":15},"Check the technical implementation in LMS",{"type":11,"content":189},[190],{"text":191,"type":15},"For the LMS platform WeStudy, formulate specific requirements: who sees each version, how access is captured, where responses are stored, and what data can be retrieved for evaluation. Ask to show it to test participants. Don't assume there's built-in A\u002FB testing, random distribution, or the desired export without verification.",{"type":11},{"type":11,"content":194},[195],{"text":196,"type":15},"If variants are assigned manually, a simple registry and error control is required. A student should not accidentally lose required material or receive conflicting terms. Also, check that the evaluator sees exactly the work and criteria that are specified in the plan. Technical confusion can render the comparison unusable regardless of the quality of the hypothesis.",{"type":11},{"type":11,"content":199},[200],{"text":201,"type":15},"Collect only the data necessary for the question. For the analysis of typical errors, impersonal answers and designation of the version of the material are often enough. Organizational and ethical requirements depend on the context of the program; they should be determined before the work begins, not after the results are obtained.",{"type":11},{"type":28,"attrs":204,"content":206},{"level":30,"id":205},"heading_2_8",[207],{"text":208,"type":15},"Read the result along with the uncertainty",{"type":11,"content":210},[211],{"text":212,"type":15},"In a small group, a few answers can change the percentage significantly. Therefore, show the actual number of observations next to the proportion. Don't declare two options the same just because no compelling difference is found. There may not be enough data to distinguish effects of interest to the school.",{"type":11},{"type":11,"content":215},[216],{"text":217,"type":15},"Statistical persuasiveness and practical utility are different issues. A small change may not justify complex support for a new format. At the same time, the qualitative elimination of a specific navigation error may be appropriate without a statement about a proven increase in the effectiveness of the entire course. The decision must correspond to the strength of the available evidence.",{"type":11},{"type":79,"attrs":220},{"id":221,"alt":222,"url":223,"type":79,"float":84,"width":85,"preview":86},"6925c500-0051-48b4-bb9d-a1d2672ccac3","Table: Read the result along with the uncertainty","https:\u002F\u002Fcloud.cs1.westudy.ua\u002Ffile?path=westudy_online%2Fpublic%2Fwestudy%2F0502c12e-b938-4f38-b3d9-09b5ee6833a3.png",{"type":28,"attrs":225,"content":227},{"level":30,"id":226},"heading_2_9",[228],{"text":229,"type":15},"Turn the conclusion into the next action",{"type":11,"content":231},[232],{"text":233,"type":15},"After checking, several decisions are possible: leave the change, refine it, return the previous version or collect additional data. Write down the reason and limit of use. For example, a new instruction helped participants of a specific program find the submission form; this is not proof of its suitability for all types of tasks.",{"type":11},{"type":11,"content":236},[237],{"text":238,"type":15},"Also keep the results that did not support the original hypothesis. They will help the next editor not to repeat the same attempt without new reasons. A short card with the hypothesis, conditions, observations, and decision gradually builds the team's memory of the course development.",{"type":11},{"type":28,"attrs":241,"content":243},{"level":30,"id":242},"heading_2_10",[244],{"text":245,"type":15},"Separate implementation verification from impact evaluation",{"type":11,"content":247},[248],{"text":249,"type":15},"Before analyzing the difference in results, find out whether the students actually received the intended conditions. The new block could be hidden, the link unavailable, and part of the group could see the previous version. In this case, the absence of a noticeable change does not give a clear answer to the initial hypothesis. First you need to analyze the actual implementation of the plan.",{"type":11},{"type":11,"content":252},[253],{"text":254,"type":15},"For this, it is useful to conduct a short technical rehearsal before the main period. The test participant opens the desired version, completes the task, and the team checks whether the result is stored in a form suitable for analysis. Rehearsal does not assess learning advantage. It confirms that the planned verification method can be implemented without obvious failures.",{"type":11},{"type":11,"content":257},[258],{"text":259,"type":15},"Do not inconspicuously remove from the analysis people who did not use the new material, if the original plan provided for a different approach. Their behavior can be part of the real impact of change. How to deal with such cases depends on the research question and design, so it is determined in advance or openly marked as an additional analysis.",{"type":11},{"type":11,"content":262},[263],{"text":264,"type":15},"In the report, separate two conclusions: whether the decision was implemented as intended and what is known about its outcome. This helps to avoid mistakenly rejecting a good idea due to a technical error. At the same time, the team should not endlessly attribute any unsuccessful result to the implementation without checking whether the assumption itself was sufficiently justified.",{"type":11},{"type":28,"attrs":267,"content":269},{"level":30,"id":268},"heading_2_11",[270],{"text":271,"type":15},"Make grading work consistent",{"type":11,"content":273},[274],{"text":275,"type":15},"If the result is determined by the teacher, the criterion must be agreed upon before the comparison begins. The general impression of \"this answer is better\" is difficult to compare between different reviewers. Describe a specific feature: the conclusion is based on the relevant indicator, the assumption is marked, the limitation is taken into account. If necessary, prepare examples of the application of the criterion to answers of different quality.",{"type":11},{"type":11,"content":278},[279],{"text":280,"type":15},"When possible and appropriate, the evaluator is not shown which version of the material the author received. This may reduce the impact of expectations about the new format. However, hiding an option is not always technically or meaningfully possible. Such a limitation should be noted, rather than pretending that evaluation is completely context-independent.",{"type":11},{"type":11,"content":283},[284],{"text":285,"type":15},"Before the main review, several raters can review the same examples and discuss differences. The goal is to clarify the understanding of the criterion, not to achieve the desired result for one option. If a rule changes after reviewing part of the work, the team must determine how to consistently apply it to all responses within the selected analysis.",{"type":11},{"type":11,"content":288},[289],{"text":290,"type":15},"The quality of the measurement is important even for a small local study. If the criterion is unclear, additional data can only increase the volume of non-comparable estimates. It is best to first check that the team is equally aware of the desired error, and then decide what scale of observation makes sense for the issue at hand and the school's available resources.",{"type":11},{"type":28,"attrs":293,"content":295},{"level":30,"id":294},"heading_2_12",[296],{"text":297,"type":15},"Do not convert additional findings into the original hypothesis",{"type":11,"content":299},[300],{"text":301,"type":15},"After reviewing the data, interesting differences between subgroups may emerge. For example, the new manual seems more useful to people with some experience. This may be a reason for further inspection, but such a finding should not be described as the result of a question that the team asked in advance, if it was not in the plan.",{"type":11},{"type":11,"content":304},[305],{"text":306,"type":15},"The more indicators and subgroups are viewed without a defined logic, the easier it is to find a randomly attractive difference. For a practical report, it is enough to clearly separate the planned analysis and exploratory observations. Others can be valuable if their uncertainty is preserved and the subsequent decision is not based on exaggerated confidence.",{"type":11},{"type":11,"content":309},[310],{"text":311,"type":15},"Similarly, you should not change the main indicator after the initial one did not show the desired picture. If the new format increased views, but did not give a clear answer about the quality of works, both facts can be reported. However, the increase in views does not replace the original educational goal without a separate justification for its change.",{"type":11},{"type":11,"content":314},[315],{"text":316,"type":15},"For the editorial team, a simple question is useful: Would we write the same conclusion if the option names were reversed? It does not replace the methodology, but it helps to notice commitment to one's own new idea. The purpose of the review is to support the next decision, not to confirm the correctness of the author of the update for any result.",{"type":11},{"type":28,"attrs":319,"content":321},{"level":30,"id":320},"heading_2_13",[322],{"text":323,"type":15},"Make a short decision card",{"type":11,"content":325},[326],{"text":327,"type":15},"A summary card can include the problem, the change tested, the audience, the period, the main observation, the constraints, and the next action. Save links to material versions and grading method. So another employee will be able to understand the basis of the decision without having to tell the whole story in a separate meeting.",{"type":11},{"type":11,"content":330},[331],{"text":332,"type":15},"The wording should correspond to the data: \"removed an ambiguous name after monitoring the search\" or \"the comparison result does not allow you to choose an option yet.\" Not every card has to contain a winner. Sometimes the best result of the work is the rejection of unnecessary complexity or, more precisely, the question for the next cycle of course development.",{"type":11},{"type":11,"content":335},[336],{"text":337,"type":15},"Appoint a time to return to the solution, if it is temporary. For example, after the next launch, the team will check whether new difficulties have appeared with the updated manual. This does not mean repeating the same test over and over again. This is a specific uncertainty that remains after the current analysis and is really important for further work.",{"type":11},{"type":11,"content":340},[341],{"text":342,"type":15},"Compare the importance of the problem, the uncertainty of the solution, and the cost of the error to prioritize subsequent checks. An oft-repeated obstacle in a mandatory task deserves attention sooner than a cosmetic change to an extra page. However, a complex experiment is not always the first step: first you can check whether the team has correctly understood the problem itself.",{"type":11},{"type":11,"content":345},[346],{"text":347,"type":15},"Such an order helps to direct the limited resource to the issues on which the real learning experience depends. Each subsequent check should have a solution that it can change. If neither possible outcome affects the team's actions, it is worth reconsidering the feasibility of data collection.",{"type":11},{"type":28,"attrs":350,"content":352},{"level":30,"id":351},"heading_2_14",[353],{"text":354,"type":15},"Conclusion",{"type":11,"content":356},[357],{"text":358,"type":15},"Verifying improvement means linking a specific problem to a change, an observed outcome, and an honest conclusion. Start with a question, choose an appropriate method, fix the conditions and allow for uncertainty. LMS helps organize materials and data, but is not a substitute for thoughtful design. For a small school, consistent local inspections can become a reliable basis for development without exaggerated claims of effectiveness.",{"type":11},{"type":28,"attrs":361,"content":363},{"level":30,"id":362},"heading_2_15",[364],{"text":365,"type":15},"FAQ",{"type":28,"attrs":367,"content":370},{"level":368,"id":369},3,"heading_3_16",[371],{"text":372,"type":15},"Do you need an A\u002FB test for every course change?",{"type":11,"content":374},[375],{"text":376,"type":15},"No. An obvious error or unclear name can be corrected after checking the specific situation. A controlled comparison is appropriate when the impact of alternatives needs to be assessed and there are conditions for a robust design. The method is chosen for the decision that the team really has to make.",{"type":11},{"type":28,"attrs":379,"content":381},{"level":368,"id":380},"heading_3_17",[382],{"text":383,"type":15},"How many students are needed for the experiment?",{"type":11,"content":385},[386],{"text":387,"type":15},"There is no universal number. It depends on the indicator, the expected size of the change, the variability and the distribution method. Quantitative inference requires analysis planning. If the audience is small, it is useful to start with observations and not to turn unstable percentages into categorical statements.",{"type":11},{"type":28,"attrs":390,"content":392},{"level":368,"id":391},"heading_3_18",[393],{"text":394,"type":15},"Is it possible to compare the current cohort with the previous one?",{"type":11,"content":396},[397],{"text":398,"type":15},"Such a comparison is useful for monitoring changes if differences between cohort are described. It does not in itself prove that the result was caused by the rate update. Take into account the composition of groups, teachers, deadlines and other changes in the educational process.",{"type":11},{"type":28,"attrs":401,"content":403},{"level":368,"id":402},"heading_3_19",[404],{"text":405,"type":15},"What to do if the new option did not show advantages?",{"type":11,"content":407},[408],{"text":409,"type":15},"Check that it worked as intended and that there was enough data to draw the desired conclusion. Record the result without trying to hide the uncertainty. The team can return a previous version, refine the hypothesis, or focus on another problem more important to the students.",{"type":11},{"type":8,"content":412},[413],{"type":11,"content":414},[415],{"text":6,"type":15},1791186240,"https:\u002F\u002Fcloud.cs1.westudy.ua\u002Ffile?path=westudy_online%2Fpublic%2Fwestudy%2Fe2ffc3f8-6c6d-4331-ad37-1c9a80609caf.webp","lms-course-improvement-experiments",false,"8fb18e59-6f63-41ba-8ac3-1ca02b0fc763","Гід для онлайн-школи","online-school-guide","light","news.articles"]