AI in Education · For teachers

What should a school measure in an AI feedback pilot?

The short answer

Measure three things together: the corrections teachers make before releasing feedback, what students do with the feedback, and the teacher time required for the complete process. Use the same checks in your current workflow. Keep the number of responses beside each result, and record unresolved problems before deciding what to test next.

Four things to remember

  1. Choose one task and one learning criterion before collecting results.
  2. Keep the AI draft, teacher corrections and released feedback separate.
  3. Inspect student revisions; satisfaction alone does not show that feedback helped.
  4. Count the complete teacher workload and report small samples as small samples.
Printable pilot sheet with space to record draft corrections before release, student changes after revision, and complete teacher effort. Compare the same checks in the current workflow and keep counts with totals.
Teaching visual · Download the image · AI-generated illustration, reviewed for this guide.

What question should the pilot answer?

Start with a classroom job rather than a feature list. For example: can this process help Year 9 students explain how evidence supports a claim, while keeping the teacher’s checking and follow-up manageable? That gives the team something to inspect: the paragraph, the advice, the revision and the work needed to return it. A collection of positive comments about the interface cannot answer that question.

Write down the task, the criterion, the teachers involved and the planned revision opportunity. Choose a bounded first test and name what it will leave untested. A trial of analytical paragraphs does not establish suitability for every subject or assessment. Arrange the school’s permissions and data requirements before using student work; the measurement sheet below is not an approval process.

What should you record before students see the feedback?

Keep a record of the original draft and the teacher’s changes within the school’s approved system. For each response, note whether the teacher corrected a factual error, removed advice unsupported by the work, added a missed learning need or made the next step clearer. A response can need several changes. Count affected responses as well as describing the issues, so one heavily edited script does not look like five different students.

Separate necessary corrections from personal wording preferences. Replacing a greeting is different from removing a fabricated quotation. Record the serious example, what the teacher did and whether the problem recurred. An unchanged comment is not automatically a good comment: check it against the actual work and criterion. Use the same quality questions for existing teacher-written feedback.

Our guide to evaluating an AI marking tool’s accuracy covers the detailed benchmark checks. This pilot sheet adds the next link in the chain: after the teacher has checked the feedback, can the student use it? Do not add quality, speed and student satisfaction into a single score that can conceal an unresolved error.

How will you know whether students used the feedback?

AERO describes formative assessment as gathering and interpreting evidence of learning to guide teaching. Its feedback guidance includes giving students steps to improve and time to reflect. For this pilot, choose one visible revision check before returning the work. If the criterion is explaining evidence, look for a sentence that connects the chosen evidence to the claim, rather than counting any added sentence as an improvement.

Ask students to identify the change they made and briefly explain why. Inspect the before-and-after work against the agreed criterion. Record improved, not yet improved or not enough evidence to judge. Keep absent students and missing revisions visible, with the total number offered a revision opportunity. A student who could not explain the advice may need another explanation; record that teaching response too.

These are observations about a particular task. A stronger second draft does not, by itself, show that the student can apply the skill independently later, or that the AI process caused the improvement. A later, comparable task can check transfer. Avoid presenting a small classroom pilot as proof of a general learning gain.

How should teacher effort sit beside the learning evidence?

Use the same start and finish points for the current and trial workflows. Include preparing the task and criteria, handling submissions, reviewing and correcting feedback, releasing it and completing the planned follow-up. Keep initial training or one-off setup visible separately from recurring work. Report waiting separately from active teacher time, while noting whether it delayed feedback reaching students.

The complete-cycle marking time log gives a worked method for recording these stages without timing every keystroke. Use it alongside the evidence sheet, not as a substitute for checking quality. Keep each teacher’s result visible before combining totals: a quicker process for one familiar user may still require substantial support for another.

What would a useful set of results look like?

Consider this invented example, not a school case study or a product benchmark. A teacher reviews AI drafts for twelve analytical paragraphs. Three responses need substantive corrections before release: one comment describes evidence that is not there, and two miss the same reasoning problem. The teacher corrects those drafts, then gives all twelve students time to revise.

Ten students return revisions. Seven of those ten meet the agreed explanation check, while three need another explanation. Two have not returned a revision. The record should therefore say seven of ten returned revisions met the check, with two of twelve still missing. It should not say that all twelve improved, nor describe the seven as an improvement rate caused by AI.

The same teacher records 95 active minutes across the trial’s complete cycle. An earlier comparable task took 110 minutes, but its revisions were never inspected. The observed time difference is fifteen minutes; the learning comparison is missing. A sensible next step is to apply the same revision check to the current workflow and investigate the recurring missed reasoning problem. The smaller total alone is not a reason to expand the pilot.

How can your team use the sheet at a review meeting?

Copy the blank table below for each task. Before starting, agree which problems would require correction and retesting, who will inspect the evidence and when the team will review it. At the meeting, put unresolved issues beside the benefits. Choose a concrete next step: repeat a limited test after a change, gather missing evidence, stop the trial or consider a carefully defined expansion. Do not invent a universal pass percentage.

Use the school AI pilot checklist to keep the wider permissions, responsibilities and review actions alongside these measurements. Keep identifying student information out of a shared summary, and retain the detailed evidence only in the school’s approved location.

If you are considering JeddAI, discuss a school pilot with our team. Bring the task, criterion and evidence sheet so the conversation starts with what your school needs to establish. A demonstration can help you plan a test; the results still need to come from the process you actually trial.

Copy or print this pilot measurement sheet. Task: ______ Criterion: ______ Teacher: ______ Review date: ______. Record counts and totals; keep detailed student evidence in the approved school system.
Measure Current workflow Trial workflow Issue or next check
Responses reviewed before release ______ of ______ ______ of ______ Unreviewed or missing: ______
Responses needing substantive corrections ______ of ______ reviewed ______ of ______ reviewed Error types and examples: ______
Revisions meeting the agreed check ______ of ______ returned ______ of ______ returned Criterion applied: ______
Revisions missing ______ of ______ expected ______ of ______ expected Reason / follow-up: ______
Active teacher minutes, complete cycle ______ minutes ______ minutes Same boundaries? ______
One-off setup / training ______ minutes ______ minutes Included above or separate? ______
Waiting and return delay ______ ______ Keep separate from active time: ______
Invented example: twelve analytical paragraphs. These numbers illustrate record-keeping, not a result achieved by a school or product.
Observation What the team can say What remains unknown
3 of 12 drafts needed substantive corrections Teacher review found problems before release Whether the same problems recur after a change
7 of 10 returned revisions met the check; 2 of 12 were missing Seven revisions met this task's explanation criterion Performance of missing work and independent transfer
95 active minutes versus 110 on an earlier task Observed difference: 15 minutes Comparable revision quality; whether the difference persists

Questions teachers ask

Is student satisfaction enough to judge the pilot?

It is useful feedback about the experience, but it does not show what students learned. Ask what was clear or confusing, then inspect a revision against a defined criterion. Keep the two kinds of evidence separate.

What if teachers edit nearly every AI comment?

Find out why. Changes to factual accuracy or the chosen next step matter differently from preferred wording. Record the reasons and the time involved before deciding whether the process is useful.

How many responses make a pilot conclusive?

There is no sample size here that makes a classroom pilot conclusive. State the task, the number reviewed and the limits of the evidence. Use early results to identify problems and plan the next check, not to claim whole-school effects.

Should the school combine everything into one score?

Keep quality, student use and teacher effort visible separately. A faster total should not cancel out an unresolved feedback error, and a positive experience should not conceal missing revision evidence.

About this guide

Written by Shashank Singhal and reviewed on 5 Oct 2026. Filed under AI in Education. Looking for study material? Browse Jeddle’s Australian-English study guides.

Keep reading

All teacher guides
Shopping cart0
There are no products in the cart!