The essential problem in data assessment is called overfitting, i.e. using a small dataset to predict something. The grading software must compare essays, understand what parts are great and not so great and then condense this down to a number which constitutes the grade, which in its turn must be comparable with a different essay on a totally different topic. Sounds hard, doesn’t it? That’s because it is. Very hard. But still, not impossible. Google uses similar tactics when comparing what resulting texts and images are more preferable to different search terms. The issue is just that Google uses millions of data samples for their approximations. A single school could, at best, input a few thousand essays. This is like trying to solve a 1000-piece puzzle with just 50 pieces. Sure, some pieces can end up in the right place but it’s mostly guess work. Until there is a humongous database of millions and millions of essays, this problem will most likely be hard to work around.
The only plausible solution to overfitting is specifying a specific set of rules for the computer to act upon to determine if a text makes sense or not, since computers can’t read. This solution has worked in many other applications. Right now, auto-grading vendors are throwing everything they got at coming up with these rules, it’s just that it is so hard coming up with a rule to decide the quality of creative work such as essays. Computers have a tendency of solving problems in the way they usually do: by counting.
In auto-grading, the grade predictors could, for example, be; sentence length, the number of words, number of verbs, number of complex words and so on. Do these rules make for a sensible assessment? Not according to Perelman at least. He says that the prediction rules are often set in a very rigid and limited way which restrains the quality of these assessments. For example, he has found out that:
- A longer essay is considered better than short one (a coincidence according to auto grading advocate and professor Mark D. Shermis)
- Specific word associated with complex thinking such as ’moreover’ and ’however’ leads to better grades
- Towering words such as ’avarice’ gives more points than using simple ones such as ’greed’
On other instances he found examples of rules poorly applied or just not applied at all, the software could for example not determine whether facts were true or false. In a published and automatically graded essay, the task was to discuss the main reasons why a college education is so expensive. Perelman argued that the explanation lies within the greedy teacher’s assistants who has a salary of six times that of a college president and regularly uses their complementary private jets for a south sea vacation.
The essay was awarded the highest grade possible: 6/6.
To avoid the examining eye of Perelman and his peers most vendors have restricted use of their software while development is still ongoing. So far, Perelman hasn’t gotten his hand on the most prominent systems and admits that so far he has only been able to fool a couple of systems.
If we are to believe Perelman’s claims, automatic grading of college level essays still has a long way to go. But remember that already today, lower grade essays is actually being graded by computers already. Granted, under meticulous supervision by humans but still, technological progress can move fast. Considering how much effort being asserted towards perfecting automatic grading scoring it is likely we will see a fast expansion in a not too distant future.
About the author: Hubert.ai is a young edtech company based in Stockholm, Sweden. We are working to disrupt teacher feedback by using AI conversational dialog with every student separately. Feedback is then analyzed and compiled down to a few recommendations on how you as a teacher can improve your skills and methods. Are you a teacher and would like to help us in development? Please sign up as a beta tester at our website :]
You want your students to receive meaningful feedback on their papers and grow as writers. You also, though, need to have time to recharge on weekends and holidays without the burden of endless essay stacks hanging over your head.
With my five-minute-per-essay grading system, you’ll zip through your stacks of papers, saving HOURS of time. Before I created this system, it took me 15-to-20 minutes to grade an essay. Now, with the use of a detailed coding system and a content rubric that’s laser-focused on the writing issues that matter most, I’m able to mark an essay in just under five minutes. Really.
Look, grading papers is part of the gig. We signed up to be English teachers; there’s no getting around the fact that we’re going to have to spend some time with the grading pen. The amount of time we spend on those papers, though, needs to provide a fruitful yield of learning for our students without breaking our spines.
This product includes:
25-page “5-minute Essay Grading System” e-book (PDF) with detailed directions, visual examples, and TONS of tips to help you slay your paper-load dragon
4 Essay Corrections student handout sheets (in both print-and-use PDF and editable Word format, so 8 total pages) that include the ten most common grammar/mechanics errors in teen writing and the most common content issues on four different modes of writing:
4 Essay Content Rubrics (in both print-and-use PDF and editable Word format, so 8 total pages) that include the top areas of focus on each of the following writing modes:
1-page Grammar/Mechanics Correction Tally Sheet (PDF) to help students chart their improvement over the school year
1-page Teacher Coding Cheat Sheet (in print-and-use PDF or editable Word format, so 2 pages total) to help speed along your use of the codes as you grade papers
1-page S.A.T.-style grading rubric (PDF) featuring the 4-3-2-1 system used by the College Board as of March 2016 (an alternative rubric for Expository/Argumentative writing assignments) and a 1-page S.A.T.-style grading rubric (PDF) featuring the 6-5-4-3-2-1 scale used prior to March 2016
6-page S.S.R. (Sustained Silent Reading) Program notes and student handouts (PDF) to help you set up an in-class reading program that won’t require extra paper grading by you (Note: These S.S.R. Program materials are available free in my shop. Just click HERE.)
Click on the “preview” button above to read the first few pages of the ebook and view sample materials. These 5-min. essay grading materials will work in upper-middle school and high school classrooms.
Want more ways to help teen writers? Click HERE to view my full catalog of writing lesson materials.
Thanks for stopping by!
25-page ebook (PDF); 26 more pages of handouts & support material (PDF & Word)