HomeLatest ThreadsGreatest ThreadsForums & GroupsMy SubscriptionsMy Posts
DU Home » Latest Threads » Forums & Groups » Topics » Economy & Education » Education (Group) » now pearson wants to mach...

Tue Jun 12, 2012, 02:25 AM

 

now pearson wants to machine-score essay tests.

http://www.huffingtonpost.com/todd-farley/lies-damn-lies-and-statis_b_1574711.html?ref=standardized-testing

As an anti-testing guy, and a writer, I never thought I'd be wowed by a technology that claims to be able to assess student writing without even being able to, you know, read. In the end, however, I couldn't help but be impressed by the shiny statistics provided in support of the study's own findings. There was page after page of tables filled with complicated numbers, graph after graph filled with colorful lines, plus the Pearson r, all purporting to show the viability (if not superiority) of automated essay scoring engines when compared to human readers.

Even if all that is true, what is it about the wonders of automated essay scoring that this study really is claiming? Come to find out, the study asserts very little. Perhaps most importantly, the study makes no claim about those automated scoring engines being able to read, which they emphatically cannot do. That means that even though every single one of those automated scoring engines was able to pass judgment on every single one of the essays painstakingly composed by American students, not even one of those scoring engines understood even one word from all those kids.

Provocative thoughts in those essays? The automated scoring programs failed to recognize them. Factual inaccuracies? The scoring engines didn't realize they were there. Witty asides? Over the scoring engines' heads they flew. Clichés on top of clichés? Unbothered by them the scoring systems were. A catchy turn-of-phrase? Not caught. A joke about a nitwit? Not laughed at. Irony or subtlety? Not seen. Emotion or repetition, depth or simplicity, sentiment or stupidity? Nope, the automated essay scoring engines missed 'em all. Humanity? Please.

11 replies, 3271 views

Reply to this thread

Back to top Alert abuse

Always highlight: 10 newest replies | Replies posted after I mark a forum
Replies to this discussion thread

Response to HiPointDem (Original post)

Tue Jun 12, 2012, 02:42 AM

1. I hate this so much. n/t

Reply to this post

Back to top Alert abuse Link here Permalink


Response to HiPointDem (Original post)

Tue Jun 12, 2012, 03:45 AM

2. If the machines can't actually read, I have to wonder what they base their scores on.

My guess is that they look for some sort of formulaic writing in the essays. If so, of course, we can then expect "good" teachers will teach their students to write following some formulaic approach; and eventually, formulaic writing will come to be the definition of "good writing". All to increase corporate profits.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Jim__ (Reply #2)

Tue Jun 12, 2012, 06:40 AM

4. For those of us who know nothing about these things, indeed, that would be helpful.

I can imagine that it has a spelling and grammar checker. I can imagine that it uses something like a Fleischman formula for calculating reading level. Those functions have been turned into computer code for decades. But what automated essay graders actually do remains a mystery to me.

It's strange to me that the US populace gets tied up in knots over the accuracy of educational testing, but simply accepts the outcomes the US court system, which sometimes seems to have 'similar' between evaluator correlations to essay grading but rather more serious consequences.






Reply to this post

Back to top Alert abuse Link here Permalink


Response to Jim__ (Reply #2)

Tue Jun 12, 2012, 10:38 AM

6. Here's what good writing is for the Texas TAKS test (just superseded by STAAR)

Essay must be 52 lines long.

Intro paragraph must be five lines long and contain three claims or warrants.

The body paragraphs must each be one of the claims or warrants in the same order as the intro and 14 lines long. Each claim or warrant must have three different supports: assertion, evidence, general knowledge, and so on.

The conclusion must be five lines long and repeat the three claims or warrants in the same order as the intro and the body.


Voila - you have a perfect essay - spelling is not checked nor grammar nor even complete sentences - I've had students deliberately leave those elements out and still score a commended or exemplary essay.

This was with human graders - I imagine machines will be less particular.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to mbperrin (Reply #6)

Tue Jun 12, 2012, 06:13 PM

7. Thanks for the information. I had no idea essays were so formulated.

It's surprising to me that they specify length in lines rather than sentences. The structure that you describe sounds fine - i.e. an essay that is structured that way is well-structured. But, I think that type of constraint on an essay is far too restrictive and doesn't encourage students to actually learn to write.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to Jim__ (Reply #7)

Tue Jun 12, 2012, 09:00 PM

9. I agree completely. But students must pass the test to graduate.

So we must teach this format. Thank goodness it doesn't take too long, so more can be added as well.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to HiPointDem (Original post)

Tue Jun 12, 2012, 04:59 AM

3. I'd like to see how they'd score Mark Twain

and other great writers.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to PADemD (Reply #3)

Tue Jun 12, 2012, 07:19 PM

8. Or Hemingway

 

This is a really bad idea. Really good writers know how to screw with language to make it more powerful - machines probably won't like that.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to PADemD (Reply #3)

Wed Jun 13, 2012, 08:54 AM

11. Or ee cummings

Reply to this post

Back to top Alert abuse Link here Permalink


Response to HiPointDem (Original post)

Tue Jun 12, 2012, 07:04 AM

5. The idea of using a computer to grade essays, shows a fundamental

lack of understanding the basic purpose of essays. They are about
ideas. Yes, incorrect spelling, grammar and other technical errors could be found. But, computers would be incapable of determining the
essence of the ideas presented by writers. In fact, human graders
have much of the same problem which is why essay tests are considered to be a poor form of evaluation due to the subjective responses of the graders.

Reply to this post

Back to top Alert abuse Link here Permalink


Response to HiPointDem (Original post)

Wed Jun 13, 2012, 08:31 AM

10. I had to score something like that once.

 

All the teachers at our school had to help grade essays one year. The scoring was based solely on number of syllables and number of words per sentence.

The quality of the writing was not considered relevant at all and had no influence on the grade - just the quantity of speech parts counted.

Reply to this post

Back to top Alert abuse Link here Permalink

Reply to this thread