arXiv/Article/25/08/Evaluating the Evaluators: Are readability metrics good measures of readability?
Properties
- logseq-entity
- Logseq/Entity/Article
- created-by
- Person/Isabel Cachola, Person/Daniel Khashabi, Person/Mark Dredze
- date-created
- 2025-08-26 Tue
Evaluating the Evaluators: Are readability metrics good measures of readability?
- arXiv:2508.19221 · Isabel Cachola, Daniel Khashabi, Mark Dredze (Johns Hopkins)
Summary
- Surveys Plain Language Summarization (PLS) and finds the field's standard practice is to judge readability with traditional formulas like Flesch–Kincaid Grade Level (FKGL).
- Tests 8 readability metrics against human judgments. Most, including FKGL, correlate poorly with what people actually find readable.
- Language models judge readability better — the best model reaches a 0.56 Pearson correlation with human judgments — and capture deeper factors the formulas miss, such as the background knowledge a text demands.
- Closes with recommendations for evaluating plain language summaries, and releases the analysis code and survey data.
Why it matters here
- Direct evidence that readability formulas measure surface features, not comprehension — which is why My/Pref/Writing/Use Plain language treats the formula as a diagnostic, not a goal.