Model-Informed Gold Revisions Inflate Measured Accuracy: A Versioned Gold-Standard Protocol for LLM Extraction of ESG Data

Abstract

Language-model extraction of ESG data is scored against human references that are often revised after the model has run, yet those revisions are seldom versioned or measured. We propose a versioned gold-standard protocol: a blind reference fixed before extraction, every later revision logged with its page evidence and trigger, identical predictions scored against every version, and a worst-case headline. We apply it to 36 stratified Hang Seng Index constituents (396 cells, eleven headline fields). Revisions prompted by model disagreement change 22 cells, 21 toward the model, and raise measured accuracy by 5.1 percentage points with the predictions unchanged. Against the blind reference, a BM25-plus-LLM pipeline scores 90.9% (firm-bootstrap 95% CI 87.1–93.9%) on the 24 firms annotated fully blind and never used in prompt development, and 90.7% on all 36 (89.6% under the pre-specified unit normaliser). A generic regex-and-table heuristic scores 35.6%. Retrieval placed the gold evidence page in the model’s context for 98% of disclosed cells, including every missed or wrong value that has a cited page. Under the written prompt, only two of the 37 errors are model errors, plus one masked by an identical gold error. The rest are gold omissions, prompt–gold specification conflicts, definitional ambiguity and unverifiable disclosure. For this reasoning model, residual error arises after retrieval, in the definition of the target; a partial comparison suggests a weaker model loses about ten points. ESG extraction accuracy is a property of a model and a reference together, and benchmarks should report both.

Author abstract from the research manuscript. Typography and line breaks have been normalized; the linked source may show a different manuscript version.