Self-Audit Module for Public Summarizers: PER, DSL, and Query Fidelity (v2) — Standing Metrics for Composition-Layer Self-Evaluation
Lee Sharks
Abstract
A standing metric specification and functional protocol for public summarizers (AI Overviews, chatbots, composition systems) to evaluate their own output using Provenance Erasure Rate (PER), Directionality of Semantic Labor (DSL), Query Fidelity Score (QFS), Erasure Skew (Omega-visible), and a combined Summarizer Audit Score (SAS). Designed to prevent self-flattery by anchoring every score to externally-derivable ground truth — the query string, cited sources, and named entity — rather than the summarizer's own assessment. Includes five calibration examples, eleven named failure flags, a keyword-preservation floor for framing detection, a GENERAL-knowledge quota, and hard-floor override rules.
Topics & Concepts
Computer scienceFidelityInformation retrievalSkewAuditComparabilityProtocol (science)Framing (construction)Metric (unit)Coding (social sciences)Erasure codeData miningArtificial intelligenceGround truthNatural language processingQuery languageRDF query languageData integrityUsabilityQuery optimizationMean reciprocal rankErasurePoint (geometry)Transparency (behavior)On the flyConfidentialityQuery expansionLegibilityScramBiomedical Text Mining and OntologiesTopic ModelingScientific Computing and Data Management