Classify data quality defects in the target system, propose schema-aware fixes, auto-apply above a confidence threshold or route to review below it — every patch with an audit log entry.
In short: After a data migration, defects surface in the target system that classical ETL validation never catches — format drift, missing required fields, referential inconsistency, duplicates, encoding mismatch. The Data Quality Remediation Agent scans extracts, classifies each defect, proposes a schema-aware fix, and computes a confidence per proposal. Above a configured threshold the fix is applied automatically; below it, the proposal lands in a review queue for a human reviewer. Stack: Claude (Anthropic direct or Azure OpenAI Switzerland), Azure Document Intelligence for unstructured sources, schema-aware diffing against the target model. Cross-industry, particularly effective after SAP, CRM or ERP cut-overs with thousands of records and a mixed defect class profile. Every fix produces a structured audit entry — compatible with GoBD documentation requirements and Swiss FADP traceability.
A data migration isn't finished when the records arrive in the target system — it's finished when they're usable there. Between those two states sits the invisible part of the work: data quality defects that slip through ETL validation because they don't violate a strict schema rule. Date fields that arrive sometimes as YYYY-MM-DD and sometimes as DD.MM.YYYY. Phone numbers with and without country prefix. Required fields filled with placeholders instead of real values. Foreign keys pointing at records deleted in the source but still referenced in the target. A mixed UTF-8 / Latin-1 inventory that only surfaces on the first report export.
Classical ETL validation only kicks in where a hard schema rule is broken — everything that is structurally conformant but semantically or contextually wrong falls through. In practice that means: after every larger cut-over comes a phase of manual data cleanup, where domain experts open records one by one, check them, and correct them. On hundreds of records that is still defensible; on tens of thousands it is a quarterly project. On top of that, manual cleanup rarely produces a usable audit trail — who changed what, when, and why is hard to reconstruct after the fact. That bottleneck is exactly what a remediation agent addresses: not the data transfer, but the post-migration cleanup.
The agent works on extracts from the target system — structured (tables, JSON) or unstructured (PDF, scans, free-text fields). For unstructured sources, Azure Document Intelligence is layered in first so the agent can reason over structured representations. Per record and per field it classifies a defect into a class (format drift, missing required field, referential inconsistency, encoding mismatch, duplicate, logic violation), proposes a schema-aware fix, and computes a confidence. The schema-aware diff shows explicitly which field is being set to which value and why — not a black box, but a structured patch.
The confidence is checked against a project-wide configurable threshold. Above the threshold (e.g. > 0.95) the fix is applied automatically; below it, the proposal lands in a review queue. Reviewers see the original value, the proposal, the classification, and the reasoning — and they sign off in batches. Model-side, Claude is used for longer reasoning chains; Azure OpenAI Switzerland is the option when Swiss data residency is a contractual condition. Guardrails: a curated eval set per defect class, cost caps per run, hard escalation on anything that changes required fields or re-points foreign keys, and a complete rollback set for every patch batch.
Important: the agent does not replace semantic data sign-off. It catches structural and consistency defects — it does not catch whether a value is correct against the real world. A postcode that doesn't match the city can be flagged when a reference dataset is present; a schema-conformant number in an amount field that is simply wrong falls through. This boundary is communicated explicitly, so the compliance officer and the domain side can set the threshold realistically.
A Swiss manufacturer — composite, not a real client — has finished a SAP migration. In the target system sit roughly 15,000 customer records; an estimated 3% carry data quality defects. The 3% is a scenario assumption, not a measured outcome. The mix is typical: roughly half are clearly fixable format drifts (phone number sometimes with, sometimes without +41; postcode appearing as four or five digits after accidental leading-zero erosion; dates in two conventions). The other half is ambiguous: address fields with multiple plausible spellings (St. Gallen vs. Sankt Gallen vs. St-Gallen), company names with unclear legal-form conventions, and duplicates that differ minimally in one field.
The agent (Claude, accessed via Azure OpenAI Switzerland for Swiss FADP data residency, with Azure Document Intelligence for scanned delivery addresses) classifies each record. For roughly 60% of the defects, confidence sits above 0.95 — format drifts with an unambiguous source pattern are auto-patched. The remaining 40% land in the review queue with full reasoning exposed. A two-person data-owner team works the queue in a week instead of a quarter. Every applied patch — auto or manual — lands in the audit log with input snapshot, classification, proposal, confidence, and reviewer signature (where applicable).
Typical effect: the cleanup phase after a larger cut-over moves from weeks toward days — directional, not a published benchmark, and strongly dependent on defect mix and threshold. The leverage doesn't sit in any individual fix but in the separation between clearly fixable and ambiguous: the agent absorbs the long tail of structurally unambiguous fixes, while the domain side concentrates on the borderline cases that need experience. As a by-product a complete audit trail emerges per patch: what was the original value, which classification, which proposal, which confidence, who signed off (if manual). This trail is compatible with GoBD documentation requirements and Swiss FADP traceability — it isn't reconstructed after the fact, it falls out of the cleanup itself. The precondition for the effect is realistic threshold calibration; a threshold set too high reduces the agent to useless automation, while one set too low only shifts the problem into the review loop.
Structurally and consistency-wise, broad coverage: format drift (date, phone number, postcode spellings), missing required fields with a derivable default, referential inconsistency (foreign keys pointing to non-existent records), encoding issues (UTF-8 / Latin-1 mix), duplicate records with minor spelling differences, logic violations (negative quantities, future dates as historical values). What it does not detect: semantically wrong values that are schema-conformant but factually wrong — a real address with a wrong postcode is only flagged when a reference dataset is present; a simply incorrect amount in an amount field falls through. This boundary is communicated explicitly so the threshold can be set realistically.
Confidence is computed per defect class and per record, based on structural signals: a clear format violation with an unambiguous source pattern → high; a proposal with multiple plausible readings → low. The threshold is configurable per project — we recommend starting with a high threshold (e.g. > 0.95) and adjusting based on the first review round. Initial calibration runs through a sample set: you manually review the first ~200 proposals, and the agent learns the acceptance boundaries of your data context. Calibration is part of setup, not a black box — you see per defect class how the threshold affects the auto-apply rate.
Yes, with an approval gate. Standard workflow: the agent generates a structured fix patch (a diff per record and field), a reviewer signs off in batches, the patch is applied, and an audit entry is created per affected record. With very high confidence, auto-write can be configured — but we only recommend it for very narrow defect classes such as pure date-format drifts with an unambiguous source pattern. For all semantic or ambiguous fixes, the human stays in the loop. Every patch batch is rollback-capable, so an accidentally applied auto-write can be reverted within the audit trail.
This use case sits in the Agent Systems & Workflows practice and connects to migration and audit use cases.
Practice: how agents are built, operated, and made audit-ready in production environments — patterns, guardrails, platform choice.
Prepare migration objects per backlog entry — parse schema, detect Z-tables, propose mappings, escalate cleanly.
Audit-ready diff reports with discrepancy classification and sign-off packages for cut-over.
Data quality remediation pays off when defect volume exceeds human review capacity. For small datasets, classical manual cleanup is often cheaper. In a non-binding 30-minute discovery call we'll talk honestly about whether your case is a candidate.
Book a discovery call