Work / 06
AI Data Governance Platform
When a dataset quietly breaks, how long until someone notices, and can the system say why?
Summary
A monitoring platform for analytics data: rules run against modelled datasets, failures become incidents, each dataset gets a health score, and an LLM script turns a technical failure into an explanation an analyst can read. The earlier project that the agent-governance work grew out of.
2026 · Public repo, live demo
How it flowsOne request or record, start to finish
- GitHub Archiveingested with Python
- PostgreSQLraw storage plus governance metadata tables
- dbtfact and dimension models: commits, pull requests, issues, repository, user, date
- Rule engineduplicate and null checks, foreign keys; outcomes logged
- Health score + incidents1 - failed rules / total rules
- Explanation + Power BILLM-written incident notes, monitoring dashboard
What I built
- A governance metadata layer: dataset_registry, rule_catalog, rule_runs, dataset_health_scores and incidents.
- A Python rule runner with example rules such as duplicate commit SHAs, null commits, foreign key validation and duplicate issues.
- A health score per dataset, health_score = 1 - failed_rules / total_rules, so unstable datasets are easy to find.
- Incidents generated automatically from rule failures, with severity, status and timestamps, and a Power BI dashboard over them.
DecisionsWhat I chose, what I passed on, why
Treating datasets like production services
over: Fixing quality problems in the dashboard layer
By the time a dashboard shows a wrong number, the damage is already downstream. Detection belongs next to the data.
Real vs simulatedWhat this does and does not show
The data is GitHub Archive events. The health score is a simple ratio and does not weight rules by severity. Orchestration, alerting and a data catalog integration are listed as future work, not built.
Next
- Orchestration with Airflow or Dagster.
- Alerting to Slack or email.
- Severity-weighted health scores.