Best Practices

Data Quality Governance: The Fallout System Approach

Learn how the Nexus Model Architecture's Fallout System tracks data quality issues without blocking pipelines, enabling systematic resolution and continuous improvement.

By Van Rij Data Partners January 15, 2025

The Problem with Traditional Data Quality

Traditional data quality approaches follow an all-or-nothing model: one bad record causes the entire ETL pipeline to fail. This creates a cascade of problems:

All-or-nothing: One bad record blocks thousands of good records
Limited visibility: Error logs are cryptic and scattered
Slow resolution: Manual investigation required for each failure
Business impact: Reports missing data without explanation

The result? Frustrated stakeholders, delayed insights, and IT teams stuck in firefighting mode.

The Fallout System: Fail Gracefully, Track Everything

This "fail gracefully" philosophy ensures business continuity while maintaining complete visibility into data quality issues.

%%{init: {'theme':'base'}}%% graph TD subgraph Traditional["Traditional ETL"] A[fa:fa-database Source Data] --> B{fa:fa-exclamation-triangle One Bad Record?} B -- Yes --> C[fa:fa-stop-circle Pipeline Fails] B -- No --> D[fa:fa-check-circle Load Success] end subgraph NMA["NMA Fallout System"] E[fa:fa-database Source Data] --> F{fa:fa-question-circle Any Bad Records?} F -- Yes --> G[fa:fa-folder-open Log to Fallout Table] F -- No ----> H[fa:fa-check-double Load Good Records] G --> H end style Traditional fill:#fff3e0 style NMA fill:#e8f5e9

How It Works

The Fallout System consists of two core tables:

Fallout.Fallout

The main tracking table that logs every data quality issue:

CREATE TABLE [Fallout].[Fallout](
    [FalloutId] [int] IDENTITY(1,1) NOT NULL,
    [ErrorCodeId] [int] NOT NULL,                    -- Type of error
    [SourceSystemId] [bigint] NOT NULL,              -- Which source
    [SourceSystemIdentifier] [nvarchar](100) NOT NULL, -- Which record
    [InsertDate] [datetime] NOT NULL,                -- When detected
    [SolveDate] [datetime] NULL,                     -- When resolved
    CONSTRAINT [PK_Fallout] PRIMARY KEY ([FalloutId])
)

Fallout.ErrorCode

A taxonomy of error types with resolution guidance:

11-19: Referential Integrity (missing foreign keys)
20-29: Data Format (invalid types, dates, numbers)
30-39: Business Logic (rule violations)
40-49: Data Quality (duplicates, missing fields)

Real-World Results

The Fallout System delivers measurable business value:

Pricewise: Saved 40+ hours/week in manual reconciliation
Creative Clicks: Reduced data errors by 85%
Business transparency: Stakeholders see what's missing and why
Faster resolution: Systematic analysis replaces ad-hoc investigation

The Resolution Workflow

%%{init: {'theme':'base'}}%% flowchart TD A(fa:fa-search Detection) --> B(fa:fa-sort-amount-down Prioritization) B --> C(fa:fa-user-secret Investigation) C --> D(fa:fa-tools Resolution) D --> E(fa:fa-sync-alt Reprocessing) E --> F(fa:fa-check-square Verification) style A fill:#e0f2fe,stroke:#3b82f6 style B fill:#e0f2fe,stroke:#3b82f6 style C fill:#e0f2fe,stroke:#3b82f6 style D fill:#fef3c7,stroke:#f59e0b style E fill:#fef3c7,stroke:#f59e0b style F fill:#dcfce7,stroke:#10b981

Conclusion: Quality Without Compromise

The Fallout System proves that data quality doesn't have to mean pipeline failures. By logging issues transparently while continuing to process good data, organizations can:

Maintain business continuity
Build trust through transparency
Resolve issues systematically
Improve data quality over time

It's a fundamental shift from reactive firefighting to proactive governance—and it works.