You have a working model. Staging looks clean. Stakeholders are aligned. Two weeks from launch. This is the moment to ask twelve questions about your data. If you cannot answer all of them confidently, the launch date should move. The cost of answering them after a production failure is an order of magnitude higher.
Domain 1: Schema and Structure
1. Can you produce the complete lineage of every feature in your training set? Not approximately. Complete, documentable lineage from source to feature store to training dataset.
2. Are there field names appearing in multiple upstream systems with different meanings? Customer ID, account ID, product ID are the most common offenders. Document how each is disambiguated.
3. Have all schema-breaking changes in the past 90 days been reviewed for impact on model inputs? If a field was renamed, retyped, or removed in the last three months, verify the feature engineering was updated accordingly.
Domain 2: Data Quality
4. Does every critical feature have a defined quality threshold? Completeness, range, uniqueness, and freshness thresholds should be documented and enforced. Checking the data visually is not a threshold.
5. What is your current null rate for each feature, and is it within expected bounds? A null rate stable at 2% that suddenly jumps to 18% is a signal. Without a baseline, you will not see the jump.
6. Has the distribution of each key feature been compared between training and recent production data? Training-serving skew is one of the most common causes of post-launch degradation. A distribution comparison before launch is not optional.
Domain 3: Pipeline Reliability
7. What happens when an upstream source is unavailable? Does the pipeline wait, fail fast, or use stale data? Is the behavior documented? Does the AI system handle each case correctly?
8. What is the P99 latency of your feature delivery pipeline under production load? If you have not tested under realistic load, you do not know this number. Measure it before launch, not after.
9. Is there a tested fallback for every feature that could be delayed or unavailable? Fallbacks should be explicit and tested.
Domain 4: Governance and Ownership
10. Is there a named owner for every data source feeding the model? When something breaks, who is accountable? If the answer is the data team, that is not specific enough.
11. Are there data contracts in place for the three most critical pipeline inputs? The highest-value features should have formal contracts with enforcement. Advisory documentation is not a contract.
12. Do you have a defined process for handling a contract violation after launch? Who gets alerted? What is the SLA for resolution? Does the AI system degrade gracefully or halt?
Run through these questions two weeks before your target launch date, not two days. The questions you cannot answer are your pre-launch backlog. Addressing them before launch costs hours. Addressing them after costs weeks.
Want help running this checklist? Talk to the Accelerate Data team.