Spot missing fields and type drift in your DynamoDB tables

Build me a data quality explorer for our Amazon DynamoDB tables. I open it whenever we ship a schema change or onboard someone new to the data model. DynamoDB lets every item carry different attributes, and only key attributes can be formally declared, so non key attributes are completely unpoliced and nobody can tell me what is actually in a table. The app should show me what is really there, let me declare what ought to be there, and list the items that disagree.

Start on a table picker. Use List Tables to show every table in the connected region and let me choose one. Once I pick a table, call Describe Table and show its structure: the partition key and sort key, and every global and local secondary index with the attributes each one is keyed on. Present this as the declared schema and make clear it is the only part of the model DynamoDB actually enforces.

Then sample the table with Scan. Sampling must be capped and paginated. Use a modest page size and follow LastEvaluatedKey into ExclusiveStartKey on each subsequent request, stopping the moment the sampled item count reaches a cap I control. Default the cap to about 1000 items and expose it as an input on the view. Always tell me how many items were sampled and whether the cap was hit before the table ran out. Never run an unbounded full table scan, and never scan automatically on page load without me asking for a refresh.

From the sample build an attribute profile. For every attribute name seen anywhere in the sample, show the number of sampled items carrying it, the percentage of sampled items carrying it, and the DynamoDB types observed for it with a count per type. When one attribute shows more than one type across the sample, for example sometimes a string and sometimes a number, flag it as a type conflict and make it visually prominent, because those are the bugs worth chasing. Sort free form attributes by coverage descending.

Split the profile into two clearly separated groups. One group is key attributes, meaning anything named in the table key schema or in any index key schema from Describe Table. The other group is free form attributes, meaning everything else. This separation is the point of the app, so do not blend the two into one flat list.

Handle single table design properly. Derive an entity type for each sampled item by reading the prefix of its partition key value up to a delimiter, defaulting to a hash character and letting me change the delimiter. Group the attribute profile by entity type as well as showing the table wide view, since a customer item and an order item in the same table have nothing to do with each other. Items whose partition key has no delimiter fall into an untyped group.

Give me a rules panel where I declare, per entity type, which attributes are required. Rules are always per entity type and never global. Bake in that a missing secondary index key attribute is not automatically a violation: sparse indexes are intentional in DynamoDB and teams leave index keys off items deliberately to keep those indexes small, so an attribute is only ever reported as missing when I have declared it required for that entity type.

Persist the rules with Put Item into a small config table whose name I supply, writing one config item per profiled table so the rules load back with Get Item the next time I open the app. Alongside the rules, store a snapshot of the current profile, meaning the coverage percentage per attribute plus the sample size and a timestamp, so the app can compare against it on my next visit.

Add a violations view that evaluates the sampled items against my saved rules and lists every failing item. Each row shows the item key, its entity type, and a specific human readable reason such as missing required attribute email, or attribute status expected string but found number. One reason per row, written so a new teammate understands it without reading the rules panel first.

Clicking a violation row opens the full item, fetched fresh with Get Item so I am never editing a stale copy from the sample. Give me two modes. Read only, for when I just want to inspect the item. And edit, where I can correct an existing attribute or add a missing one inline, one attribute at a time, committed with Update Item behind an explicit confirm step that shows the before and after value. The app must never bulk edit items automatically: no fix all button, no background repair, no cascading writes. Every write is one item, one attribute, one confirmation.

Finally add a share button that composes a short drift summary and posts it to our engineering channel using the Slack Bot Send a Message operation, with the channel configurable. The summary covers attribute coverage changes against the last stored snapshot, meaning attributes that newly appeared, attributes that disappeared, and attributes whose coverage moved by more than a threshold such as five percentage points, plus the current violation count and the sample size the numbers came from. Keep it to a handful of lines that read well in Slack rather than a full dump of the profile.

Throughout, label coverage numbers as sample based rather than exact table statistics, so nobody mistakes an 80 percent reading for a guarantee about the whole table.