Pre-2.1 ALTERed TEXT columns: mixed-row 'invalid hash length: 19' panics on 2.1.4; schema-encoding-drift check reports clean; rows not repairable via SQL

[marcodelpin]

Follow-up from #11131 (comment) — filing as a separate issue as requested by @timsehn.

We hit what looks like a variant of #11131 in production-style usage, and the repair tool from the zachmu/schema-repair-tool branch does not detect it.

Setup

• Single dolt sql-server instance hosting ~140 databases. SELECT dolt_version() → 2.1.4 (lineage: 2.0.x era in April → 2.1.2 → 2.1.4).
• The affected TEXT/LONGTEXT columns were ALTERed by an application schema migration during the 2.0.x era (the unreadable rows cluster around the original data-import date, pre-ALTER) — consistent with the Dolt 2.0.8: adaptive out-of-line TEXT written via migration is unreadable — "invalid hash length: 19" on read (regression vs 2.0.4) #11131 root cause (ModifyColumn skipping the row rewrite).

Symptom — same signature as #11131, on 2.1.4

Any read that forces a decode of an affected cell panics server-side and drops the connection:

Error 1105 (HY000): panic recovered: invalid hash length: 19
...
github.com/dolthub/dolt/go/store/hash.New
github.com/dolthub/dolt/go/store/val.AdaptiveValue.convertToTextStorage
	github.com/dolthub/dolt/go/store/val/adaptive_value.go:202
github.com/dolthub/dolt/go/store/val.(*TupleDesc).GetStringAdaptiveValue
	github.com/dolthub/dolt/go/store/val/tuple_descriptor.go:618
github.com/dolthub/dolt/go/libraries/doltcore/sqle/index.prollyRowIter.Next

(gms in the recovered stack: v0.20.1-0.20260605175459-433dbaebc97f)

How it differs from the original #11131 report

1. Mixed tables. Only the rows that existed before the ALTER are unreadable (e.g. 133 of 436 rows in one table, clustered around the original data-import date). Rows written after the ALTER read fine — including recent rows with >2KB values, so this is not value-size-correlated. SELECT COUNT(*) is fine; SELECT COUNT(LENGTH(col)) panics; per-row probes isolate exactly which rows are bad.
2. schema-encoding-drift check reports clean. We built the tool from zachmu/schema-repair-tool @ 709aad077e and ran it (read-only) against the live data dir of the affected databases: no schema-encoding drift detected on every one of them, including a database where the panic is 100% reproducible. So either the heterogeneous-bucket classification misses this shape, or the tool (its build reports dolt version 2.0.8) can't read schema records written/migrated by 2.1.x.
3. No SQL-level self-service. UPDATE t SET col = NULL WHERE id = '<bad-row>' and DELETE FROM t WHERE id = '<bad-row>' both panic the same way (row materialization decodes the old value), so we can't even null-out or drop the affected rows from SQL.
4. Scale: out of ~140 databases on the instance, 19 have an affected table of one shape, 5 of another, and 4 have it on their primary data table.

Questions

1. Is the schema-repair-tool branch expected to detect/repair mixed-row tables on 2.1.x-written storage? If not, is there a recommended path for this variant?
2. Given that the writer was 2.0.x-era ALTER (so presumably schema-tag drift with intact row bytes, as diagnosed in Dolt 2.0.8: adaptive out-of-line TEXT written via migration is unreadable — "invalid hash length: 19" on read (regression vs 2.0.4) #11131), would a 2.1.4-built recover-rows-style pass be the right tool, and is there a build of it that understands current storage?

We can share a scrubbed copy of an affected database directory privately, and we're on Discord if real-time is easier. For now we've patched our application to skip-and-warn on this signature during its own data migrations, so we're unblocked operationally — but the affected cells remain unreadable.

[zachmu]

Hi @marcodelpin, I'm looking into this now. It sounds like the repair tool isn't working for some reason. The best way to make traction on this is to join our discord and mention me there so we can debug in real time.

[zachmu]

I've pushed changes to the https://github.com/dolthub/dolt/tree/zachmu/schema-repair-tool branch. The only way they help, though, is if the tables you were trying to migrate had no primary key. The original tool lacked support for tables with no primary key, and would incorrectly report no schema drift detected under those circumstances. I've tested locally and verified that the new version correctly handles tables without a primary key.

If your table schemas were not keyless, then this fix is unlikely to help you and the problem is likely something else.

Either way, please join our discord server so that we can discuss in real time:

https://discord.com/invite/RFwfYpu

[marcodelpin]

Thanks @zachmu — rebuilt from the branch tip (d0a5bf3d96) and have results, including a full production repair. Two findings, one of which explains the false "no drift detected" in my original report.

Answering your question first: our tables are NOT keyless — every affected table has an explicit primary key. The most common shape:

CREATE TABLE `events` (
  `id` char(36) NOT NULL,
  ...
  `old_value` longtext,
  `new_value` longtext,
  PRIMARY KEY (`id`),
  ...
)

The "no schema-encoding drift detected" in my report was a false clean on our side — but there's a UX trap worth fixing. Re-running the rebuilt tool with the working directory inside a database directory, it detects our drift just fine. Our original run invoked check with the cwd at the multi-db data root (the directory containing the database dirs, sql-server style): in that case the tool apparently resolves zero databases, prints the same no schema-encoding drift detected, and exits 0 — indistinguishable from a genuinely clean scan. I reproduced this: same binary, same data, cwd at the data root → "clean"; cwd inside any db dir → 28 findings. Suggestion: print the number of databases/tables scanned (or error out when zero) so a no-op scan can't masquerade as a clean bill of health.

With the correct cwd, check detects our mixed-row variant on PK tables, across copies of all ~140 databases on the instance:

28 total — 2 drift (repair by default), 26 heterogeneous (recover-rows), 0 safe-empty

14 databases, all the same shape: the two LONGTEXT columns of one table (events.old_value / events.new_value), declared StringAdaptiveEnc, observed mixed-legacy-and-adaptive. 13 heterogeneous (they received writes after the ALTER), 1 pure drift (no post-ALTER writes).

Production repair: done and verified. After a backup we stopped the dolt sql-server and ran recover-rows per column on the 13 heterogeneous databases (~8k rows rewritten across 26 columns) plus repair on the pure-drift one. Post-repair check was clean, and the previously-panicking probe (SELECT COUNT(LENGTH(col)) FROM events) read clean on 14/14 databases with stock dolt 2.1.6 — it panicked on every one of them before (invalid hash length: 19). Content intact.

We then followed up with migrate-adaptive on the same 28 columns to land on the canonical v2 adaptive encoding rather than staying on the legacy end-state (~23k legacy rows rewritten). Verified the same way on a copy first, then in production: check clean, all 14 databases reading fine with stock 2.1.6.

From our side the incident is fully resolved — thanks for the quick turnaround. The keyless fixes weren't what unblocked us (our tables all have PKs), but the cwd finding above probably explains the original mystery, and the tool handled the PK mixed-row case end-to-end — detection, recover-rows, and migrate-adaptive — without issues.

[zachmu]

That's great to hear.

Please be advised that this tool doesn't migrate historical revisions of the database, only the current head. So this problem may exist at other revisions and come up again in the context of historical queries. If you need those revisions migrated as well, we can help you out with that, just let us know.