Merged PRs

dolt

• 11366: Include full table scan in index statistics
  Have StatsController also create an "empty" statistic entry to indicate no index for costing.
• 11365: Add dolt_squash_history procedure
  This new procedure will take the content of HEAD, and squash history before it. Default behavior is to squash every commit from height 2 (commit right after initial commit) to HEAD. It is impossible to rewrite the initial commit.
  This will be much faster than rebasing all history.
• 11358: Fix left outer merge join wrong results with residual filters
  Tests added in dolthub/go-mysql-server#3655
  Fixes #11350
  Based on #11351
• 11354: add missing execution call for sqlparser.Execute statements
  We were missing calls to iter.Next() and iter.Close(), when executing prepared statements through dolt sql -q "...",
  some tests in: dolthub/go-mysql-server#3653
  fixes: #11345
• 11351: kvexec: fix left outer merge join wrong results with residual filters
  Fixes #11350 — wrong results from mergeJoinKvIter for left outer merge joins whose key runs contain duplicates and whose join condition carries a residual (non-merge-key) filter.
  Two one-line-class defects, both in the match: loop:
  1. l.matchedLeft = ok overwrote earlier successes. A left row that matched mid-run was re-flagged unmatched by a later failed pairing in the same run, so the run transition emitted a spurious null-extended row for it. Changed to l.matchedLeft = l.matchedLeft || ok; the run-transition reset remains the sole reset point.
  2. io.EOF during the left advance dropped the final owed null-extension. The compare: loop handles left-EOF via oldLeftKey; the match: loop's advance branch returned the error immediately, losing the null-extended row for a final unmatched left row. Now it emits that row (when isLeftJoin && !matchedLeft) before surfacing EOF.
     Minimal repro (returns fr:app|NULL instead of nl:app|NULL before this patch):

  create table b2 (cat varchar(50) not null, code varchar(16) not null, lang varchar(20) not null,
  primary key(cat, code, lang), key(code));
  create table t2 (id varchar(36) primary key, code varchar(16), lang varchar(16), key(code, lang));
  insert into b2 values ('cat0','P1','de:app'),('cat1','P1','fr:app'),('cat2','P1','nl:app');
  insert into t2 values ('t1','P1','de'),('t2','P1','es'),('t3','P1','fr');
  select /*+ MERGE_JOIN(p,w) */ p.lang, w.lang
  from b2 p left join t2 w on w.code = p.code and w.lang = substring_index(p.lang, ':', 1);

  The regression test registers the kvexec ExecBuilder on the test engine (as the CLI/server engines do) so the kvexec iterator — not the GMS fallback, which was already correct — is what's exercised; the new test fails 7 assertions without the fix and passes with it. kvexec suite and enginetest TestJoinQueries/TestJoinOps pass. Also verified against a 15k-row synthetic (anti-join now agrees exactly with NOT EXISTS: 5,000) and the production database where this was found (197-row orphan audit no longer inflates to ~15k when the planner picks the merge join path).
  🤖 Generated with Claude Code
  https://claude.ai/code/session_01VpTxMfzCXHoM8ojJn5aQiv
• 11344: Fail amend commits when branch head has moved
  When a SQL transaction ran CALL DOLT_COMMIT('--amend') after another connection had added a commit or merged a branch, the amend silently replaced the transaction HEAD commit up to the new HEAD commit. The data from the erased commits survived, but the commit itself, including merge records we're gone.
  • datas.CommitOptions.Amend is replaced with AmendedCommit (hash.Hash), the address of the commit an amend replaces.
  • datas.BuildNewCommit rejects an amend unless AmendedCommit is still the dataset HEAD, relying on the CAS in CommitWithWorkingSet` for atomic comparison.
  • dsess.doCommit runs validateAmendedHead before the working set merge to specify the head conflict over a row conflict.
  • Amend is refused mid merge, cherry-pick, in parity with git and revert, a Dolt-specific addition because it follows the same merge state.
  • The admin createchunk command's HEAD check bypass moves from the Amend flag to a new explicit CommitOptions.Force field.
    Fix #9072
• 11343: Add retries to branch creation on ErrOptimisticLockFailed errors
  Adds a retry loop to NewBranchAtCommit so branch creation can recover from a concurrent optimistic-lock conflict on the working set (e.g. from a background auto-GC cycle). This matches the existing retry behavior of other write paths (e.g. SQL transaction commit).
• 11333: Bug fixes for secondary indexes using virtual columns
  When a virtual generated column is used in a secondary index, rebuilding the index can generate incorrect index data in some situations. For example, dropping a column in the table will trigger a table rewrite as well a rebuilding the secondary indexes. In this case, the virtual generated column expressions were not properly being evaluated to store the correct data in the index.

go-mysql-server

• 3657: check stats prov for empty stat
  Instead of creating the (potentially same) empty stat every time, we should check if it has already been made in the StatsProvider.

• 3656: Bug fix for CASE expressions with ExtendedType instances
  Related to: dolthub/doltgresql#2980
  Doltgres fix: dolthub/doltgresql#2987

• 3653: adding prepare tests for insert and update

• 3652: Allow aliasing column names in a table function

  Depends on: dolthub/vitess#477

• 3644: add ExtendedTableFunction interface to support table function schema …
  …on OUT parameters

• 3643: Use buildScalar to get GetField expression for ON UPDATE columns
  fixes #11346
  Building the GetField expression for a column using its index in a table was causing indexing issues when there was also a CTE. Instead, we need to build the the GetField using buildScalar to properly resolve the column within the scope.

• 3641: Bug fix for a panic when calling a stored procedure without a database selected

• 3639: have coster consider not picking secondary indexes
  This PR adds a "no index" option to the coster and some additional heuristics for index costing.
  Before, we would always pick any available index, which would sometimes be sub-optimal, especially if the index is a non-covering secondary index.
  Additionally, the normal_dist and exp_dist tables are now seeded to reduce random variability between test runs.
  Benchmarks:
  #11336 (comment)

• 3638: Bug fixes for secondary indexes using virtual columns
  When updating a row in a secondary index, any virtual generated columns need their generation expression evaluated to get the correct data stored in the secondary index. There were a few edge case with virtual columns and secondary indexes where this evaluation wasn't happening. This PR closes those gaps.
  The first case is when a table rewrite is performed and secondary indexes are rebuilt (e.g. when dropping a column on a table). Not properly evaluating a virtual column's expression caused incorrect data to be stored in the index when it was rebuilt.
  A second case is for referential actions on a foreign key that update a table and its secondary index. Not properly evaluating a virtual column's expression here can also cause incorrect data to be written to the secondary index.
  Fix for failing Dolt CI integration test in: #11333

• 3634: Add support for multiple expressions in functional indexes

• 3623: memory: use stable sort for secondary index ordering
  memory.TableData.sortSecondaryIndexes sorted secondary index storage with sort.Slice, which uses Go's unstable pdqsort algorithm. Rows that tie on the indexed columns (same key, different primary key) could be reordered relative to each other on every sort, so repeated index rebuilds of the same data could produce different physical row orders.
  This PR switches to sort.SliceStable, which preserves the relative (insertion) order of tied rows, making secondary index storage ordering deterministic across rebuilds.
  Fixes #2877

• 3618: Error on SELECT @@SESSION., matching MySQL
  GMS silently returned the global value when a GLOBAL-only system variable was read with an explicitly-qualified SESSION/LOCAL scope (e.g. SELECT @@SESSION.innodb_autoinc_lock_mode), instead of raising MySQL's ERROR 1238 (ErrSystemVariableGlobalOnly).
  buildSysVar's case for SetScope_None/SetScope_Session already carries specifiedScope, which is empty for a bare @@foo reference and non-empty ("session"/"local") only when the scope was expli...

Merged PRs

dolt

• 11332: /go/cmd/dolt/cli: only emit backspace characters in a tty
  The CLI's progress spinners (dolt push, pull, fetch, assist, and import/sql progress output) redraw themselves in place by writing backspace control characters. This was done unconditionally, with no check for whether output was going to a terminal.
  When stdout/stderr is captured to a file or pipe — a script, a CI job, dolt push > push.log 2>&1 — those backspaces become literal bytes in the log instead of cursor movements. Rendering the log then "erases" the surrounding text. The worst case: an error is printed while the spinner is active, and the spinner's trailing backspaces overwrite the error text, so the captured log looks empty exactly when something went wrong.
  This PR fixes this so that TTY is detected and redraw is gated on it.
• 11327: Add support for functional indexes with multiple expressions
  Picks up support for functional indexes with multiple expressions from GMS (dolthub/go-mysql-server#3634) and adds additional tests for Dolt.
  Depends on: dolthub/go-mysql-server#3634
• 11326: unsafe adaptive value
• 11325: gitblobstore: fix premature flush and prune of pending write that was not referenced in current manifest
  CheckAndPut("manifest") flushed all queued pendingWrites into the commit, regardless of whether the manifest being written referenced them. That commits a table file into the git tree as an unreferenced entry, which a later manifest flush then legitimately prunes and evicts from the in-memory cache — while the file is still live (about to be added, or being re-read by addTableFiles' refCheckAllSources retry loop). The result is Blob not found. Stale pendingWrites from earlier failed pushes in the same process keep the condition sticky, which is why a restart clears it.
  This PR only flushes the pending writes the manifest actually references. Unreferenced writes stay pending until a manifest that references them is written, so they're committed and referenced atomically. After this change a live table file is always either pending and cached (not in the tree, so unprunable) or committed and referenced (never pruned) — an unreferenced tree entry can no longer coexist with being live.
• 11320: go: sql_server_driver: Fix a race on *SqlServer.Cmd access. Start the command before we spawn its wait goroutine.
• 11316: go: sqle/cluster: Improve dolt_cluster_ack_writes_timeout_secs circuit breaker behavior.
  Previously, the circuit breaker only had an open and a closed state. It would only go from open to closed when the replication thread fully quiesced. This PR adds a half-open state, where it lets a single commit probe for replication success within the configured timeout. If that wait succeeds, the circuit breaker closes.
  This lets high write throughput use cases reenable their configured replication waits even when writes arrive more requently than their typical replication delay.
• 11314: go: sqle/cluster: controller.go: Make DROP DATABASE replication participate in dolt_cluster_ack_writes_timeout_secs.
• 11313: go: Fix some data races surfaced by running go-sql-server-driver tests under -race.
• 11312: go/store/nbs: Fix a race between the journal persister and PruneTableFiles.
  When PruneTableFiles runs, it is willing to delete the journal file if there is no longer a reference to it in the manifest. This can race with bootstrapJournal making a new journal file. The journal file can be on disk, and seen by PruneTableFiles and thus selected for removal, before the journal has been added to the chunk store proper.
  This change makes the journal writer participate in the retained files machinery, same as the table file persister, so that the journal is correctly considered not-safe-for-delete-by-PruneTableFiles before it is ever created on disk and potentially selected for deletion by PruneTableFiles.
• 11305: fix stats functions to include schema
  We lookup stats with empty string for schema. Since this is not always the case, we sometimes miss stats.
• 11290: Bump golang.org/x/net v0.54.0 => v0.55.0 for Go 1.27 compatibility
  Go 1.27 change:
  • https://github.com/golang/go/discussions/60746
  • golang/go#67810
    Problem found in
  • Homebrew/homebrew-core#288803
    Build error:

  $ go1.27rc1 build -v ./cmd/dolt
  google.golang.org/grpc/internal/transport
  # google.golang.org/grpc/internal/transport
  ../../../../go/pkg/mod/google.golang.org/grpc@v1.79.3/internal/transport/handler_server.go:271:18: undefined: http2.TrailerPrefix
  

  golang.org/x/net upgrade

  $ go get golang.org/x/net@v0.55.0 && go mod tidy
  go: upgraded golang.org/x/net v0.54.0 => v0.55.0
  

  Build successful:

  $ go1.27rc1 build -v ./cmd/dolt
  

go-mysql-server

• 3634: Add support for multiple expressions in functional indexes
• 3627: fix stats functions to include schema
  We include the schema in the stats key, but only sometimes fill in the field.
• 3626: Prevent index offset when comparing rows in topRowsIter
  Fixes #11300
  Appending the row order number to the end of a sql.Row during a Top-N Heap Sort in topRowsIter was causing an index offset when evaluating SortFields that were subqueries, thus resulting in incorrect result.
  This PR
  • modifies topRowsHeap to instead take a rowWithOrder struct that separates out the sql.Row from the order number while still taking the order number into account when sorting the heap.
  • refactoring Sorter.LesserRow logic into a new CompareRows function to allow checking for row equality
  • moves top row(s) iterators to its own file as part of an effort to make our iterators more organized (#3620). This file also includes topRowsHeap since it is only ever used by topRowIter (see dolthub/go-mysql-server#3622 (comment) for next steps)
  • removes ValueRowSortersince it's actually never used anywhere and doesn't even fully implement sort.Interface
• 3617: special case on extended type for getting compare type

Closed Issues

• 11265: Dolt INSERT … SELECT with CTE + JSON_TABLE disconnects
• 3622: Clean up: move sorters in sql/expression/sort.go out of expression package

Performance

Merged PRs

dolt

• 11310: cluster: wait for outstanding DROP DATABASE replication during graceful transition to standby
  dolt_cluster_transition_to_standby determines whether a standby replica is fully caught up by waiting on three replication subsystems: per-database commit hooks, users/grants, and branch control. It never accounted for outstanding DROP DATABASE replications, which are tracked separately in Controller.outstandingDropDatabases and driven by independent fire-and-forget goroutines.
  As a result, the transition could report a standby as fully caught up and succeed while a DROP DATABASE had not yet replicated to it. This PR makes it so that dolt_cluster_transition_to_standby also waits for outstanding DROP DATABASE statements to replicate.
• 11308: go.mod: bump eventsapi_schema to latest revision

go-mysql-server

• 3626: Prevent index offset when comparing rows in topRowsIter
  Fixes #11300
  Appending the row order number to the end of a sql.Row during a Top-N Heap Sort in topRowsIter was causing an index offset when evaluating SortFields that were subqueries, thus resulting in incorrect result.
  This PR
  • modifies topRowsHeap to instead take a rowWithOrder struct that separates out the sql.Row from the order number while still taking the order number into account when sorting the heap.
  • refactoring Sorter.LesserRow logic into a new CompareRows function to allow checking for row equality
  • moves top row(s) iterators to its own file as part of an effort to make our iterators more organized (#3620). This file also includes topRowsHeap since it is only ever used by topRowIter (see dolthub/go-mysql-server#3622 (comment) for next steps)
  • removes ValueRowSortersince it's actually never used anywhere and doesn't even fully implement sort.Interface
• 3621: bug fix: honor named window reference
  An existing bug in GMS was not properly applying a named window reference. We had enginetests for named window references, but they weren't sufficient to catch this because the aggregate function they used produced identical values in both cases. New enginetest cases are added to prevent a regression.
  These two issues were identified by Ito automated review, in dolthub/doltgresql#2913
• 3619: sql: Fix ALTER USER to allow the Identity field of the User record to be updated.
  CREATE USER ... IDENTITY WITH <plugin> AS '<identity>' correctly parsed and persisted the Identity field. ALTER USER correctly parsed the Identity field but failed to persist the changes. The end result is that GMS had a bug where attempt to alter the identity field on an existing user seemed to succeed but was not reflected in the data going forward.
  Fix the bug so that updates to Identity are reflected and persisted going forward.

Closed Issues

• 11300: ORDER BY subquery loses effect as soon as LIMIT and OFFSET become involved

Performance

This release is nearly identical to 2.1.11. There are no novel merged PRs in this release.

This release is a minor version bump because the authentication of tokens with auth plugin authentication_dolt_jwt changed in 2.1.11. This could potentially effect deployments which are currently minting JWTs tokens for authentication to Dolt, but it is not expected to effect any existing deployment. See #11296 for more details.

Performance

Merged PRs

dolt

• 11302: /go/libraries/doltcore/sqle/{dsess,enginetest}: add regression test and lazy load db
  Creating databases and then querying them across multiple sessions intermittently fails with could not resolve initial root for database /.
  A DoltTransaction snapshots the noms root of every known database when it begins. If one session creates a database after another session's transaction has started, the second session can resolve that database from the shared provider, but its transaction has no start-point for it, so TransactionRoot errors. The only code that added late-appearing databases to a transaction was the read-replica clone path; nothing handled the concurrent-create case.
  This PR makes it so that in DoltSession.addDB, if a database isn't in the transaction's snapshot, lazily register it (tx.AddDb) with its current root on first reference — same behavior the clone path already relies on. Also normalized AddDb's map key to the base name to match NewDoltTransaction/GetInitialRoot.
• 11298: fsck should accept stash objects
  fixes: #11292
• 11296: go: jwtauth: When using auth plugin authentication_dolt_jwt, validate the Subject claim against the expected claims.
  A bug in JWT token authentication meant that an incoming token's Subject claim was not validated against the set of expected claims listed in the User record.
  The end result is that a given, valid JWT token was not scoped to authenticating a single user account. Any valid JWT token bearing the correct signature, audience and issuer could be used to connect as any user account which was configured to accept that JWKS source, issuer and audenience field.
  The fungible token behavior is too big of a footgun to leave in place as the default. To restore fungible token behavior, leave the sub= claim off the expected claims in the User Identity field.
  This is a backward incompatible change. Existing workflows expecting fungible tokens to work may need to change token issuance or User record Identity fields to upgrade.
• 11286: New engine init interface
  This lets integrators register engine initialization logic before connections are accepted.
• 11285: Correctly convert foreign key lookups across encoding type boundaries
  Tests are in Doltgres, should be a no-op for Dolt
  Companion PR: dolthub/doltgresql#2914
• 11283: go: sqle: Cleanup comments and function names in DatabaseProvider around creatingDatabases.
  Restructure the regression test to be a go-sql-server-driver test and to run a self-clone.
• 11281: go: sqle: Fix concurrent-clone deadlock between two remotesapi sql-servers
  DoltDatabaseProvider.CloneDatabaseFromRemote held the provider RWMutex write lock across the entire clone, including the network fetch of all chunks from the remote. When a server serves a --remotesapi-port, that fetch hits a peer's remotesapi whose handler needs a read lock on this same provider (via SessionDatabase). Two sql-servers that CALL DOLT_CLONE from each other's remotesapi at the same time therefore deadlock: each holds its own provider write lock while blocked on the peer, and neither peer can take the read lock needed to answer. DOLT_PULL is immune because it operates on an already-registered database and does not hold the write lock across its fetch. This is the lock the existing // TODO: This holds the database provider lock across the entire duration of the clone ... was warning about.
  The fix narrows the lock scope so the network fetch does not run under the provider write lock:
  • Reserve the database name under a brief lock acquisition (a new creatingDatabases set), release the lock for the fetch, then re-acquire it only to register the finished database (registerNewDatabase still requires the lock).
  • The reservation keeps concurrent CREATE DATABASE / dolt_clone / undrop of the same name mutually exclusive, so those races behave as before even though the lock is dropped for the fetch.
  • A shared databaseNameUnavailableLocked helper performs the availability checks for all creation paths and preserves the interrupted-create in-progress marker behaviour (ErrIncompleteDatabaseDir / dbfactory.MarkDatabaseInProgress). The in-memory reservation and the on-disk marker are complementary: the marker guards crash recovery across processes, while creatingDatabases serialises concurrent in-process operations while the lock is dropped.
  • Names in creatingDatabases and deletingDatabases are normalised with formatDbMapKeyName (Dolt database names are case-insensitive), matching how p.databases is keyed. Unlike deletingDatabases, creatingDatabases deliberately does not gate database enumeration: an in-progress clone is simply invisible until it registers, so enumeration / remotesapi reads never block on it.
    Tests:
  • integration-tests/bats/sql-server-remotesrv.bats: a regression test that starts two --remotesapi-port servers cloning from each other concurrently. It hangs (clones time out) on the buggy build and passes on the fixed build; it also fails if a clone errors for any non-deadlock reason.
  • go/libraries/doltcore/sqle/database_provider_test.go: TestCreatingDatabaseReservation covering the reservation semantics — conflict, case-insensitivity across create/clone/undrop, release, and that a reservation does not block AllDatabases.
    Verified with go test -race ./libraries/doltcore/sqle/ and the new bats test; gofmt / goimports clean.
    Fixes #11280
• 11269: Fix spurious unique key error when staging workspace rows
  Staging a delete and an insert that reuse the same unique key via dolt_workspace_* failed with a spurious duplicate unique key error, because rows were applied in workspace order and checked per row.
  • Updater now applies all deletes first and defers the puts to statement completion
    Fix #11195
• 11267: go: grpcreresolve: Add a load balancer implementation which requests ResolveNow for endpoints when we see Unavailable or DeadlineExceeded.
  When running Dolt within a k8s cluster with a service mesh like Istio, the L4 connection to the first-hop load balancer can stay open even when the endpoint associated with that name moves on. gRPC's default pick_first load balancer will not reresolve the endpoints associated with the name unless the L4 connection is lost, which causes persistent failures instead of quick recovery.
  Here we implement a new load balancer strategy which just wraps pick_first. It listens for Done on the RPCs, and it calls resolver.ResolveNow if it sees Unavailable or DeadlineExceeded. The DNS resolver in gRPC already debounces resolve requests, so this does not spam reresolves too quickly in the case of true upstream unavailability.
• 11266: Bug fix:adaptive encoding inserts to a unique index
  Uses the ValueStore from the ExtendedTupleComparator instance when set, so that out-of-band data can be correctly loaded.
  Related: dolthub/doltgresql#2886
• 11247: Fix slow clone/fetch from git remotes
  Reading a git-backed table file went chunk by chunk, and because git cat-file cannot serve a byte range, each read re-streamed the whole object from the start. Fetching a large table spawned thousands of git processes and stalled.
  • Spool a git-backed table file to a local temp file on open and serve its chunk reads locally
  • Look up the sizes of a chunked object's parts in a single git cat-file --batch-check call
  • Detect an early reader close by end of stdout, so a partial read is torn down the same way on every platform
    Fix #11236
• 11246: Skip incomplete database directories during discovery
  Dolt can leave a half-written directory on disk when creating or cloning a database, affecting INFORMATION_SCHEMA and similar queries to error. Dolt now places a marker file in the directory before creation starts and durably removes it once the database is complete.
  • Dolt storage without a repo state file is now skipped with a warning at load time
  • Write .dolt_safe_to_ignore marker file when creation a database
  • Skip marked files or incomplete directories with a warning
  • CREATE DATABASE or clone over a leftover directory now returns an error asking for removal
    Fix #11206
• 11052: Support covering secondary indexes on DOLT_DIFF() table function.
  DOLT_DIFF provides a way to efficiently compare the changes to a table at two commits. If the user only cares about changes to rows within a range of key values, DOLT_DIFF exposes an index that wraps the primary index on the underlying table.
  This PR adds additional indexes to DOLT_DIFF that similarly wrap the underlying table's secondary indexes. This allows a user to efficiently compare changes to rows within a range of secondary key values.
  This only works if the secondary index is covering, since otherwise the index can't be used to detect modified rows if the only modified columns are outside the index.

go-mysql-server

• 3619: sql: Fix ALTER USER to allow the Identity field of the User record to be updated.
  CREATE USER ... IDENTITY WITH <plugin> AS '<identity>' correctly parsed and persisted the Identity field. ALTER USER correctly parsed the Identity field but failed to pe...

Merged PRs

dolt

• 11257: go: merge: Fix two bugs in merge related to adaptive encoding.
  When the left and right side of a merge have identical columns but the columns have different encoding, the merge logic did the wrong thing.
  1. When running check expressions, it would (try to) read the tuples using the wrong encoding in some cases.
  2. When performing a structural merge, it would fail to correctly rewrite the tuples in the resulting tables in some cases.
     In most cases this resulted in failed merges. In some cases, this resulted in corrupted tables where the misencoded values could no longer be successfully read.
     Developed with red/green tests which are left as regression tests.
• 11254: go: doltcore/merge,prolly/tree: Make merge paths recover from panics encountered during merge and return errors directly, instead of crashing.
  Not recovering while in a goroutine causes the process to crash. Especially in server context, we would rather return the error.
• 11252: go/store/types: serial_message: Fix a bug which caused a panic when debug printing a schema with a CheckConstraint.
• 11248: Add a global credential helper for remotesapi
• 11234: Fix dolt_log system table on shallow clones
  On a shallow clone, the DOLT_LOG system table failed to properly show commits when ghost commits were encountered. The table now skips ghost commits so it stops at the shallow boundary.
  • Skip ghost commits in DOLT_LOG walk
  • Report the row count as inexact only on shallow clones; full clones keep the existing O(1) fast count
  • Add DoltDB.IsShallow(), backed by a new chunks.GhostChunkStore interface
    Fix #11230

go-mysql-server

• 3605: Use TableIds instead of names when mapping filters to tables
  Split off from #3591
  Using table names prevented us from being able to distinguish between tables from different databases with the same unqualified name.
  This PR also updates places where we were not using the correct TableId
  • when condensing a SubqueryAlias into a TableAlias, the new TableAlias should have the same TableId as the SubqueryAlias
  • A view should be getting its TableId from the scope via its name, not the name of its first column
• 3603: Only get projection expressions once when creating filterSet during filter pushdown
  Split off from #3591
  getProjectionExpressions is only called once, instead of for each Filter node. tableAliases is also removed because it's never actually used anywhere.
• 3602: Create single override point for SplitConjunction
SplitConjunction works differently for Doltgres expressions. The integration with Doltgres relied on analyzer.SplitConjunction and memo.SplitConjunction being replaced and called instead of expression.SplitConjunction; however, expression.SplitConjunction was still being called in many places in the analyzer. Not calling the correct version of SplitConjunction was preventing filter expressions from being properly pushed down in Doltgres. Furthermore, expression.SplitConjunction was still being called in rowexec and calling analyzer.SplitConjunction from the rowexec package causes a cyclical import.
  To avoid confusion between the various SplitConjunction instances, this PR replaces the various override variables with a single one in the expression package. This allows SplitConjunction to be consistently replaced throughout GMS without requiring GMS developers to have knowledge of SplitConjunction working differently in Doltgres.
  This change is integrated into Dolgres in dolthub/doltgresql#2865.
  This issue was originally encountered while working on #3591.
• 3601: cache GeneralizeTypes output and avoid fmt.Sprintf in HashOfSimple
  This PR adds some optimizations to CASE statements and HashLookups
• 3598: Fix JSON_LENGTH and member-access lookups to match MySQL
  JSON_LENGTH returned NULL for empty arrays and the JSON null literal, and member-access paths (.key/.*) on an array returned [] instead of NULL.
  • Empty array now has length 0, JSON null has length 1
  • Member access on a non object yields SQL NULL, at any depth
  • New allocation free path scanner for the pre check
    Fix #11224
    Close #11235
• 3594: allow using star expr after sql value for Doltgres

vitess

• 474: go/mysql: add Conn.WaitForClientActivity to detect a departed client
  On context cancelation, the function returns with a nil error. If the client unexpectedly writes to the connection or closes it, then this method will return a non-nil error.
  Unlike the first attempt at this method, this new implementation accounts for LoadDataInFile. It uses a preempt-able read lease to let the handler read from the client socket even while WaitForClientActivity is outstanding. The Peek becomes active once again after the reads are issued and completed.
• 473: Revert "Merge pull request #472 from dolthub/aaron/vitess-async-read-eof"
  This reverts commit 02705d5447c2e7062c3ae31f6909d4e5e1c97812, reversing changes made to 0893abc805429d8a31ea9fa395ff118fa95474da.

Closed Issues

• 11244: Support refreshable custom gRPC metadata for remotes
• 11230: Select from dolt_log error When clone from repository with depth 1
• 11224: Dolt JSON_LENGTH('[]') Bug

Performance

Merged PRs

dolt

• 11238: go: sqle/clusterdb: Fix spurious error logs. Interact with quiescable events database protocol.
  This also enables quiesced event reloads even when cluster replication is enabled.
• 11237: go: store/pull,remotestorage: Add optional logging to dolt pull/fetch/push.
  If you set PUSH_LOG=true when running dolt pull, dolt fetch or dolt push, some extra logging output will appear in .dolt/temptf/push-*.log.

go-mysql-server

• 3594: allow using star expr after sql value for Doltgres

Closed Issues

Merged PRs

dolt

• 11217: remove the dolt archive command
  The dolt archive command was a precursor to GC support using archives. It offered the ability to test and roll back is problems arose. Now that archives are the default and have been for awhile, this command seems unnecessary.
• 11215: fix silent data loss for large JSON values with control characters
  Large JSON values containing control characters were silently dropped on read. The stored text was re-parsed strictly, and an over-eager unescape step corrupted any non-HTML \uXXXX escape.
  • preserve non-HTML unicode escapes in unescapeHTMLCodepoints
  • guard against truncated escapes
    Fix #11210
    Depends on dolthub/go-mysql-server#3589
    Close #11218
• 11214: go,integration-tests: Enable session aware safepoint controller by default.
  Fixes a regression in the session-aware controller where fast-complete no-op GC was not working as intended.
• 11192: reject column reference args in dolt_diff with a clear error
  DOLT_DIFF requires literal commit and table arguments, but an argument referencing a column (e.g. branch.name from a correlated subquery) panics at evaluation with an unknown field index.
  Refs #11187

go-mysql-server

• 3595: Rollback event-driven stale-client detection.
• 3593: server/handler.go: Use event-driven notification to quickly cancel a running query after a client-disconnect, instead of polling socket state in a platform-dependent way.
• 3590: server: Use pollForClosedConnection on all query paths, including DML, DDL, OkResults, empty result schemas, etc.
• 3589: fix JSON control-character escaping and out-of-band text reads
  Escape every control character in JSON output and unwrap lazily loaded text so JSON functions read it correctly.
  • escape U+0000 to U+001F as \uXXXX, keeping the short forms for backspace, tab, newline, form feed, and carriage return.
  • unwrap StringWrapper before evaluating JSON functions, fixing JSON_VALID and friends on large keyless TEXT
  • add serializer, JSON_QUOTE, and function tests
    Block #11215
• 3587: go.mod: Bump github.com/lestrrat-go/strftime to v1.2.0. Fixes a lock leak which can deadlock queries in Dolt.
• 3585: Add right side lookup overhead for LookupJoins
  LookupJoins were getting costed lower than they should be, resulting in bad join strategies in some cases.
  This PR adjusts the coster and includes an overhead for the right side of a lookup join.
  An example of the bad join is:

  explain SELECT SUM(CASE WHEN s_quantity = 0 THEN 1 ELSE 0 END) AS stockouts, SUM(ol.ol_amount) AS total_cost FROM stock s JOIN order_line ol ON s.s_i_id = ol.ol_i_id WHERE  s.s_i_id <= 7500;
  plan
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------
  Project
  ├─ columns: [sum(case  when s.s_quantity = 0 then 1 else 0 end as case when s_quantity = 0 then 1 else 0 end) as stockouts, sum(ol.ol_amount) as total_cost]
  └─ GroupBy
  ├─ select: SUM(CASE  WHEN s.s_quantity = 0 THEN 1 ELSE 0 END as CASE WHEN s_quantity = 0 THEN 1 ELSE 0 END), SUM(ol.ol_amount)
  ├─ group:
  └─ LookupJoin
  ├─ TableAlias(ol)
  │   └─ Table
  │       ├─ name: order_line
  │       └─ columns: [ol_i_id ol_amount]
  └─ Filter
  ├─ s.s_i_id <= 7500
  └─ TableAlias(s)
  └─ IndexedTableAccess(stock)
  ├─ index: [stock.s_i_id]
  ├─ columns: [s_i_id s_quantity]
  └─ keys: ol.ol_i_id
  (17 rows)
  

  The more optimal plan uses a HashJoin

  explain SELECT /*+ HASH_JOIN(ol, s) */ HINT SUM(CASE WHEN s_quantity = 0 THEN 1 ELSE 0 END) AS stockouts,
  SUM(ol.ol_amount) AS total_cost
  FROM stock s
  JOIN order_line ol ON s.s_i_id = ol.ol_i_id
  WHERE s.s_i_id <= 7500;
  plan
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------
  Project
  ├─ columns: [sum(case  when s.s_quantity = 0 then 1 else 0 end as case when s_quantity = 0 then 1 else 0 end) as stockouts, sum(ol.ol_amount) as total_cost]
  └─ GroupBy
  ├─ select: SUM(CASE  WHEN s.s_quantity = 0 THEN 1 ELSE 0 END as CASE WHEN s_quantity = 0 THEN 1 ELSE 0 END), SUM(ol.ol_amount)
  ├─ group:
  └─ HashJoin
  ├─ s.s_i_id = ol.ol_i_id
  ├─ TableAlias(ol)
  │   └─ Table
  │       ├─ name: order_line
  │       └─ columns: [ol_i_id ol_amount]
  └─ HashLookup
  ├─ left-key: (ol.ol_i_id)
  ├─ right-key: (s.s_i_id)
  └─ Filter
  ├─ s.s_i_id <= 7500
  └─ TableAlias(s)
  └─ Table
  ├─ name: stock
  └─ columns: [s_i_id s_quantity]
  

  Source: I made it up.

vitess

• 473: Revert "Merge pull request #472 from dolthub/aaron/vitess-async-read-eof"
  This reverts commit 02705d5447c2e7062c3ae31f6909d4e5e1c97812, reversing changes made to 0893abc805429d8a31ea9fa395ff118fa95474da.
• 472: go/mysql: Conn: Add WaitForClientActivity, which has semantics that allow a server to quickly cancel inflight work if a client goes away.
• 471: add not valid on fkey and check constraints

Closed Issues

• 11210: 3 Dolt JSON Bugs for large JSON payloads / parallel execution of JSON payloads.
• 10854: Feature Request: git-backed dolt support for "gcrypt::" transport protocol
• 11205: Pre-2.1 ALTERed TEXT columns: mixed-row 'invalid hash length: 19' panics on 2.1.4; schema-encoding-drift check reports clean; rows not repairable via SQL
• 11209: sql-server: DOLT_CLONE from a file:// remote fails "manifest referenced table file for which there is no chunkSource" after the remote is rewritten in the same server process

Merged PRs

dolt

• 11200: fix: Improve remote interactions against file remotes from within long-lived processes.
  Dolt currently has a singleton cache which can cache file remotes across calls to things like dolt_clone and dolt_fetch. This change ensures that Dolt Rebases the store when it is interacting with the remote so that it always sees the latest novelty.
• 11198: go/utils/publishrelease/buildindocker.sh: Bump CC toolchain. gcc 16.1.0, musl 1.2.6, binutils 2.46.0, mimalloc 3.3.2.
• 11197: bug fix for blame view
  Recent change made results start including working set changes, which we don't want
• 11135: fix: make findCommonCommit deterministic for criss-cross merges
• 10976: fix(sql): use LEFT JOIN in blame view to handle schema-only reverts

go-mysql-server

• 3587: go.mod: Bump github.com/lestrrat-go/strftime to v1.2.0. Fixes a lock leak which can deadlock queries in Dolt.
• 3582: fix: panic in regexp functions over large text in group by
  REGEXP_REPLACE on a large TEXT column inside a GROUP BY panic with SIGSEGV.
  • Add a ScriptTest and RegexScriptTests for similar regressions on REGEXP_REPLACE, REGEXP_SUBSTR, REGEXP_LIKE, and REGEXP_INSTR.
    Depends on dolthub/go-icu-regex#13
• 3581: Fix LIKE 'prefix%' dropping rows with multibyte text
  LIKE 'prefix%' is rewritten into an index range for speed, but the upper bound used byte 255 (U+00FF), which is not the largest character. Any value after prefix above U+00FF falls outside the range and was silently dropped as a WHERE/JOIN filter. Instead, we build the bound per collation:
  • utf8mb4_0900_bin matches are an exact code-point range, so emit GTE prefix AND LT next (prefix and last character incremented by one skipping the surrogate code points).
  • Any other collation GTE prefix AND <LIKE> so the index still narrows the scan while the retained LIKE keeps results correct. These collations may not sort by raw character numbers, or have unique rules (i.e., treating trailing spaces as nothing, so "ab" and "ab " count as equal).
  • Fix TPCH and IMDB plan expectations with old baked-in bound.
    Fix #11182
    Close #11188
• 3572: Rewrite CachedResults node and cachedResultsIter to not use MemoryManager
  This PR is a follow-up to #3561 and further addresses the memory leak mentioned in #3560. While the initial fix in #3561 made it seem like the issue was that we were not properly disposing of the CachedResults node, the real issue was that a cachedResultsIter that never wrote to the CachedResults node was never disposed of in the MemoryManager if it was never closed.
  This PR rewrites the CachedResults node and cachedResultsIter to never actually make use of the MemoryManager. Since we do not make use of parallel partitions, there's no need to have a locking cache. Instead, the results of the cachedResultsIter are stored directly in the CachedResults node. This greatly simplifies both the CachedResults node and the cachedResultsIter.
  This PR also moves the emptyCacheIter and EmptyIter out of the plan package and de-dupes any code related to empty iters.

Closed Issues

• 11131: Dolt 2.0.8: adaptive out-of-line TEXT written via migration is unreadable — "invalid hash length: 19" on read (regression vs 2.0.4)
• 11037: Allow passing --date to DOLT_MERGE (as is the case for DOLT_COMMIT)
• 11196: GC kill-connections safepoint leaks connection slots: dss_concurrent_connections counts sessions whose client socket is already gone (pegs at max_connections)
• 7996: Dolt does not support CREATE TABLE ... AS ... with a table definition
• 3154: Undefined behavior for dolt checkout <remote branch> when two remote branches have the same name
• 10802: bug: DOLT_CHECKOUT -b ignores --no-overwrite-ignore when called directly via SQL
• 11042: dolt_merge_base is non-deterministic on multi-LCA criss-cross merges
• 11183: REGEXP_REPLACE over a TEXT column in a parallel GROUP BY grouping key SIGSEGVs in go-icu-regex (ICU not thread-safe)
• 11180: 3-way DOLT_MERGE fails on table with VECTOR INDEX: expected prollyIndex, found: durable.proximityIndex
• 11182: LIKE 'prefix %' gives wrong results in WHERE filter vs projection when matching rows have multibyte chars after the prefix

Merged PRs

dolt

• 11179: Set version string in initial server response
  MySQL servers return a version string in the initial connection response. Some clients read this and expect certain versions of MySQL or MariaDB. We return 8.0.31 by default, since that corresponds with the MySQL version we advertise feature parity with.
  Currently, we use the @@version system variable as the source of truth for the version string, and this can be configued via a server config file. But we extract the config for the listener before we initialize system variables. Previously this meant that the version string on connection defaulted to the value specified in our vitess dependency, which was hardcoded to 8.0.31.
  This PR changes the behavior to extract a string from the config if it exists.
  We need to use a client integration test for this, since the Go SQL driver ignores the version in the connection info and does not provide a way to view it.
• 11153: Bug fix: Doltgres JSONB use with dolt_patch()
  Fixes dolthub/doltgresql#2732

Closed Issues

• 11166: Expose go-mysql-server's server.Config.Version in dolt sql-server (advertise a MariaDB version for client compatibility)
• 11180: 3-way DOLT_MERGE fails on table with VECTOR INDEX: expected prollyIndex, found: durable.proximityIndex

Performance