What's New in v24.2

Docker image

docker pull cockroachdb/cockroach-unstable:v24.2.0-alpha.1

Source tag

To view or download the source code for CockroachDB v24.2.0-alpha.1 on Github, visit v24.2.0-alpha.1 source tag.

General changes

• The following metrics were added for observability into the source of disk writes:
  • storage.category-pebble-wal.bytes-written
  • storage.category-pebble-compaction.bytes-written
  • storage.category-pebble-ingestion.bytes-written
  • storage.category-pebble-memtable-flush.bytes-written
  • storage.category-raft-snapshot.bytes-written
  • storage.category-encryption-registry.bytes-written
  • storage.category-crdb-log.bytes-written
  • storage.category-sql-row-spill.bytes-written
  • storage.category-sql-col-spill.bytes-written
  • storage.category-unspecified.bytes-written #119024
• The storage.value_blocks.enabled cluster setting has been removed; value blocks are always enabled. #122164
• The following metrics were added for improved observability into disk bandwidth:
  • storage.disk.read-max.bytespersecond
  • storage.disk.write-max.bytespersecond #121553
• Added two new cluster settings, storage.sstable.compression_algorithm_backup_storage and storage.sstable.compression_algorithm_backup_transport, which in addition to the existing cluster setting storage.sstable.compression_algorithm, can be used to alter the compression algorithm used for various types of SSTs. #124245

Enterprise edition changes

• SHOW CHANGEFEED JOB, SHOW CHANGEFEED JOBS, and SHOW JOBS no longer expose user sensitive information like client_key. #122529
• Added the new option encode_json_value_null_as_object to JSON-formatted changefeeds that outputs 'null'::jsonb as {"__crdb_json_null__": true} instead of null, to disambiguate between SQL-null and JSON-null. With this option enabled, if the literal value {"__crdb_json_null__": true} is present in a JSON value, it will have the same representation as JSON-null with this option enabled. If such a value is encountered in a changefeed, a (rate-limited) warning will be printed to the DEV channel. #122848
• Added an error message for changefeed options and parameters that are not supported by the Apache Pulsar sink. #124581
• Scheduled changefeeds now pause after being restored onto a different cluster, and after completion of physical cluster replication to avoid inadvertent concurrent execution of the same schedule on multiple clusters. #124631
• The DEBUG_PAUSE_ON option has been removed and replaced with the restore.restore_after_failure pause point to match other pause points used throughout CockroachDB. You can set this pause point by running: SET CLUSTER SETTING jobs.debug.pausepoints = 'restore.after_restore_failure'. #125158

SQL language changes

• Default schema privilege changes will now be reflected and can be monitored in the pg_default_acl table. #121506
• The schema of the pg_catalog.pg_proc virtual table now matches exactly that of PostgreSQL versions 14–16. The following changes are applied:
  • proisagg and proiswindow columns are removed (done in v11).
  • protransform column is removed (done in v12).
  • prosqlbody column is added (done in v14).
  • prosupport and prokind columns are moved into their correct spot (these were incorrectly present at the end of the columns list). #121966
• Added SHOW EXTERNAL CONNECTIONS and SHOW EXTERNAL CONNECTION <connection name>. These queries display redacted connection URIs and other useful information, such as the connection type. Access to these queries is restricted to the owner of the connection or users with USAGE privilege. admin or root users will have unrestricted access to all connections. #123545
• Using the CREATE STATISTICS query without the AS OF SYSTEM TIME option could contend with concurrent transactions and cost performance. Running CREATE STATISTICS without specifying AS OF SYSTEM TIME now uses a default of -1us. #124488
• The nodes field of the EXPLAIN ANALYZE output has been renamed to sql nodes to clarify that this field describes SQL processing and it does not include any information about KV nodes that might have participated in the query execution. #124681
• EXPLAIN ANALYZE output now has a new field KV nodes that includes all KV nodes that were used to serve read requests by a particular SQL operator. #124681
• Fixed the Regions field in the EXPLAIN ANALYZE output to include regions of KV nodes. Previously, only regions of SQL nodes involved in query processing were included. #124888
• Allow foreign keys to be created over stored computed columns. However, most ON UPDATE and ON DELETE options for foreign key constraints are not allowed with computed columns. Only ON UPDATE (NO ACTION|RESTRICT) and ON DELETE (NO ACTION|RESTRICT|CASCADE) are supported. #124851
• EXPLAIN ANALYZE output now has a new field used follower read to SQL operators whenever their reads were served by the follower replicas. Previously, this information was only available in the trace. #125152
• The new attribute historical: AS OF SYSTEM TIME ... is now included in EXPLAIN ANALYZE output whenever the query performs historical reads. #125152
• EXPLAIN ANALYZE statements are now supported when executed via UI SQL shell. #125455
• Histograms are no longer constructed using a default sample size of 10k. Samples are now dynamically sized based on table size unless the sample count has been set in the table or cluster settings. #125345
• The optimizer will now generate plans utilizing partial indexes with IS NOT NULL predicates in more cases. #125487
• The SHOW TYPES statement now includes user-defined composite types. It omitted those types ever since composite types were added in v23.1. #124730
• Added the COMMENT ON TYPE statement for implicit transactions. #124942

Operational changes

• Added a distinction between row updates (row) and resolved timestamp (resolved) messages in some changefeed metrics. #122645
• Modified the default Grafana dashboards to include a breakdown by message type for the changefeed_emitted_rows metric. #122802
• Updated the generated doc descriptions for system visible cluster settings to reflect these are read-only for CockroachDB Serverless and read-write for the other deployments. #124028
• Added the cluster setting debug.zip.redact_addresses.enabled that allows the user to enable or disable redaction of fields like hostname and IP addresses. #123544
• Improved disk usage metric reporting over volumes that dynamically change their size over the life of the cockroach process. #124999
• crdb_internal.cluster_execution_insights.txt and crdb_internal.cluster_txn_execution_insights.txt have been removed from the debug zip. These files contained cluster-wide insights for statements and transactions. Users can still rely on the per-node execution insights in crdb_internal.node_execution_insights.txt and crdb_internal.node_txn_execution_insights.txt. #125654
• Removed the stuck rangefeed cancel feature and its related cluster setting kv.rangefeed.range_stuck_threshold, because it was only available in non-mux rangefeeds. Previously, the stuck rangefeed cancel feature was introduced to restart single rangefeeds automatically if they had not received KV updates for some time. #125663
• Fixed a bug where collection of debug information for very long-running jobs could use excessive space in the job_info system table and/or cause some interactions with the jobs system to become slow. #126084
• All system tables are now visible in application virtual clusters that were created before v24.1. #125979

DB Console changes

• The Goroutine Scheduling Latency graph has been added to the Runtime dasboard. #121369
• Updated the time format to use a . (dot) as separation between seconds and milliseconds, which affects many pages in the DB Console. #121456
• The Storage dashboard now contains a graph categorizing disk writes that contribute to sys.host.disk.write.bytes according to the source of the write (WAL, compactions, SSTable ingestion, memtable flushes, raft snapshots, encryption registry, logs, SQL columnar spill, or SQL row spill). #119024
• The favicon now renders properly for the DB Console along with other image files. #121380
• The color of Unavailable Ranges in the Summary panel of the Replication dashboard is now red when nonzero. #123120
• Removed the $ sign on the Databases and Jobs pages in the DB Console. #125370
• Added two graphs to the Storage dashboard that display count and size of L0 SSTables in Pebble. This provides increased visibility into L0 compaction issues. #125653
• Removed the p95 metrics from the tooltip on the Job Latency graph of the TTL dashboard, because there are no p95 values computed for any of the metrics. #122345
• Updated the Storage dashboard graphs to show most metrics on a per-store basis when viewing a single node's metrics. This provides increased visibility into issues caused by specific stores on each node. #125699

Bug fixes

• Fixed a crash introduced in v23.2.5 and v24.1.0-beta.2 that could occur when planning statistics collection on a table with a virtual computed column using a user-defined type when the newly introduced cluster setting sql.stats.virtual_computed_columns.enabled is set to true. This setting was introduced in v23.2.4 and v24.1.0-alpha.1. #123926
• Fixed handling in the declarative schema changer when columns are included in the STORING() clause of CREATE INDEX. It now checks if the column is virtual beforehand, and properly detects when a column is already handled by an existing index when the column name has UTF-8 characters. #124981
• Fixed an issue where adding a column with a default value of an empty array would not succeed. #125284
• ALTER TABLE ... ADD CONSTRAINT UNIQUE will now fail with a well-formed error message and code 42601 if a statement tries to add a UNIQUE constraint on an expression. #125282
• Resolved a log message that read: expiration of liveness record ... is not greater than expiration of the previous lease ... after liveness heartbeat. This message is no longer possible. #124885
• Fixed a bug in v24.1, v23.2, and v23.1 where using the changefeed.aggregator.flush_jitter cluster setting with min_checkpoint_frequency set to 0 could cause panics. #125317
• Fixed a bug where the public schema would be created with the wrong owner. Previously, the admin role would own the public schema. Now the database owner is also the owner of the public schema. The ownership can be altered after the schema is created. #124894
• Previously, CockroachDB would hit an internal error when evaluating inserts into REGIONAL BY ROW tables where the source is a VALUES clause with a single row and at least one boolean expression. The bug was introduced in v23.2.0 and is now fixed. #125492
• Fixed a bug in logging where an error code was misreported for canceled queries. This bug affected the SQL_PERF (slow query logs) and SQL_EXEC (sql exec logs) logging channels. #124371
• Fixed a bug in which constant LIKE patterns containing certain sequences of backslashes did not become constrained scans. This bug has been present since v21.1.13 when support for building constrained scans from LIKE patterns containing backslashes was added. #125507
• Fixed a bug introduced in v22.1 where cockroach-sql does not recognize the --format flag. #124735
• Fixed a bug where CREATE TABLE with index expressions could hit undefined column errors on transaction retries. #125910
• Fixed a bug where some DDL and administrative statements used within a common table expression would fail with an unrecognized relational expression type internal error. #125959
• Fixed a bug in cockroach debug tsdump where the command fails when a custom SQL port is used and the --format=raw flag is provided. #125626
• Attempts to alter the data type of a column used in a computed column expression are now blocked. #125870
• Fixed the statistics estimation code in the optimizer so it does not use the empty histograms produced if histogram collection has been disabled during stats collection due to excessive memory utilization. Now the optimizer will rely on distinct counts instead of the empty histograms and should produce better plans as a result. This bug has existed since CockroachDB v22.1. #125968

Performance improvements

• More efficient query plans are now generated for queries with text similarity filters, e.g., text_col % 'foobar'. These plans are generated if the optimizer_use_trigram_similarity_optimization session setting is enabled, which it is by default. This setting is disabled by default in previous versions. #121973
• Schema changes that cause a data backfill, such as adding a non-nullable column or changing the primary key, will now split and scatter the temporary indexes used to perform the change. This reduces the chance of causing a write hotspot that can slow down foreground traffic. #120736
• Multiple or large numbers of grants on tables and types within one transaction now run faster. #123414
• Lock operations are now removed from query plans when the optimizer can prove that no rows would be locked. #124830
• Some privilege checks when scanning the crdb_internal.system_jobs internal table now happen once before the scan, instead of once for each row. This will improve performance for queries that read from crdb_internal.system_jobs. #117438
• Improved the initial range descriptor scan on startup. Node startup should now be faster on larger stores. #123959
• Previously, concurrent transactions that constructed large write batches could cause hotspots. This was because the transaction record for all ranges would coalesce on a single range, which would then cause this range's leaseholder to perform all intent resolution work. This is fixed by distributing transaction records randomly across the ranges the write batch touches. In turn, hotspots are prevented. #125744

Contributors

This release includes 1234 merged PRs by 97 authors.