On this page 
The cluster setting server.child_metrics.enabled enables the exporting of child metrics, which are additional Prometheus time series with extra labels.
The metrics and their potential child metrics are determined by the specific feature the cluster is using. The number of child metrics can significantly increase based on their associated labels, which increases cardinality. This page will help you understand the potential size of the Prometheus scrape payload for your workload when child metrics are enabled.
Enable child metrics
server.child_metrics.enabled is disabled by default. To enable it, use the SET CLUSTER SETTING statement:
SET CLUSTER SETTING server.child_metrics.enabled = true;
All clusters
An RPC (Remote Procedure Call) connection is a communication method used in distributed systems, like CockroachDB, to allow one program to request a service from a program located in another computer on a network without having to understand the network's details. In the context of CockroachDB, RPC connections are used for inter-node communication. For instance, if Node 1 sends a request to Node 2, and Node 2 dials back (sends request back to Node 1), it ensures that communication is healthy in both directions. This is referred to as a "bidirectionally connected" and "heartbeating" RPC connection.
When child metrics is enabled, for all clusters the rpc.connection.* metrics are exported per-peer with labels for remote_node_id, remote_addr, and class. The class label may have the following values: system, default, and raft. The cardinality increases with the number of nodes. An aggregated metric is also included.
For example:
# HELP rpc_connection_healthy Gauge of current connections in a healthy state (i.e. bidirectionally connected and heartbeating)
# TYPE rpc_connection_healthy gauge
rpc_connection_healthy{node_id="1"} 10
rpc_connection_healthy{node_id="1",remote_node_id="0",remote_addr="127.0.0.1:29004",class="system"} 1
rpc_connection_healthy{node_id="1",remote_node_id="0",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="system"} 0
rpc_connection_healthy{node_id="1",remote_node_id="0",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="system"} 1
rpc_connection_healthy{node_id="1",remote_node_id="1",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29004",class="default"} 1
rpc_connection_healthy{node_id="1",remote_node_id="1",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29004",class="system"} 1
rpc_connection_healthy{node_id="1",remote_node_id="2",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="default"} 1
rpc_connection_healthy{node_id="1",remote_node_id="2",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="raft"} 1
rpc_connection_healthy{node_id="1",remote_node_id="2",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29002",class="system"} 1
rpc_connection_healthy{node_id="1",remote_node_id="3",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="default"} 1
rpc_connection_healthy{node_id="1",remote_node_id="3",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="raft"} 1
rpc_connection_healthy{node_id="1",remote_node_id="3",remote_addr="crlMBP-X3HQX3YY6JMTMz.local:29000",class="system"} 1
Following is a list of the metrics that have child metrics:
| CockroachDB Metric Name | Description When Aggregated | Type | Unit |
rpc.connection.avg_round_trip_latency |
Sum of exponentially weighted moving average of round-trip latencies, as measured through a gRPC RPC. Dividing this Gauge by rpc.connection.healthy gives an approximation of average latency, but the top-level round-trip-latency histogram is more useful. Instead, users should consult the label families of this metric if they are available (which requires prometheus and the cluster setting 'server.child_metrics.enabled'); these provide per-peer moving averages. This metric does not track failed connection. A failed connection's contribution is reset to zero. | GAUGE | NANOSECONDS |
rpc.connection.failures |
Counter of failed connections. This includes both the event in which a healthy connection terminates as well as unsuccessful reconnection attempts. Connections that are terminated as part of local node shutdown are excluded. Decommissioned peers are excluded. | COUNTER | COUNT |
rpc.connection.healthy |
Gauge of current connections in a healthy state (i.e. bidirectionally connected and heartbeating) | GAUGE | COUNT |
rpc.connection.healthy_nanos |
Gauge of nanoseconds of healthy connection time On the prometheus endpoint scraped with the cluster setting 'server.child_metrics.enabled' set, the constituent parts of this metric are available on a per-peer basis and one can read off for how long a given peer has been connected | GAUGE | NANOSECONDS |
rpc.connection.heartbeats |
Counter of successful heartbeats. | COUNTER | COUNT |
rpc.connection.inactive |
Gauge of current connections in an inactive state and pending deletion; these are not healthy but are not tracked as unhealthy either because there is reason to believe that the connection is no longer relevant,for example if the node has since been seen under a new address | GAUGE | COUNT |
rpc.connection.unhealthy |
Gauge of current connections in an unhealthy state (not bidirectionally connected or heartbeating) | GAUGE | COUNT |
rpc.connection.unhealthy_nanos |
Gauge of nanoseconds of unhealthy connection time. On the prometheus endpoint scraped with the cluster setting 'server.child_metrics.enabled' set, the constituent parts of this metric are available on a per-peer basis and one can read off for how long a given peer has been unreachable | GAUGE | NANOSECONDS |
Secure clusters
When child metrics is enabled, for secure clusters the security.certificate.expiration.client is exported per SQL user with a label for sql_user. The sql_user label may have the values of the cluster's users who are logged into a node using client security certificates. The cardinality increases with the number of SQL users. An aggregated metric is also included, however since this is a sum of the child metric values which represent timestamps, it is not usable.
For example:
# HELP security_certificate_expiration_client Minimum expiration for client certificates, labeled by SQL user. 0 means no certificate or error.
# TYPE security_certificate_expiration_client gauge
security_certificate_expiration_client{node_id="1"} 3.756110141e+09
security_certificate_expiration_client{node_id="1",sql_user="florence"} 1.878055127e+09
security_certificate_expiration_client{node_id="1",sql_user="root"} 1.878055014e+09
Following is a list of the metrics that have child metrics:
| CockroachDB Metric Name | Description When Aggregated | Type | Unit |
security.certificate.expiration.client |
Minimum expiration for client certificates, labeled by SQL user. 0 means no certificate or error. | GAUGE | SECONDS |
Virtual clusters
When child metrics is enabled, for virtual clusters the kv.tenant_rate_limit.* metrics and other kv-related metrics are exported per virtual cluster with a label for tenant_id. The tenant_id label may have the values: system or the id of the virtual cluster. The cardinality increases with the number of virtual clusters. An aggregated metric is also included.
Note:
With virtual clusters, while the tenant_id label on kv metrics is only exported when child metrics is enabled, the tenant label on SQL metrics is exported whether child metrics is enabled or disabled.
For example:
# HELP kv_tenant_rate_limit_read_bytes_admitted Number of read bytes admitted by the rate limiter
# TYPE kv_tenant_rate_limit_read_bytes_admitted counter
kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1"} 41
kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1",tenant_id="3"} 41
kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1",tenant_id="4"} 0
kv_tenant_rate_limit_read_bytes_admitted{store="1",node_id="1",tenant_id="system"} 0
# HELP sysbytes Number of bytes in system KV pairs
# TYPE sysbytes gauge
sysbytes{store="1",node_id="1"} 41979
sysbytes{store="1",node_id="1",tenant_id="3"} 278
sysbytes{store="1",node_id="1",tenant_id="4"} 225
sysbytes{store="1",node_id="1",tenant_id="system"} 41476
Following is a list of the metrics that have child metrics:
| CockroachDB Metric Name | Description When Aggregated | Type | Unit |
abortspanbytes |
Number of bytes in the abort span | GAUGE | BYTES |
gcbytesage |
Cumulative age of non-live data | GAUGE | SECONDS |
intentage |
Cumulative age of locks | GAUGE | SECONDS |
intentbytes |
Number of bytes in intent KV pairs | GAUGE | BYTES |
intentcount |
Count of intent keys | GAUGE | COUNT |
keybytes |
Number of bytes taken up by keys | GAUGE | BYTES |
keycount |
Count of all keys | GAUGE | COUNT |
kv.tenant_rate_limit.current_blocked |
Number of requests currently blocked by the rate limiter | GAUGE | COUNT |
kv.tenant_rate_limit.num_tenants |
Number of tenants currently being tracked | GAUGE | COUNT |
kv.tenant_rate_limit.read_batches_admitted |
Number of read batches admitted by the rate limiter | COUNTER | COUNT |
kv.tenant_rate_limit.read_bytes_admitted |
Number of read bytes admitted by the rate limiter | COUNTER | BYTES |
kv.tenant_rate_limit.read_requests_admitted |
Number of read requests admitted by the rate limiter | COUNTER | COUNT |
kv.tenant_rate_limit.write_batches_admitted |
Number of write batches admitted by the rate limiter | COUNTER | COUNT |
kv.tenant_rate_limit.write_bytes_admitted |
Number of write bytes admitted by the rate limiter | COUNTER | BYTES |
kv.tenant_rate_limit.write_requests_admitted |
Number of write requests admitted by the rate limiter | COUNTER | COUNT |
livebytes |
Number of bytes of live data (keys plus values) | GAUGE | BYTES |
livecount |
Count of live keys | GAUGE | COUNT |
lockbytes |
Number of bytes taken up by replicated lock key-values (shared and exclusive strength, not intent strength) | GAUGE | BYTES |
lockcount |
Count of replicated locks (shared, exclusive, and intent strength) | GAUGE | COUNT |
rangekeybytes |
Number of bytes taken up by range keys (e.g. MVCC range tombstones) | GAUGE | BYTES |
rangekeycount |
Count of all range keys (e.g. MVCC range tombstones) | GAUGE | COUNT |
rangevalbytes |
Number of bytes taken up by range key values (e.g. MVCC range tombstones) | GAUGE | BYTES |
rangevalcount |
Count of all range key values (e.g. MVCC range tombstones) | GAUGE | COUNT |
sysbytes |
Number of bytes in system KV pairs | GAUGE | BYTES |
syscount |
Count of system KV pairs | GAUGE | COUNT |
totalbytes |
Total number of bytes taken up by keys and values including non-live data | GAUGE | BYTES |
valbytes |
Number of bytes taken up by values | GAUGE | BYTES |
valcount |
Count of all values | GAUGE | COUNT |
Clusters with changefeeds
When child metrics is enabled and changefeeds with metrics labels are created on the cluster, the changefeed.* metrics are exported per changefeed metric label with a label for scope. The scope label may have the values set using the metrics_label option. The cardinality increases with the number of changefeed metric labels. An aggregated metric is also included.
For example, when you create two changefeeds with the metrics labels employees and office_dogs, the counter metric changefeed_error_retries exports child metrics with a scope for employees and office_dogs. In addition, the default scope will also be exported which includes changefeeds started without a metrics label.
HELP changefeed_error_retries Total retryable errors encountered by all changefeeds
# TYPE changefeed_error_retries counter
changefeed_error_retries{node_id="1"} 0
changefeed_error_retries{node_id="1",scope="default"} 0
changefeed_error_retries{node_id="1",scope="employees"} 0
changefeed_error_retries{node_id="1",scope="office_dogs"} 0
Following is a list of the metrics that have child metrics:
| CockroachDB Metric Name | Description When Aggregated | Type | Unit |
changefeed.admit_latency |
Event admission latency: a difference between event MVCC timestamp and the time it was admitted into changefeed pipeline; Note: this metric includes the time spent waiting until event can be processed due to backpressure or time spent resolving schema descriptors. Also note, this metric excludes latency during backfill | HISTOGRAM | NANOSECONDS |
changefeed.aggregator_progress |
The earliest timestamp up to which any aggregator is guaranteed to have emitted all values for | GAUGE | TIMESTAMP_NS |
changefeed.backfill_count |
Number of changefeeds currently executing backfill | GAUGE | COUNT |
changefeed.backfill_pending_ranges |
Number of ranges in an ongoing backfill that are yet to be fully emitted | GAUGE | COUNT |
changefeed.batch_reduction_count |
Number of times a changefeed aggregator node attempted to reduce the size of message batches it emitted to the sink | GAUGE | COUNT |
changefeed.checkpoint_progress |
The earliest timestamp of any changefeed's persisted checkpoint (values prior to this timestamp will never need to be re-emitted) | GAUGE | TIMESTAMP_NS |
changefeed.cloudstorage_buffered_bytes |
The number of bytes buffered in cloudstorage sink files which have not been emitted yet | GAUGE | COUNT |
changefeed.commit_latency |
Event commit latency: a difference between event MVCC timestamp and the time it was acknowledged by the downstream sink. If the sink batches events, then the difference between the oldest event in the batch and acknowledgement is recorded; Excludes latency during backfill | HISTOGRAM | NANOSECONDS |
changefeed.emitted_batch_sizes |
Size of batches emitted emitted by all feeds | HISTOGRAM | COUNT |
changefeed.emitted_bytes |
Bytes emitted by all feeds | COUNTER | BYTES |
changefeed.emitted_messages |
Messages emitted by all feeds | COUNTER | COUNT |
changefeed.error_retries |
Total retryable errors encountered by all changefeeds | COUNTER | COUNT |
changefeed.filtered_messages |
Messages filtered out by all feeds. This count does not include the number of messages that may be filtered due to the range constraints. | COUNTER | COUNT |
changefeed.flush_hist_nanos |
Time spent flushing messages across all changefeeds | HISTOGRAM | NANOSECONDS |
changefeed.flushed_bytes |
Bytes emitted by all feeds; maybe different from changefeed.emitted_bytes when compression is enabled | COUNTER | BYTES |
changefeed.flushes |
Total flushes across all feeds | COUNTER | COUNT |
changefeed.internal_retry_message_count |
Number of messages for which an attempt to retry them within an aggregator node was made | GAUGE | COUNT |
changefeed.kafka_throttling_hist_nanos |
Time spent in throttling due to exceeding kafka quota | HISTOGRAM | NANOSECONDS |
changefeed.lagging_ranges |
The number of ranges considered to be lagging behind | GAUGE | COUNT |
changefeed.message_size_hist |
Message size histogram | HISTOGRAM | BYTES |
changefeed.parallel_io_in_flight_keys |
The number of keys currently in-flight which may contend with batches pending to be emitted | GAUGE | COUNT |
changefeed.parallel_io_pending_rows |
Number of rows which are blocked from being sent due to conflicting in-flight keys | GAUGE | COUNT |
changefeed.parallel_io_queue_nanos |
Time that outgoing requests to the sink spend waiting in a queue due to in-flight requests with conflicting keys | HISTOGRAM | NANOSECONDS |
changefeed.parallel_io_result_queue_nanos |
Time that incoming results from the sink spend waiting in parallel io emitter before they are acknowledged by the changefeed | HISTOGRAM | NANOSECONDS |
changefeed.running |
Number of currently running changefeeds, including sinkless | GAUGE | COUNT |
changefeed.schema_registry.registrations |
Number of registration attempts with the schema registry | COUNTER | COUNT |
changefeed.schema_registry.retry_count |
Number of retries encountered when sending requests to the schema registry | COUNTER | COUNT |
changefeed.sink_batch_hist_nanos |
Time spent batched in the sink buffer before being flushed and acknowledged | HISTOGRAM | NANOSECONDS |
changefeed.sink_io_inflight |
The number of keys currently inflight as IO requests being sent to the sink | GAUGE | COUNT |
changefeed.size_based_flushes |
Total size based flushes across all feeds | COUNTER | COUNT |
Clusters with row-level TTL jobs
When child metrics is enabled and row-level TTL jobs are created on the cluster with the ttl_label_metrics storage parameter enabled, the jobs.row_level_ttl.* metrics are exported per TTL job with ttl_label_metrics enabled with a label for relation. The value of the relation label may have the format: {database}_{schema}_{table}_{primary key}. The cardinality increases with the number of TTL jobs with ttl_label_metrics enabled. An aggregated metric is also included.
For example:
# HELP jobs_row_level_ttl_total_rows Approximate number of rows on the TTL table.
# TYPE jobs_row_level_ttl_total_rows gauge
jobs_row_level_ttl_total_rows{node_id="1"} 0
jobs_row_level_ttl_total_rows{node_id="1",relation="default"} 0
jobs_row_level_ttl_total_rows{node_id="1",relation="defaultdb_public_events__events_pkey_"} 0
jobs_row_level_ttl_total_rows{node_id="1",relation="defaultdb_public_events_using_date__events_using_date_pkey_"} 0
Following is a list of the metrics that have child metrics:
| CockroachDB Metric Name | Description When Aggregated | Type | Unit |
jobs.row_level_ttl.delete_duration |
Duration for delete requests during row level TTL. | HISTOGRAM | NANOSECONDS |
jobs.row_level_ttl.num_active_spans |
Number of active spans the TTL job is deleting from. | GAUGE | COUNT |
jobs.row_level_ttl.rows_deleted |
Number of rows deleted by the row level TTL job. | COUNTER | COUNT |
jobs.row_level_ttl.rows_selected |
Number of rows selected for deletion by the row level TTL job. | COUNTER | COUNT |
jobs.row_level_ttl.select_duration |
Duration for select requests during row level TTL. | HISTOGRAM | NANOSECONDS |
jobs.row_level_ttl.span_total_duration |
Duration for processing a span during row level TTL. | HISTOGRAM | NANOSECONDS |
jobs.row_level_ttl.total_expired_rows |
Approximate number of rows that have expired the TTL on the TTL table. | GAUGE | COUNT |
jobs.row_level_ttl.total_rows |
Approximate number of rows on the TTL table. | GAUGE | COUNT |
Metrics of type histogram
When child metrics is enabled with changefeeds or row-level TTL jobs, be aware that metrics of type HISTOGRAM will increase cardinality quickly.
For example, when you create two changefeeds with the metrics labels employees and office_dogs, the histogram metric changefeed_flush_hist_nanos exports child metrics for each bucket for each metrics label. In addition, the default scope will also be exported which includes changefeeds started without a metrics label. Therefore, in this example, changefeed_flush_hist_nanos exports child metrics for each bucket for the scope values: default, employees and office_dogs:
# HELP changefeed_flush_hist_nanos Time spent flushing messages across all changefeeds
# TYPE changefeed_flush_hist_nanos histogram
changefeed_flush_hist_nanos_bucket{node_id="1",le="5e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="5.572592853587432e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="6.210758222370743e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="6.922005377068515e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="7.714703539349157e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="8.598180362184556e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="9.582831688033088e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.0680243876372875e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.1903330140009282e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.326648249442152e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.4785741108131227e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.6478983046833196e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.8366132632234223e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.046939589088547e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.2813521851760063e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.5426093767255764e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.833785368441068e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.1583064185550647e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.5199911554958534e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.9230955115614314e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="4.372362802333632e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="4.873079541115184e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="5.431137645156319e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="6.053103765649553e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="6.746296557296375e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="7.518872796674253e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="8.37992336275598e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="9.339580208980864e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.0409135585614676e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.1601174915283792e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.2929724885225649e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.4410418498852003e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.6060679028781363e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.789992503590971e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.9949798866972237e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.2234421319319225e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.478067546953807e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.7618523005723442e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.0781356785666904e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.4306393769506477e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.823511295046164e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="4.261374343677016e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="4.749380842807073e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="5.2932731487183495e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="5.899451224126824e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="6.575047946331352e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="7.328013039544164e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="8.167206619031862e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="9.102503447797786e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.0144909132590578e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.1306689626513626e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.2601515562088194e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.4044623113132697e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.5652993278314426e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.7445551695997415e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="1.9443391341601053e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.167002072794196e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.415164052912417e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="2.691745188300199e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="3.0000000000000085e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",le="+Inf"} 2
changefeed_flush_hist_nanos_sum{node_id="1"} 9.79696709e+08
changefeed_flush_hist_nanos_count{node_id="1"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.572592853587432e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.210758222370743e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.922005377068515e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="7.714703539349157e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="8.598180362184556e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="9.582831688033088e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.0680243876372875e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.1903330140009282e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.326648249442152e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.4785741108131227e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.6478983046833196e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.8366132632234223e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.046939589088547e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.2813521851760063e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.5426093767255764e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.833785368441068e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.1583064185550647e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.5199911554958534e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.9230955115614314e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.372362802333632e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.873079541115184e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.431137645156319e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.053103765649553e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.746296557296375e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="7.518872796674253e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="8.37992336275598e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="9.339580208980864e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.0409135585614676e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.1601174915283792e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.2929724885225649e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.4410418498852003e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.6060679028781363e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.789992503590971e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.9949798866972237e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.2234421319319225e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.478067546953807e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.7618523005723442e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.0781356785666904e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.4306393769506477e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.823511295046164e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.261374343677016e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="4.749380842807073e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.2932731487183495e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="5.899451224126824e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="6.575047946331352e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="7.328013039544164e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="8.167206619031862e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="9.102503447797786e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.0144909132590578e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.1306689626513626e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.2601515562088194e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.4044623113132697e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.5652993278314426e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.7445551695997415e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="1.9443391341601053e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.167002072794196e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.415164052912417e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="2.691745188300199e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="3.0000000000000085e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="default",le="+Inf"} 0
changefeed_flush_hist_nanos_sum{node_id="1",scope="default"} 0
changefeed_flush_hist_nanos_count{node_id="1",scope="default"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.572592853587432e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.210758222370743e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.922005377068515e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="7.714703539349157e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="8.598180362184556e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="9.582831688033088e+08"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.0680243876372875e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.1903330140009282e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.326648249442152e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.4785741108131227e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.6478983046833196e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.8366132632234223e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.046939589088547e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.2813521851760063e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.5426093767255764e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.833785368441068e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.1583064185550647e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.5199911554958534e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.9230955115614314e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.372362802333632e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.873079541115184e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.431137645156319e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.053103765649553e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.746296557296375e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="7.518872796674253e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="8.37992336275598e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="9.339580208980864e+09"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.0409135585614676e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.1601174915283792e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.2929724885225649e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.4410418498852003e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.6060679028781363e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.789992503590971e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.9949798866972237e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.2234421319319225e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.478067546953807e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.7618523005723442e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.0781356785666904e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.4306393769506477e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.823511295046164e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.261374343677016e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="4.749380842807073e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.2932731487183495e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="5.899451224126824e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="6.575047946331352e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="7.328013039544164e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="8.167206619031862e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="9.102503447797786e+10"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.0144909132590578e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.1306689626513626e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.2601515562088194e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.4044623113132697e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.5652993278314426e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.7445551695997415e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="1.9443391341601053e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.167002072794196e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.415164052912417e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="2.691745188300199e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="3.0000000000000085e+11"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="employees",le="+Inf"} 0
changefeed_flush_hist_nanos_sum{node_id="1",scope="employees"} 0
changefeed_flush_hist_nanos_count{node_id="1",scope="employees"} 0
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.572592853587432e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.210758222370743e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.922005377068515e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="7.714703539349157e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="8.598180362184556e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="9.582831688033088e+08"} 1
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.0680243876372875e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.1903330140009282e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.326648249442152e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.4785741108131227e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.6478983046833196e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.8366132632234223e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.046939589088547e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.2813521851760063e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.5426093767255764e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.833785368441068e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.1583064185550647e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.5199911554958534e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.9230955115614314e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.372362802333632e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.873079541115184e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.431137645156319e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.053103765649553e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.746296557296375e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="7.518872796674253e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="8.37992336275598e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="9.339580208980864e+09"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.0409135585614676e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.1601174915283792e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.2929724885225649e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.4410418498852003e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.6060679028781363e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.789992503590971e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.9949798866972237e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.2234421319319225e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.478067546953807e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.7618523005723442e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.0781356785666904e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.4306393769506477e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.823511295046164e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.261374343677016e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="4.749380842807073e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.2932731487183495e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="5.899451224126824e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="6.575047946331352e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="7.328013039544164e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="8.167206619031862e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="9.102503447797786e+10"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.0144909132590578e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.1306689626513626e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.2601515562088194e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.4044623113132697e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.5652993278314426e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.7445551695997415e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="1.9443391341601053e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.167002072794196e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.415164052912417e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="2.691745188300199e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="3.0000000000000085e+11"} 2
changefeed_flush_hist_nanos_bucket{node_id="1",scope="office_dogs",le="+Inf"} 2
changefeed_flush_hist_nanos_sum{node_id="1",scope="office_dogs"} 9.79696709e+08
changefeed_flush_hist_nanos_count{node_id="1",scope="office_dogs"} 2
Was this helpful?
