--- title: Google VertexAI monitoring integration source: https://docs.newrelic.com/docs/infrastructure/google-cloud-platform-integrations/gcp-integrations-list/google-vertexai-monitoring-integration --- [New Relic integrations](https://docs.newrelic.com/docs/infrastructure/introduction-infra-monitoring) with the [Google Cloud Platform (GCP)](https://cloud.google.com/) include one that reports [Google Cloud Vertex AI](https://cloud.google.com/vertex-ai) data to New Relic. This document explains how to activate the GCP Vertex AI integration and describes the data it reports. ## Features Vertex AI is Google Cloud's unified platform for building, deploying, and managing machine-learning models and generative-AI applications. New Relic Vertex AI integration collects performance, throughput, latency, and resource utilization metrics across locations, endpoints, indexes, feature stores, and feature online stores. ## Activate integration [#activate] To enable the integration, follow standard procedures to [connect your GCP service to New Relic](https://docs.newrelic.com/docs/connect-google-cloud-platform-services-infrastructure): - [Connect with Workload Identity Federation (recommended)](https://docs.newrelic.com/docs/infrastructure/google-cloud-platform-integrations/get-started/install-and-configure/connect-gcp-workload-identity-federation) - [Connect with service account or user account](https://docs.newrelic.com/docs/infrastructure/google-cloud-platform-integrations/get-started/install-and-configure/connect-gcp-service-account) ## Polling frequency [#polling] New Relic integrations query your GCP services according to a polling interval that varies by integration. The polling frequency for Google Cloud Vertex AI is 5 minutes. The resolution is 1 data point every minute. ## Workload Identity Federation [#wif] ### Find and use data [#find-data-wif] After you enable the integration, your Vertex AI resources appear as entities in the New Relic entity explorer. To see dashboards and manage services, go to [one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infrastructure > GCP. All Vertex AI metrics available in GCP Cloud Monitoring are collected as dimensional metrics in the `Metric` event type. Additional metrics beyond this table are collected automatically. See [Google's Vertex AI metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp_a_b#gcp-aiplatform) for the complete list. #### Entities **Vertex AI entities** | Entity | Entity type | Resource type | | -------------------- | --------------------------------- | ---------------------------------------------- | | Location | `GCPAIPLATFORMLOCATION` | `aiplatform.googleapis.com/Location` | | Endpoint | `GCPAIPLATFORMENDPOINT` | `aiplatform.googleapis.com/Endpoint` | | Index | `GCPAIPLATFORMINDEX` | `aiplatform.googleapis.com/Index` | | Featurestore | `GCPAIPLATFORMFEATURESTORE` | `aiplatform.googleapis.com/Featurestore` | | Feature Online Store | `GCPAIPLATFORMFEATUREONLINESTORE` | `aiplatform.googleapis.com/FeatureOnlineStore` | | Pipeline Job | `GCPAIPLATFORMPIPELINEJOB` | `aiplatform.googleapis.com/PipelineJob` | ### Metric data [#metrics-wif] #### Key metrics — Location | Metric name | Unit | Description | | -------------------------------------------------------------------------- | ------------ | ---------------------------------------------------------------------- | | `gcp.aiplatform.executing_vertexai_pipeline_jobs` | Count | Number of pipeline jobs currently being executed in the location. | | `gcp.aiplatform.executing_vertexai_pipeline_tasks` | Count | Number of pipeline tasks currently being executed in the location. | | `gcp.aiplatform.online_prediction_requests_per_base_model` | Count | Number of online prediction requests per project per base model. | | `gcp.aiplatform.quota.online_prediction_requests_per_base_model.usage` | Count | Current quota usage for online prediction requests per base model. | | `gcp.aiplatform.quota.online_prediction_requests_per_base_model.limit` | Count | Current quota limit for online prediction requests per base model. | | `gcp.aiplatform.quota.online_prediction_requests_per_base_model.exceeded` | Count | Number of attempts that exceeded the online prediction requests quota. | | `gcp.aiplatform.online_prediction_input_tokens_per_minute_per_base_model` | Count | Online prediction input tokens per minute per project per base model. | | `gcp.aiplatform.online_prediction_output_tokens_per_minute_per_base_model` | Count | Online prediction output tokens per minute per project per base model. | | `gcp.aiplatform.publisher.online_serving.token_count` | Count | Total tokens served by publisher models in the location. | | `gcp.aiplatform.publisher.online_serving.first_token_latencies` | Milliseconds | Latency to first response token from publisher models. | For the complete list of location metrics, see [Google's Vertex AI metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp_a_b#gcp-aiplatform). #### Key metrics — Endpoint | Metric name | Unit | Description | | ---------------------------------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------ | | `gcp.aiplatform.prediction.online.accelerator.duty_cycle` | Percent | Average fraction of time over the past sample period during which the accelerators were actively processing. | | `gcp.aiplatform.prediction.online.accelerator.memory.bytes_used` | Bytes | Accelerator memory allocated by the deployed model replica. | | `gcp.aiplatform.prediction.online.cpu.utilization` | Percent | CPU utilization of the deployed model replica. | | `gcp.aiplatform.prediction.online.memory.bytes_used` | Bytes | Memory in use by the deployed model replica. | | `gcp.aiplatform.prediction.online.network.received_bytes_count` | Bytes | Network bytes received by the deployed model replica. | | `gcp.aiplatform.prediction.online.network.sent_bytes_count` | Bytes | Network bytes sent by the deployed model replica. | | `gcp.aiplatform.prediction.online.prediction_count` | Count | Number of online predictions served by the endpoint. | | `gcp.aiplatform.prediction.online.prediction_latencies` | Milliseconds | Online prediction latency of the deployed model. | | `gcp.aiplatform.prediction.online.error_count` | Count | Number of online prediction errors returned by the endpoint. | | `gcp.aiplatform.prediction.online.response_count` | Count | Number of online prediction responses returned by the endpoint, faceted by response code. | | `gcp.aiplatform.prediction.online.replicas` | Count | Number of active replicas serving the deployed model. | | `gcp.aiplatform.prediction.online.target_replicas` | Count | Target number of active replicas for the deployed model. | For the complete list of endpoint metrics, see [Google's Vertex AI metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp_a_b#gcp-aiplatform). #### Key metrics — Index | Metric name | Unit | Description | | -------------------------------------------------------------- | ------------ | ------------------------------------------------------------------------ | | `gcp.aiplatform.matching_engine.current_shards` | Count | Current number of shards backing the index. | | `gcp.aiplatform.matching_engine.stream_update.request_count` | Count | Number of stream-update requests sent to the index. | | `gcp.aiplatform.matching_engine.stream_update.datapoint_count` | Count | Number of datapoints successfully upserted or removed via stream update. | | `gcp.aiplatform.matching_engine.stream_update.latencies` | Milliseconds | Latency between a stream-update response and the update taking effect. | For the complete list of index metrics, see [Google's Vertex AI metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp_a_b#gcp-aiplatform). #### Key metrics — Featurestore | Metric name | Unit | Description | | --------------------------------------------------------------------- | ------------ | ------------------------------------------------------------------------------------------ | | `gcp.aiplatform.featurestore.cpu_load` | Percent | Average CPU load across nodes in the Featurestore online storage. | | `gcp.aiplatform.featurestore.cpu_load_hottest_node` | Percent | CPU load on the hottest node in the Featurestore online storage. | | `gcp.aiplatform.featurestore.node_count` | Count | Number of nodes provisioned for the Featurestore online storage. | | `gcp.aiplatform.featurestore.online_entities_updated` | Count | Number of entities updated in the Featurestore online storage. | | `gcp.aiplatform.featurestore.online_serving.request_count` | Count | Number of online serving requests handled by the Featurestore, faceted by EntityType. | | `gcp.aiplatform.featurestore.online_serving.latencies` | Milliseconds | Online serving request latency at the Featurestore, faceted by EntityType. | | `gcp.aiplatform.featurestore.online_serving.request_bytes_count` | Bytes | Online serving request size at the Featurestore, faceted by EntityType. | | `gcp.aiplatform.featurestore.online_serving.response_size` | Bytes | Online serving response size at the Featurestore, faceted by EntityType. | | `gcp.aiplatform.featurestore.storage.stored_bytes` | Bytes | Total data stored in the Featurestore. | | `gcp.aiplatform.featurestore.storage.billable_processed_bytes` | Bytes | Billable bytes processed for Featurestore offline data. | | `gcp.aiplatform.featurestore.streaming_write.offline_processed_count` | Count | Number of streaming-write requests processed into offline storage. | | `gcp.aiplatform.featurestore.streaming_write.offline_write_delays` | Seconds | Time from when the streaming-write API is called until the record reaches offline storage. | For the complete list of featurestore metrics, see [Google's Vertex AI metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp_a_b#gcp-aiplatform). #### Key metrics — Feature Online Store | Metric name | Unit | Description | | -------------------------------------------------------------------------- | ------------ | ---------------------------------------------------------------------------------------------- | | `gcp.aiplatform.featureonlinestore.online_serving.request_count` | Count | Number of online serving requests handled by the Feature Online Store, faceted by FeatureView. | | `gcp.aiplatform.featureonlinestore.online_serving.serving_latencies` | Milliseconds | Online serving request latency at the Feature Online Store, faceted by FeatureView. | | `gcp.aiplatform.featureonlinestore.online_serving.serving_bytes_count` | Bytes | Online serving response size at the Feature Online Store, faceted by FeatureView. | | `gcp.aiplatform.featureonlinestore.running_sync` | Count | Number of syncs currently running at the Feature Online Store. | | `gcp.aiplatform.featureonlinestore.serving_data_ages` | Seconds | Age of the data being served by the Feature Online Store. | | `gcp.aiplatform.featureonlinestore.serving_data_by_sync_time` | Count | Breakdown of records in the Feature Online Store by synced timestamp. | | `gcp.aiplatform.featureonlinestore.storage.bigtable_cpu_load` | Percent | Average CPU load across Bigtable nodes backing the Feature Online Store. | | `gcp.aiplatform.featureonlinestore.storage.bigtable_cpu_load_hottest_node` | Percent | CPU load on the hottest Bigtable node backing the Feature Online Store. | | `gcp.aiplatform.featureonlinestore.storage.bigtable_nodes` | Count | Number of Bigtable nodes backing the Feature Online Store. | | `gcp.aiplatform.featureonlinestore.storage.stored_bytes` | Bytes | Total data stored in the Feature Online Store. | For the complete list of feature online store metrics, see [Google's Vertex AI metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp_a_b#gcp-aiplatform). #### Key metrics — Pipeline Job | Metric name | Unit | Description | | ------------------------------------------------- | ------- | ------------------------------------------------------------------------- | | `gcp.aiplatform.pipelinejob.duration` | Seconds | Runtime seconds of the pipeline job being executed, from creation to end. | | `gcp.aiplatform.pipelinejob.task_completed_count` | Count | Total number of completed pipeline tasks in the pipeline job. | For the complete list of pipeline job metrics, see [Google's Vertex AI metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp_a_b#gcp-aiplatform). ### Metrics-only resource types [#metrics-only] New Relic also collects metrics for the following Vertex AI resource types. These resource types don't create entities in the entity explorer, so query their metrics directly on the `Metric` event type. **Publisher Model — 15 metrics** | Metric name | Unit | Description | | ------------------------------------------------------------------------------- | ------------ | ------------------------------------------------------------------------------ | | `gcp.aiplatform.publisher.online_serving.character_count` | Count | Accumulated input/output character count. | | `gcp.aiplatform.publisher.online_serving.characters` | Count | Input/output character count distribution. | | `gcp.aiplatform.publisher.online_serving.consumed_throughput` | Count | Overall throughput used (accounting for burndown rate) in terms of characters. | | `gcp.aiplatform.publisher.online_serving.consumed_token_throughput` | Count | Overall throughput used (accounting for burndown rate) in terms of tokens. | | `gcp.aiplatform.publisher.online_serving.dedicated_character_limit` | Per second | Dedicated limit in characters per second. | | `gcp.aiplatform.publisher.online_serving.dedicated_character_project_max_limit` | Per second | Project max limit in characters per second that can be consumed. | | `gcp.aiplatform.publisher.online_serving.dedicated_gsu_limit` | Count | Dedicated limit in GSU. | | `gcp.aiplatform.publisher.online_serving.dedicated_gsu_project_max_limit` | Count | Project max limit in GSU that can be consumed. | | `gcp.aiplatform.publisher.online_serving.dedicated_token_limit` | Per second | Dedicated limit in tokens per second. | | `gcp.aiplatform.publisher.online_serving.dedicated_token_project_max_limit` | Per second | Project max limit in tokens per second that can be consumed. | | `gcp.aiplatform.publisher.online_serving.first_token_latencies` | Milliseconds | Duration from request received to first token sent back to the client. | | `gcp.aiplatform.publisher.online_serving.model_invocation_count` | Count | Number of model invocations (prediction requests). | | `gcp.aiplatform.publisher.online_serving.model_invocation_latencies` | Milliseconds | Model invocation latencies (prediction latencies). | | `gcp.aiplatform.publisher.online_serving.token_count` | Count | Accumulated input/output token count. | | `gcp.aiplatform.publisher.online_serving.tokens` | Count | Input/output token count distribution. | **Deployment Resource Pool — 8 metrics** | Metric name | Unit | Description | | ----------------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `gcp.aiplatform.prediction.online.deployment_resource_pool.accelerator.duty_cycle` | Percent | Average fraction of time over the past sample period during which the accelerator(s) were actively processing. | | `gcp.aiplatform.prediction.online.deployment_resource_pool.accelerator.memory.bytes_used` | Bytes | Amount of accelerator memory allocated by the deployment resource pool replica. | | `gcp.aiplatform.prediction.online.deployment_resource_pool.cpu.utilization` | Percent | Fraction of CPU allocated by the deployment resource pool replica and currently in use. May exceed 100% if the machine type has multiple CPUs. | | `gcp.aiplatform.prediction.online.deployment_resource_pool.memory.bytes_used` | Bytes | Amount of memory allocated by the deployment resource pool replica and currently in use. | | `gcp.aiplatform.prediction.online.deployment_resource_pool.network.received_bytes_count` | Bytes | Number of bytes received over the network by the deployment resource pool replica. | | `gcp.aiplatform.prediction.online.deployment_resource_pool.network.sent_bytes_count` | Bytes | Number of bytes sent over the network by the deployment resource pool replica. | | `gcp.aiplatform.prediction.online.deployment_resource_pool.replicas` | Count | Number of active replicas used by the deployment resource pool. | | `gcp.aiplatform.prediction.online.deployment_resource_pool.target_replicas` | Count | Target number of active replicas needed for the deployment resource pool. | **Reasoning Engine — 8 metrics** | Metric name | Unit | Description | | --------------------------------------------------------------------------- | ------------ | ----------------------------------------------------------------------------- | | `gcp.aiplatform.reasoning_engine.cpu.allocation_time` | CPU-seconds | Container CPU allocation in seconds. | | `gcp.aiplatform.reasoning_engine.memory.allocation_time` | GiB-seconds | Container memory allocation in Gigabytes-seconds. | | `gcp.aiplatform.reasoning_engine.memory_bank.generate_memories_token_count` | Count | The number of tokens used in memory generation requests per Reasoning Engine. | | `gcp.aiplatform.reasoning_engine.memory_bank.memory_lro_latency` | Milliseconds | The total latency of memory LROs. | | `gcp.aiplatform.reasoning_engine.memory_bank.memory_mutation_count` | Count | The number of memory mutation operations on the Reasoning Engine. | | `gcp.aiplatform.reasoning_engine.memory_bank.memory_retrieval_count` | Count | The number of memory entities retrieved per Reasoning Engine. | | `gcp.aiplatform.reasoning_engine.request_count` | Count | Number of requests reaching the instance. | | `gcp.aiplatform.reasoning_engine.request_latencies` | Milliseconds | Distribution of request latency in milliseconds reaching the instance. | **Index Endpoint — 6 metrics** | Metric name | Unit | Description | | -------------------------------------------------------- | ------------ | --------------------------------------------------------------------------------------- | | `gcp.aiplatform.matching_engine.current_replicas` | Count | Number of active replicas used by the DeployedIndex. | | `gcp.aiplatform.matching_engine.current_shards` | Count | Number of shards of the DeployedIndex. | | `gcp.aiplatform.matching_engine.query.latencies` | Milliseconds | Server side request latency. | | `gcp.aiplatform.matching_engine.query.request_count` | Count | Number of requests received. | | `gcp.aiplatform.matching_engine.cpu.request_utilization` | Percent | The fraction of the requested CPU that is currently in use on a match server container. | | `gcp.aiplatform.matching_engine.memory.used_bytes` | Bytes | The memory used in bytes for a match server container. | **Model — 3 metrics** | Metric name | Unit | Description | | ------------------------------------------------------------------- | ----- | ------------------------------------------------------------- | | `gcp.aiplatform.model_monitoring.feature_attribution_deviation` | Count | The feature attribution deviation value of the given feature. | | `gcp.aiplatform.model_monitoring.feature_drift_deviation` | Count | The drift deviation value of the given feature. | | `gcp.aiplatform.model_monitoring.prediction_output_drift_deviation` | Count | The drift deviation value of the prediction output. | **Online Evaluator — 2 metrics** | Metric name | Unit | Description | | --------------------------------------------------- | ----- | ----------------------------------------------------------- | | `gcp.aiplatform.online_evaluator.evaluations_count` | Count | The number of evaluations of the Online Evaluator resource. | | `gcp.aiplatform.online_evaluator.scores` | Count | The distribution of scores of the Online Evaluator. | ## Service account or user account [#service-account] ### Find and use data [#find-data] After activating the integration and waiting a few minutes (based on the [polling frequency](#polling)), data will appear in the New Relic UI. To [find and use your data](https://docs.newrelic.com/docs/infrastructure/integrations/find-use-infrastructure-integration-data), including links to your dashboards and alert settings, go to [one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infrastructure > GCP > (select an integration). Data is attached to the following [event types](https://docs.newrelic.com/docs/data-apis/understand-data/new-relic-data-types/#event-data): | Entity | Event Type | Provider | | -------------------- | ------------------------------------- | ------------------------------- | | Endpoint | `GcpVertexAiEndpointSample` | `GcpVertexAiEndpoint` | | Feature store | `GcpVertexAiFeaturestoreSample` | `GcpVertexAiFeaturestore` | | Feature Online Store | `GcpVertexAiFeatureOnlineStoreSample` | `GcpVertexAiFeatureOnlineStore` | | Location | `GcpVertexAiLocationSample` | `GcpVertexAiLocation` | | Index | `GcpVertexAiIndexSample` | `GcpVertexAiIndex` | | PipelineJob | `GcpVertexAiPipelineJobSample` | `GcpVertexAiPipelineJob` | For more on how to use your data, see [Understand and use integration data](https://docs.newrelic.com/docs/infrastructure/integrations/find-use-infrastructure-integration-data). ### VertexAI Endpoint data | Metric | Unit | Description | | ------------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------------- | | `prediction.online.accelerator.duty_cycle` | Percent | Average fraction of time over the past sample period during which the accelerator(s) were actively processing. | | `prediction.online.accelerator.memory.bytes_used` | Bytes | Amount of accelerator memory allocated by the deployed model replica. | | `prediction.online.error_count` | Count | Number of online prediction errors. | | `prediction.online.memory.bytes_used` | Bytes | Amount of memory allocated by the deployed model replica and currently in use. | | `prediction.online.network.received_bytes_count` | Bytes | Number of bytes received over the network by the deployed model replica. | | `prediction.online.network.sent_bytes_count` | Bytes | Number of bytes sent over the network by the deployed model replica. | | `prediction.online.prediction_count` | Count | Number of online predictions. | | `prediction.online.prediction_latencies` | Milliseconds | Online prediction latency of the deployed model. | | `prediction.online.private.prediction_latencies` | Milliseconds | Online prediction latency of the private deployed model. | | `prediction.online.replicas` | Count | Number of active replicas used by the deployed model. | | `prediction.online.response_count` | Count | Number of different online prediction response codes. | | `prediction.online.target_replicas` | Count | Target number of active replicas needed for the deployed model. | ### VertexAI Featurestore data | Metric | Unit | Description | | ------------------------------------------------------ | ------------ | -------------------------------------------------------------------------------------- | | `featurestore.cpu_load` | Percent | The average CPU load for a node in the Featurestore online storage. | | `featurestore.cpu_load_hottest_node` | Percent | The CPU load for the hottest node in the Featurestore online storage. | | `featurestore.node_count` | Count | The number of nodes for the Featurestore online storage. | | `featurestore.online_entities_updated` | Count | Number of entities updated on the Featurestore online storage. | | `featurestore.online_serving.latencies` | Milliseconds | Online serving latencies by EntityType. | | `featurestore.online_serving.request_bytes_count` | Bytes | Request size by EntityType. | | `featurestore.online_serving.request_count` | Count | Featurestore online serving count by EntityType. | | `featurestore.online_serving.response_size` | Bytes | Response size by EntityType. | | `featurestore.storage.billable_processed_bytes` | Bytes | Number of bytes billed for offline data processed. | | `featurestore.storage.stored_bytes` | Bytes | Bytes stored in Featurestore. | | `featurestore.streaming_write.offline_processed_count` | Count | Number of streaming write requests processed for offline storage. | | `featurestore.streaming_write.offline_write_delays` | Seconds | Time (in second) since the write API is called until it is written to offline storage. | ### VertexAI FeatureOnlineStore data | Metric | Unit | Description | | ----------------------------------------------------------- | ------------ | -------------------------------------------------------------- | | `featureonlinestore.online_serving.request_count` | Count | Number of serving count by FeatureView. | | `featureonlinestore.online_serving.serving_bytes_count` | Bytes | Serving response size by FeatureView. | | `featureonlinestore.online_serving.serving_latencies` | Milliseconds | Online serving latencies by FeatureView. | | `featureonlinestore.running_sync` | Milliseconds | Number of running syncs at given point of time. | | `featureonlinestore.serving_data_ages` | Seconds | Measure of the serving data age in seconds. | | `featureonlinestore.serving_data_by_sync_time` | Count | Breakdown of data in Feature Online Store by synced timestamp. | | `featureonlinestore.storage.bigtable_cpu_load` | Percent | The average CPU load of nodes in the Feature Online Store. | | `featureonlinestore.storage.bigtable_cpu_load_hottest_node` | Percent | The CPU load of the hottest node in the Feature Online Store. | | `featureonlinestore.storage.bigtable_nodes` | Count | The number of nodes for the Feature Online Store(Bigtable). | | `featureonlinestore.storage.stored_bytes` | Count | Bytes stored in the Feature Online Store. | ### VertexAI Location data | Metric | Unit | Description | | ---------------------------------------------------------- | ----- | ------------------------------------------------------- | | `online_prediction_requests_per_base_model` | Count | Number of requests per base model. | | `quota.online_prediction_requests_per_base_model.exceeded` | Count | Number of attempts to exceed the limit on quota metric. | | `quota.online_prediction_requests_per_base_model.limit` | Count | Current limit on quota metric. | | `quota.online_prediction_requests_per_base_model.usage` | Count | Current usage on quota metric. | | `executing_vertexai_pipeline_jobs` | Count | Number of pipeline jobs being executed. | | `executing_vertexai_pipeline_tasks` | Count | Number of pipeline tasks being executed. | ### VertexAI Index data | Metric | Unit | Description | | ----------------------------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------- | | `matching_engine.stream_update.datapoint_count` | Count | Number of successfully upserted or removed datapoints. | | `matching_engine.stream_update.latencies` | Milliseconds | The latencies between the user receives a UpsertDatapointsResponse or RemoveDatapointsResponse and that update takes effect. | | `matching_engine.stream_update.request_count` | Count | Number of stream update requests. | ### VertexAI Pipeline Job data | Metric | Unit | Description | | ---------------------------------- | ------- | -------------------------------------------------------------------------- | | `pipelinejob.duration` | Seconds | Runtime seconds of the pipeline job being executed (from creation to end). | | `pipelinejob/task_completed_count` | Count | Total number of completed Pipeline Tasks. |