--- title: Google Cloud Dataflow monitoring integration source: https://docs.newrelic.com/docs/infrastructure/google-cloud-platform-integrations/gcp-integrations-list/google-cloud-dataflow-monitoring-integration --- [New Relic integrations](https://docs.newrelic.com/docs/infrastructure/introduction-infra-monitoring) with the [Google Cloud Platform (GCP)](https://cloud.google.com/) include one that reports [Google Cloud Dataflow](https://cloud.google.com/dataflow) data to New Relic. This document explains how to activate the GCP Cloud Dataflow integration and describes the data it reports. ## Features Cloud Dataflow is Google Cloud's fully managed streaming and batch data-processing service based on Apache Beam. New Relic Dataflow integration collects worker resource, throughput, lag, and job-state metrics for each Dataflow job. ## Activate integration [#activate] To enable the integration, follow standard procedures to [connect your GCP service to New Relic](https://docs.newrelic.com/docs/connect-google-cloud-platform-services-infrastructure): - [Connect with Workload Identity Federation (recommended)](https://docs.newrelic.com/docs/connect-google-cloud-platform-services-infrastructure) - [Connect with service account or user account](https://docs.newrelic.com/docs/connect-google-cloud-platform-services-infrastructure) ## Polling frequency [#polling] New Relic integrations query your GCP services according to a polling interval that varies by integration. The polling frequency for Google Cloud Dataflow is 5 minutes. The resolution is 1 data point every minute. > #### ⚠️ IMPORTANT > > Cloud Dataflow supports 1-minute polling intervals for WIF integrations in limited preview. To request access, contact [New Relic support](https://support.newrelic.com/). ## Workload Identity Federation [#wif] ### Find and use data [#find-data-wif] After you enable the integration, your Dataflow resources appear as entities in the New Relic entity explorer. To see dashboards and manage services, go to [one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infrastructure > GCP. All Dataflow metrics available in GCP Cloud Monitoring are collected as dimensional metrics in the `Metric` event type. Additional metrics beyond this table are collected automatically. See [Google's Dataflow metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp_d_h#gcp-dataflow) for the complete list. #### Entities **Dataflow entities** | Entity | Entity type | Resource type | | ------ | ---------------- | -------------- | | Job | `GCPDATAFLOWJOB` | `dataflow_job` | ### Metric data [#metrics-wif] #### Key metrics — Job | Metric name | Unit | Description | | ------------------------------------------ | ------- | --------------------------------------------------------------------------- | | `gcp.dataflow.job.current_num_vcpus` | Count | Number of vCPUs currently allocated to the Dataflow job. | | `gcp.dataflow.job.total_vcpu_time` | Seconds | Total vCPU-seconds consumed by the Dataflow job. | | `gcp.dataflow.job.elements_produced_count` | Count | Number of elements produced by the Dataflow job, faceted by transform. | | `gcp.dataflow.job.system_lag` | Seconds | Maximum time an element has been waiting in the Dataflow pipeline. | | `gcp.dataflow.job.data_watermark_age` | Seconds | Age of the data watermark for the Dataflow job. | | `gcp.dataflow.job.per_stage_system_lag` | Seconds | System lag observed at each stage of the Dataflow job. | | `gcp.dataflow.job.user_counter` | Count | User-defined counters emitted by the Dataflow job, faceted by counter name. | For the complete list of Dataflow metrics, see [Google's Dataflow metrics documentation](https://cloud.google.com/monitoring/api/metrics_gcp_d_h#gcp-dataflow). ## Service account or user account [#service-account] ### Find and use data [#find-data] To find your integration data, go to [one.newrelic.com > All capabilities](https://one.newrelic.com/all-capabilities) > Infrastructure > GCP and select an integration. Data is attached to the following [event type](https://docs.newrelic.com/docs/data-apis/understand-data/new-relic-data-types/#event-data): | Entity | Event Type | Provider | | ------ | ---------------------- | ---------------- | | Job | `GcpDataflowJobSample` | `GcpDataflowJob` | For more on how to use your data, see [Understand and use integration data](https://docs.newrelic.com/docs/infrastructure/integrations/find-use-infrastructure-integration-data). ## Metric data [#metrics] This integration collects GCP Dataflow data for Job. ### Dataflow Job data | Metric | Unit | Description | | ---------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `job.BillableShuffleDataProcessed` | Bytes | The billable bytes of shuffle data processed by this Dataflow job. | | `job.CurrentNumVcpus` | Count | The number of vCPUs currently being used by this Dataflow job. This is the current number of workers times the number of vCPUs per worker. | | `job.CurrentShuffleSlots` | Count | The current shuffle slots used by this Dataflow job. | | `job.DataWatermarkAge` | Seconds | The age (time since event timestamp) up to which all data has been processed by the pipeline. | | `job.ElapsedTime` | Seconds | Duration that the current run of this pipeline has been in the Running state so far, in seconds. When a run completes, this stays at the duration of that run until the next run starts. | | `job.Elements` | Count | Number of elements added to the pcollection so far. | | `job.EstimatedBytes` | Bytes | An estimated number of bytes added to the pcollection so far. Dataflow calculates the average encoded size of elements in a pcollection and mutiplies it by the number of elements. | | `job.IsFailed` | Count | Has this job failed. | | `job.PerStageDataWatermarkAge` | Seconds | The age (time since event timestamp) up to which all data has been processed by this stage of the pipeline. | | `job.PerStageSystemLag` | Seconds | The current maximum duration that an item of data has been processing or awaiting processing in seconds, per pipeline stage. | | `job.SystemLag` | Seconds | The current maximum duration that an item of data has been processing or awaiting processing, in seconds. | | `job.TotalMemoryUsageTime` | Other | The total GB seconds of memory allocated to this Dataflow job. | | `job.TotalPdUsageTime` | Other | The total GB seconds for all persistent disk used by all workers associated with this Dataflow job. | | `job.TotalShuffleDataProcessed` | Bytes | The total bytes of shuffle data processed by this Dataflow job. | | `job.TotalStreamingDataProcessed` | Bytes | The total bytes of streaming data processed by this Dataflow job. | | `job.TotalVcpuTime` | Seconds | The total vCPU seconds used by this Dataflow job. | | `job.UserCounter` | Count | A user-defined counter metric. |