Google Cloud Composer monitoring integration

BETA

We offer a cloud integration for reporting your GCP Composer data to our platform. Here we explain how to activate the integration and what data it collects.

Activate the integration

To enable the integration follow standard procedures to connect your GCP service.

Configuration and polling

You can change the polling frequency and filter data using configuration options.

Default polling information for the GCP Composer integration:

  • New Relic polling interval: 5 minutes

Find and use data

To find your integration data, go to one.newrelic.com > Infrastructure > GCP and select the integration.

Data is attached to the following event types:

Entity Event Type Provider
Environment GcpComposerEnvironmentSample GcpComposerEnvironment
Workflow GcpComposerWorkflowSample GcpComposerWorkflow

For more on how to use your data, see Understand and use integration data.

Metric data

This integration collects GCP Composer data for Environment and Workflow.

Composer Environment data

Metric Unit Description

environment.api.Request

Count Number of Composer API requests seen so far.

environment.api.RequestLatencies

Milliseconds Distribution of Composer API call latencies.

environment.dag_processing.ParseError

Count Number of errors raised during parsing DAG files.

environment.dag_processing.Processes

Count Number of currently running DAG parsing processes.

environment.dag_processing.ProcessorTimeout

Count Number of file processors terminated due to processing timeout.

environment.dag_processing.TotalParseTime

Seconds Number of seconds taken to scan and import all DAG files once.

environment.DagbagSize

Count The current dag bag size.

environment.database.cpu.ReservedCores

Count Number of cores reserved for the database instance.

environment.database.cpu.UsageTime

Seconds CPU usage time of the database instance.

environment.database.cpu.Utilization

Count CPU utilization ratio (from 0.0 to 1.0) of the database instance.

environment.database.disk.BytesUsed

Bytes Used disk space in bytes on the database instance.

environment.database.disk.Quota

Bytes Maximum data disk size in bytes of the database instance.

environment.database.disk.Utilization

Count Disk quota usage ratio (from 0.0 to 1.0) of the database instance.

environment.database.memory.BytesUsed

Bytes Memory usage of the database instance in bytes.

environment.database.memory.Quota

Bytes Maximum RAM size in bytes of the database instance.

environment.database.memory.Utilization

Count Memory utilization ratio (from 0.0 to 1.0) of the database instance.

environment.executor.OpenSlots

Count Number of open slots on executor.

environment.executor.RunningTasks

Count Number of running tasks on executor.

environment.FinishedTaskInstance

Count Overall task instances.

environment.NumCeleryWorkers

Count Number of Celery workers.

environment.SchedulerHeartbeat

Count Scheduler heartbeats.

environment.TaskQueueLength

Count Number of tasks in queue.

environment.worker.PodEviction

Count The number of Airflow worker pods evictions.

environment.ZombieTaskKilled

Count Number of zombie tasks killed.

Composer Workflow data

Metric Unit Description

workflow.Run

Count Number of workflow runs completed so far.

workflow.RunDuration

Seconds Duration of workflow run completion.

workflow.task.Run

Count Number of workflow tasks completed so far.

workflow.task.RunDuration

Seconds Duration of task completion.

For more help

If you need more help, check out these support and learning resources: