Amazon RDS Enhanced Monitoring integration

New Relic infrastructure integrations include an integration for collecting enhanced Amazon RDS data; this supplements the basic New Relic RDS integration with real-time metrics about the operating system the database instance runs on.

Features

This New Relic integration allows you to monitor and alert on RDS Enhanced Monitoring. You can use integration data and to monitor the DB processes and identify potential trouble spots as well as to profile the DB allowing you to improve and optimize their response and cost.

Enable enhanced monitoring

Important

Enabling this integration will incur some additional charges to your Amazon CloudWatch account. In addition there are some limitations and CPU metric data collection differences that are explained in Amazon's enhanced monitoring documentation.

You must first have the New Relic Amazon RDS monitoring integration enabled before enabling RDS Enhanced Monitoring. Be sure that you have completed the steps in Connect AWS services to New Relic.

New Relic uses AWS Lambda in order to collect RDS logs and provide near-real time data of your RDS instances, this capability is called RDS Enhanced Monitoring in AWS.

Follow these steps to enable RDS Enhanced Monitoring integration:

Specify the RDS instances that need Enable RDS Enhanced Monitoring. You can specify this when creating or modifying the instance: under Monitoring, set Enable Enhanced Monitoring to Yes. We recommend setting the data Granularity to 15 seconds.
After enhanced monitoring is enabled, a stream called RDSOSMetrics is created in AWS CloudWatch Logs. Enhanced monitoring metrics are available via this stream. Create a lambda function and subscribe it to that stream in the following steps to obtain the data.
Create a new AWS Lambda function from the Serverless Repository: Go to Lambda > Create Function > Browse serverless App repository, check the box for Show apps that create custom IAM roles or resource policies, and search for NewRelic-log-ingestion.
Populate the LICENSE_KEY environment variable with your New Relic account .
Review all optional parameters and adapt them based on your use cases.
Select Deploy to create a new CloudFormation stack, a new function called newrelic-log-ingestion, and the required role.
Go to the newrelic-log-ingestion function.
Continue with the procedure to stream logs to the Lambda function.

Tip

The newrelic-log-ingestion function requires the AWSLambdaBasicExecutionRole policy which contains the minimum permissions (as recommended by AWS). A custom IAM role name can be defined at install time, otherwise, an appropriate Role will be created, which will require CAPABILITY_IAM to be acknowledged.

Stream logs to Lambda function

To link the RDSOSMetrics log stream to the Lambda function (JSON format):

From AWS Console > CloudWatch > Logs, select RDSOSMetrics log group, and apply Actions > Create Lambda subscription filter.
For the Lambda function, select newrelic-log-ingestion.
From the Log Format dropdown, select JSON as the Log format.
Enter a Subscription filter name.
At the bottom, click the Start streaming button to save your Lambda subscription filter.

Once completed, the Lambda function will send all the log lines from RDSOSMetrics to New Relic's ingest services.

Configuration and polling

You can change the polling frequency and filter data using configuration options.

Default polling information for the Amazon RDS Enhanced Monitoring integration:

New Relic polling interval:
- 30 seconds on average (collected via CloudWatch Logs)
- Configurable when setting up AWS Lambda
Amazon CloudWatch data interval: 1 minute

Find and use data

To find your integration data, go to one.newrelic.com > All capabilities > Infrastructure > AWS and select the RDS > Enhanced monitoring dashboard link.

You can query and explore your data using the DatastoreSample event type, with a provider value of RdsDbInstance .

For more on how to use your data, see Understand and use integration data.

Metric data

New Relic collects the following enhanced RDS data:

Metric data for all DB engines (except MS SQL Server)

Group	Metrics	Description
`General`	`engine`	The database engine for the DB instance.
	`instanceId`	The DB instance identifier.
	`instanceResourceId`	A region-unique, immutable identifier for the DB instance, also used as the log stream identifier.
	`numVCpus`	The number of virtual CPUs for the DB instance.
	`timestamp`	The time at which the metrics were taken.
	`uptime`	The amount of time that the DB instance has been active.
	`version`	The version of the OS metrics' stream JSON format.
`cpuUtilization`	`guest`	The percentage of CPU in use by guest programs.
	`idle`	The percentage of CPU that is idle.
	`irq`	The percentage of CPU in use by software interrupts.
	`nice`	The percentage of CPU in use by programs running at lowest priority.
	`steal`	The percentage of CPU in use by other virtual machines.
	`system`	The percentage of CPU in use by the kernel.
	`total`	The total percentage of the CPU in use. This value excludes the nice value.
	`user`	The percentage of CPU in use by user programs.
	`wait`	The percentage of CPU unused while waiting for I/O access.
`diskIO` (not available for Amazon Aurora)	`avgQueueLen`	The number of requests waiting in the I/O device's queue.
	`avgReqSz`	The average request size, in kilobytes.
	`await`	The number of milliseconds required to respond to requests, including queue time and service time.
	`device`	The identifier of the disk device in use.
	`readIOsPS`	The number of read operations per second.
	`readKb`	The total number of kilobytes read.
	`readKbPS`	The number of kilobytes read per second.
	`rrqmPS`	The number of merged read requests queued per second.
	`tps`	The number of I/O transactions per second.
	`util`	The percentage of CPU time during which requests were issued.
	`writeIOsPS`	The number of write operations per second.
	`writeKb`	The total number of kilobytes written.
	`writeKbPS`	The number of kilobytes written per second.
	`wrqmPS`	The number of merged write requests queued per second.
`fileSys`	`maxFiles`	The maximum number of files that can be created for the file system.
	`total`	The total number of disk space available for the file system, in kilobytes.
	`used`	The amount of disk space used by files in the file system, in kilobytes.
	`usedFilePercent`	The percentage of available files in use.
	`usedFiles`	The number of files in the file system.
	`usedPercent`	The percentage of the file-system disk space in use.
`loadAverageMinute`	`fifteen`	The number of processes requesting CPU time over the last 15 minutes.
	`five`	The number of processes requesting CPU time over the last 5 minutes.
	`one`	The number of processes requesting CPU time over the last minute.
`memory`	`active`	The amount of assigned memory, in kilobytes.
	`buffers`	The amount of memory used for buffering I/O requests prior to writing to the storage device, in kilobytes.
	`cached`	The amount of memory used for caching file system–based I/O.
	`dirty`	The amount of memory pages in RAM that have been modified but not written to their related data block in storage, in kilobytes.
	`free`	The amount of unassigned memory, in kilobytes.
	`hugePagesFree`	The number of free huge pages. Huge pages are a feature of the Linux kernel.
	`hugePagesRsvd`	The number of committed huge pages.
	`hugePagesSize`	The size for each huge pages unit, in kilobytes.
	`hugePagesSurp`	The number of available surplus huge pages over the total.
	`hugePagesTotal`	The total number of huge pages for the system.
	`inactive`	The amount of least-frequently used memory pages, in kilobytes.
	`mapped`	The total amount of file-system contents that is memory mapped inside a process address space, in kilobytes.
	`pageTables`	The amount of memory used by page tables, in kilobytes.
	`slab`	The amount of reusable kernel data structures, in kilobytes.
	`total`	The total amount of memory, in kilobytes.
	`writeback`	The amount ofn kilobytes.
`network`	`rx`	The number of bytes received per second.
`network`	`tx`	The number of bytes uploaded per second.
`process`	`cpuUsedPc`	The percentage of CPU used by the process.
	`rss`	The amount of RAM allocated to the process, in kilobytes.
	`memoryUsedPc`	The amount of memory used by the process, in kilobytes.
	`processName`	The name of the process.
`swap`	`cached`	The amount of swap memory, in kilobytes, used as cache memory.
	`free`	The total amount of swap memory free, in kilobytes.
	`total`	The total amount of swap memory available, in kilobytes.
`tasks`	`blocked`	The number of tasks that are blocked.
	`running`	The number of tasks that are running.
	`sleeping`	The number of tasks that are sleeping.
	`stopped`	The number of tasks that are stopped.
	`total`	The total number of tasks.
	`zombie`	The number of child tasks that are inactive with an active parent task.

Metric data for MS SQL

Group	Metrics	Description
`disks`	`totalKb`	The total space of the disk, in kilobytes.
	`usedKb`	The amount of space used on the disk, in kilobytes.
	`usedPc`	The percentage of space used on the disk.
	`availKb`	The space available on the disk, in kilobytes.
	`availPc`	The percentage of space available on the disk.
	`rdCountPS`	The number of read operations per second
	`rdBytesPS`	The number of bytes read per second.
	`wrCountPS`	The number of write operations per second.
	`wBytesPS`	The amount of bytes written per second.
`memory`	`commitToKb`	The amount of pagefile-backed virtual address space in use, that is, the current commit charge. This value is composed of main memory (RAM) and disk (pagefiles).
	`commitLimitKb`	The maximum possible value for the commitTotKb metric. This value is the sum of the current pagefile size plus the physical memory available for pageable contents–excluding RAM that is assigned to non-pageable areas.
	`commitPeakKb`	The largest value of the commitTotKb metric since the operating system was last started.
	`kernTotKb`	The sum of the memory in the paged and non-paged kernel pools, in kilobytes.
	`kernPagedKb`	The amount of memory in the paged kernel pool, in kilobytes.
	`kernNonpagedKb`	The amount of memory in the non-paged kernel pool, in kilobytes.
	`pageSize`	The size of a page, in bytes.
	`physTotKb`	The amount of physical memory, in kilobytes.
	`physAvailKb`	The amount of available physical memory, in kilobytes.
	`sqlServerTotKb`	The amount of memory committed to Microsoft SQL Server, in kilobytes.
	`sysCacheKb`	The amount of system cache memory, in kilobytes.
`network`	`rdBytesPS`	The number of bytes received per second.
`network`	`wrBytesPS`	The number of bytes sent per second.
`process`	`cpuUsedPc`	The percentage of CPU used by the process.
	`memUsedPc`	The amount of memory used by the process, in kilobytes.
	`processName`	The name of the process.
	`workingSetKb`	The amount of memory in the private working set plus the amount of memory that is in use by the process and can be shared with other processes, in kilobytes.
	`workingSetPrivKb`	The amount of memory that is in use by a process, but can't be shared with other processes, in kilobytes.
	`workingSetShareableKb`	The amount of memory that is in use by a process and can be shared with other processes, in kilobytes.
	`virtKb`	The amount of virtual address space the process is using, in kilobytes. Use of virtual address space does not necessarily imply corresponding use of either disk or main memory pages.
`system`	`handles`	The number of handles that the system is using.
	`processes`	The number of processes running on the system.
	`threads`	The number of threads running on the system.

Definitions

Term	Description
Event type	`DataStoreSample`
Provider	`RdsDbInstance`
Processes	Enhanced Monitoring allows you to monitor the following processes associated with your RDS instances. : `RDS Process`: Shows a summary of the resources used by the RDS management agent, diagnostics monitoring processes, and other AWS processes that are required to support RDS DB instances. `RDS Child Process`: Nested under RDS Processes, shows a summary of the RDS processes that support the DB instance, for example `aurora` for Amazon Aurora DB clusters and `mysqld` for MySQL DB instances. `OS Processes`: Shows a summary of the kernel and system processes, which generally have minimal impact on performance.

Term

Description

Event type

DataStoreSample

Provider

RdsDbInstance

Processes

Enhanced Monitoring allows you to monitor the following processes associated with your RDS instances. :

RDS Process: Shows a summary of the resources used by the RDS management agent, diagnostics monitoring processes, and other AWS processes that are required to support RDS DB instances.
RDS Child Process: Nested under RDS Processes, shows a summary of the RDS processes that support the DB instance, for example aurora for Amazon Aurora DB clusters and mysqld for MySQL DB instances.
OS Processes: Shows a summary of the kernel and system processes, which generally have minimal impact on performance.