New Relic Infrastructure Agent 1.12.2

Released on: 
Wednesday, August 12, 2020 - 12:00

Notes

A new version of the agent has been released. Follow standard procedures to update your Infrastructure agent.

Fixed

  • PR-50 Fixed inventory deltas submission for remote (non host) entities. This fixes inventory errors showing up on the log for DEBUG level with message error field deltas not in expected order: have delta ID XXX, want YYY.

Context

Inventory deltas are JSON diffs and they need to be applied in order aiming to provide a consistent state. Both the infrastructure agent and the Telemetry Data Platform stored their latest delta ID received/submitted.

This delta ID is incremented in each plugin entity inventory submission to the Telemetry Data Platform. When these IDs go out of sync between agent and platform, the latter provides its latest known ID for the given inventory plugin and entity. Then a reconciliation process is run by the agent for that plugin so that the next ID will match the one expected by the Telemetry Data Platform.

Root cause

For remote, non-host entities the agent side didn't store delta IDs on a per-entity basis as the Telemetry Data Platform did. This means that for a given inventory plugin or integration providing more than 1 entity there'd be delta ID order issues, which were only seen on agent's DEBUG level (verbose mode > 0). At some point all data would usually be submitted as reconciliation process attempts to fix this. But for high load integrations producing multiple remote entities this reconciliation process would be triggered every time an entity needs to be submitted as for each of them platform keeps a record, while the agent recorded ID on a per-plugin basis. Therefore, for high-load integrations, the agent reconciliation process might enter in race-condition loop when trying to fix the delta IDs.

Fix

This release fixes this behaviour by keeping track of delta IDs in the same way our Telemetry Data Platform does (or a per-plugin, per-entity basis).

As state might be stored on disk for inventory deltas, if any integration had it persisted before the fix it'll take a reconciliation run per entity to fix agent's locally stored IDs. In this case you might still see some log entries, but they'll be gone during their first submissions.