Manage your log collector lifecycle

Overview

The log collector lifecycle management solution in New Relic enables management of your log collection infrastructure across diverse environments. With this solution, you can configure and monitor the lifecycle of your log collectors.

You now have the ability to manage the entire lifecycle of your log collectors. This includes monitoring their health and configuring or deploying changes as needed.

This solution is useful for DevOps teams, system administrators, and IT operations managers who require visibility over their log collection infrastructure. You have access to tools that enhance your observability capabilities and maintain performance in both containerized and on-premise environments.

Solutions for diverse needs

• Log collector health monitoring: The Fluent Bit entity UI provides monitoring capabilities for log collectors.

• Troubleshooting: You can troubleshoot and fix configuration issues with your log collectors using the Fluent Bit entity UI, which helps identify and resolve problems.

  

Manage your Fluent Bit collector for Kubernetes clusters

The log collector lifecycle management solution currently provides support for Kubernetes environments. It enables you to:

• Gather and process logs from all your containerized applications.
• Append Kubernetes-related metadata to your logs.
• Use Fluent Bit's lightweight design to process large volumes of data without consuming a lot of resources.

Prerequisites

Before you install and configure Fluent Bit for logging with New Relic, ensure you have the following prerequisites:

• Access to a Kubernetes cluster
• The necessary tools and permissions for installation and configuration
• Network settings and security policies to allow communication with New Relic endpoints
• A New Relic account

For more information on these requirements, refer to Forward your logs to New Relic.

Set up

To install the Fluent Bit agent in Kubernetes clusters:

• Install the New Relic Kubernetes integration. This integration includes the Kubernetes plugin for logs.
• Use the configuration options available in the newrelic-logging repository during the guided install.

For details, refer to Enable Kubernetes for log management.

Monitor the health and performance of your log collector

Effective performance monitoring of the Fluent Bit agent is crucial for log management and optimization within your Kubernetes environments. By tracking key metrics, you can identify potential issues and facilitate troubleshooting.

Key performance metrics to track

• Log throughput rate: Measures the efficiency of log flow and highlights any disruptions. It includes an overall throughput assessment, which is further broken down by input and output plugins to show the flow of records and bytes processed.
• Memory buffer utilization: Helps identify resource constraints affecting log processing. It includes buffer usage, instances of overflows, queue length, and overall process and container memory utilization.
• Container events and crash loop backoffs: Alerts to abnormal conditions or failures in log pods, including errors, dropped records, and retries. These insights help in identifying and addressing issues that may disrupt log processing.

You can monitor the health and performance of your Fluent Bit agent on the Fluent Bit entity dashboard. Here’s how you can access it:

Via Integrations & Agents

1. Open the Installed tab. You’ll see the list of agents and integrations currently installed.
2. Use the search entities field or the Type filter to narrow down the list to Fluent Bit agents.
3. Select the agent that you want to check.

Via All Entities

1. Find the agent that you want to monitor under the Fluent Bit - Kubernetes group.
2. Select the agent.

Upgrade your Fluent Bit agent

Upgrade your Fluent Bit agent regularly to take advantage of the latest features, performance improvements, and security patches. You can learn directly from the New Relic UI when a newer version becomes available.

Before upgrading, ensure your current configurations are backed up. Review any release notes or documentation to understand new features or changes.

Steps to check if you are up-to-date

1. In your New Relic account, click Integrations & Agents, and then open the Installed tab.
   You’ll see the list of agents and integrations currently installed.
2. Use the search entities field or the Type filter to narrow down the list to Fluent Bit agents.
3. Identify the agent that requires an upgrade.
4. Click the Update from vx.y.x to X.Y.Z.
5. In the side panel that slides in, click View release notes.
6. On the Fluent Bit release notes page, note the latest Helm chart version.

After upgrading, verify log collection and processing to confirm the agent operates as expected. Use New Relic's monitoring tools to check that everything is running.