With our Apache Flink dashboard, you can easily track your logs, keep an eye on your instrumentation sources, and get an overview of uptime and downtime for all your applications' instances. Built with our infrastructure agent and our Prometheus OpenMetrics integration, Flink take advantage of OpenMetrics endpoint scraping, so you can view all your most important data, all in one place.
After setting up Flink with New Relic, your data will display in dashboards like these, right out of the box.
Before getting Flink data into New Relic, first install our infrastructure agent, then expose your metrics by installing Prometheus OpenMetrics.
There are two ways of installing our infrastructure agent:
You will need the Apache Flink prometheus jar file to expose metrics on a configured port. Ideally, on downloading the Apache Flink, the file named
flink-metrics-prometheus-VERSION.jar will be placed in the below path.
If the above mentioned jar file is not present in your downloaded flink directory, you can download it from other maven jar artifacts. Use the below example URL to download the
flink-metrics-prometheus-VERSION.jar file. Below is the external URL from which you can download it.
Update the flink configuration file to expose metrics on ports 9250 to 9260
Apache Flink configuration file path:
metrics.reporters: prommetrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReportermetrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactorymetrics.reporter.prom.host: localhostmetrics.reporter.prom.port: 9250-9260
Follow the below command to start an Apache Flink cluster
You should now be able to see metrics on the below URLs
- Job manager metrics
- Task manager metrics
You may also cross check if there are any other task manager ports running on your PC by running the below command:
sudo lsof -i -P -n | grep LISTEN
First, create a file named
nri-prometheus-config.yml in the below path
Now, update the fields listed below:
cluster_name: “YOUR_DESIRED_CLUSTER_NAME”, for example:
urls: ["http://YOUR_DOMAIN:9250", "http://YOUR_DOMAIN:9251"], for example:
urls: [“http://localhost:9250”, “http://localhost:9251”]
nri-prometheus-config.yml file should look like this:
integrations:- name: nri-prometheusconfig:standalone: false# Defaults to true. When standalone is set to `false`, `nri-prometheus` requires an infrastructure agent to send data.emitters: infra-sdk# When running with infrastructure agent emitters will have to include infra-sdkcluster_name: "YOUR_DESIRED_CLUSTER_NAME"# Match the name of your cluster with the name seen in New Relic.targets:- description: "YOUR_DESIRED_DESCRIPTION_HERE"urls: ["http://YOUR_DOMAIN:9250", "http://YOUR_DOMAIN:9251"]# tls_config:# ca_file_path: "/etc/etcd/etcd-client-ca.crt"# cert_file_path: "/etc/etcd/etcd-client.crt"# key_file_path: "/etc/etcd/etcd-client.key"verbose: false# Defaults to false. This determines whether or not the integration should run in verbose mode.audit: false# Defaults to false and does not include verbose mode. Audit mode logs the uncompressed data sent to New Relic and can lead to a high log volume.# scrape_timeout: "YOUR_TIMEOUT_DURATION"# `scrape_timeout` is not a mandatory configuration and defaults to 30s. The HTTP client timeout when fetching data from endpoints.scrape_duration: "5s"# worker_threads: 4# `worker_threads` is not a mandatory configuration and defaults to `4` for clusters with more than 400 endpoints. Slowly increase the worker thread until scrape time falls between the desired `scrape_duration`. Note: Increasing this value too much results in huge memory consumption if too many metrics are scraped at once.insecure_skip_verify: false# Defaults to false. Determins if the integration should skip TLS verification or not.timeout: 10s
You can use our log forwarding documentation to forward application specific logs to New Relic.
On installing infrastructure agent on the linux machines, your log file named
logging.yml should be present in this path:
If you don't see your log file in the above path, then you will need to create a log file by following the above log forwarding documentation.
Here is an example of the log name which will look similar as below:
- name: flink-u1-taskexecutor-0-u1-VirtualBox.log
Add the below script to the
logging.yml file to send Apache Flink logs to New Relic.
logs:- name: flink-<REST_OF_THE_FILE_NAME>.logfile: <FLINK-DIRECTORY>/log/flink-<REST_OF_THE_FILE_NAME>.logattributes:logtype: flink_logs
Before you can start reading your data, restart your infrastructure agent.
If you want to use our pre-built dashboard template named Apache Flink to monitor your Apache Flink metrics, follow these steps:
- Go to one.newrelic.com and click on + Add Data.
- Click on Dashboards tab.
- In the search box, search for Apache Flink.
- Click on the Apache Flink dashboard to install it in your account.
Once your application is integrated by following the above steps, the dashboard should reflect metrics on it.
To instrument the Apache Flink quickstart and to see metrics and alerts, you can also follow our Apache Flink quickstart page by clicking on the Install now button.
You can use NRQL to query your data. For example, if you want to view the total number of completed checkpoints on New Relic's Query Builder, use this NRQL query:
FROM Metric SELECT latest(flink_jobmanager_job_numberOfCompletedCheckpoints) AS 'Number of Completed Checkpoints'
If you want to further customize your Apache Flink dashboards, you can learn more about building NRQL queries and managing your in the New Relic UI: