Preparing for a peak demand event requires time: you have to establish your baselines, assess your service level agreements, and align all your teams around the same data. Scaling your infrastructure has similar considerations, but also involves projecting system needs against cost. How do you balance the two without sacrificing performance? Where is the threshold for good enough?
One solution is horizontal pod autoscaling (HPA), which is built with our Kubernetes integration with Pixie. Once you've set up HPA, your Kubernetes cluster automatically allocates more pods when demand peaks, then deallocates when demand falls. This allows you to sidestep cost concerns when anticipating demand.
The tutorial assumes that you're using Kubernetes clusters. To set up HPA, you'll need:
- Kubernetes 1.16 or higher
- Your New Relic user
- No other external metrics adapter installed in the cluster
This tutorial walks you through setting up a demo environment for HPA. You will:
- Install the New Relic Kubernetes integration
- Set up an example environment to test autoscaling
Forward metrics to Kubernetes
Clone our New Relic Pixie lab repo
Clone the following repo from Github:
git clone https://github.com/newrelic-experimental/pixie-lab-materialscd pixie-lab-materials/main./setup.sh
setup.sh script spins up a new minikube cluster using the Pixie-supported hyperkit driver. It then configures your network memory and CPU for optimal performance with Pixie and creates all the pods and services that make up the demo application.
In a new terminal window, open a minikube tunnel:
minikube tunnel -p minikube-pixie-lab
You should have two terminals:
- Your tunnel, which remains open to access your demo application.
- A place to run commands for the tutorial.
Install the Kubernetes integration with Pixie
Follow our guided install to install the New Relic Kubernetes integration. This connects New Relic to your Kubernetes cluster. Make sure to:
Check Instant service-level insights, Full-body requests, and Application profiles through Pixie to enable Pixie.
Keep all other default checked items
Once you click continue, copy and paste that command into your dev environment.
Install New Relic Metrics Adapter
To install the New Relic Metrics Adapter, use the
newrelic-k8s-metrics-adapter Helm chart. If you've used the
nri-bundle-chart to deploy any New Relic Kubernetes components, then you have access to this Helm chart.
helm upgrade --install newrelic newrelic/nri-bundle \--namespace newrelic --create-namespace --reuse-values \--set metrics-adapter.enabled=true \--set newrelic-k8s-metrics-adapter.personalAPIKey=YOUR_NEW_RELIC_PERSONAL_API_KEY \--set newrelic-k8s-metrics-adapter.config.accountID=YOUR_NEW_RELIC_ACCOUNT_ID \--set newrelic-k8s-metrics-adapter.config.externalMetrics.manipulate_average_requests.query='FROM Metric SELECT average(http.server.duration) WHERE instrumentation.provider='pixie''
Here's what these flags do:
metrics-adapter.enabled: Sets to
trueto install the metrics adapter chart
newrelic-k8s-metrics-adapter.personalAPIKey: Sets your New Relic API key.
newrelic-k8s-metrics-adapter.accountID: The ID of the account that forwards metrics.
newrelic-k8s-metrics-adapter.config.externalMetrics.external_metric_name.query: Adds a new external metric with the following information:
external_metric_name: The metric name.
query: The base NRQL query for the metric.
Confirm your NRQL query is accurate
You should test your query before sending metrics to the autoscaler from New Relic. Go to one.newrelic.com > Query your Data, then copy and paste the following NRQL query:
FROM Metric SELECT average(http.server.duration) WHERE instrumentation.provider='pixie'
Configure your autoscaler
pixie-lab-materials/main/kube directory, create a new file called
hpa.yml. The New Relic metrics adapter sends data to the controller manager, which is defined by the HPA definition in this YAML file.
kind: HorizontalPodAutoscalerapiVersion: autoscaling/v2beta2metadata:name: manipulate-scalerspec:scaleTargetRef:apiVersion: apps/v1kind: Deploymentname: manipulation-serviceminReplicas: 1maxReplicas: 10metrics:- type: Externalexternal:metric:name: manipulate_average_requeststarget:type: Valuevalue: 100
Apply the new YAML file by running:
cd pixie-lab-materials/main/kubekubectl apply -f hpa.yaml
Check your work: Add load to trigger autoscaling
- Navigate to your site deployment with
kubectl get services.
- Open the
EXTERNAL-IPfrom your frontend-service in your browser.
heyand Go v1.17 with
brew install hey.
- Send GET requests to the
hey -n 10 -c 2 -m GET http://<EXTERNAL-IP>.
- Watch your HPA autoscalling with
watch kubnectl get hpa.
You've successfully set up HPA if the pod autoscales the number of replicas as the average HTTP request time increases. You can adjust the configuration for your own services so that New Relic and HPA automatically autoscale as needed.
Get data about your architecture with APM and infrastructure agents
2Create service levels for gameday
Create service levels informed by your baseline