์‚ฌ์šฉ์ž์˜ ํŽธ์˜๋ฅผ ์œ„ํ•ด ์ œ๊ณต๋˜๋Š” ๊ธฐ๊ณ„ ๋ฒˆ์—ญ์ž…๋‹ˆ๋‹ค.

์˜๋ฌธ๋ณธ๊ณผ ๋ฒˆ์—ญ๋ณธ์ด ์ผ์น˜ํ•˜์ง€ ์•Š๋Š” ๊ฒฝ์šฐ ์˜๋ฌธ๋ณธ์ด ์šฐ์„ ํ•ฉ๋‹ˆ๋‹ค. ๋ณด๋‹ค ์ž์„ธํ•œ ๋‚ด์šฉ์€ ์ด ํŽ˜์ด์ง€๋ฅผ ๋ฐฉ๋ฌธํ•˜์‹œ๊ธฐ ๋ฐ”๋ž๋‹ˆ๋‹ค.

๋ฌธ์ œ ์‹ ๊ณ 

Kubernetes์šฉ OpenTelemetry

์‹œ์‚ฌ

์ด ๊ธฐ๋Šฅ์€ ์•„์ง ๊ฐœ๋ฐœ ์ค‘์ด์ง€๋งŒ ๊ผญ ์‚ฌ์šฉํ•ด ๋ณด์‹œ๊ธฐ ๋ฐ”๋ž๋‹ˆ๋‹ค!

์ด ๊ธฐ๋Šฅ์€ ํ˜„์žฌ ์ถœ์‹œ ์ „ ์ •์ฑ… ์— ๋”ฐ๋ผ ๋ฏธ๋ฆฌ๋ณด๊ธฐ ํ”„๋กœ๊ทธ๋žจ์˜ ์ผ๋ถ€๋กœ ์ œ๊ณต๋ฉ๋‹ˆ๋‹ค.

OpenTelemetry ์šฉ ์˜ต์ €๋ฒ„๋นŒ๋ฆฌํ‹ฐ๋Š” ๋…์ ์ ์ธ Kubernetes ๋ฆฌ์†Œ์Šค์™€ Kubernetes UI ์ด๋ฏธ ํ˜ธํ™˜๋˜๋Š” ์ตœ๊ณ  ์ˆ˜์ค€์˜ ์™€ ๊ฒฐํ•ฉ๋œ ์™„์ „ํ•œ ์˜คํ”ˆ ์†Œ์Šค ์„ค์ •์„ Kubernetes ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. Kubernetes UI๋Š” ๊ณต๊ธ‰์ž์— ๊ด€๊ณ„์—†์ด ์„ค๊ณ„๋˜์–ด ์‚ฌ์šฉ์ž์˜ ์š”๊ตฌ ์‚ฌํ•ญ์— ๋”ฐ๋ผ OpenTelemetry ๋˜๋Š” NewRulestics๋ฅผ ์„ ํƒํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

์ด ๋ฌธ์„œ์—์„œ๋Š” OpenTelemetry ์‚ฌ์šฉํ•˜์—ฌ ์ฟ ๋ฒ„๋„คํ‹ฐ์Šค ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ๋ชจ๋‹ˆํ„ฐ๋งํ•˜๋Š” ํ”„๋กœ์„ธ์Šค๋ฅผ ๊ฐ„๋žตํ•˜๊ฒŒ ์„ค๋ช…ํ•ฉ๋‹ˆ๋‹ค. ํด๋Ÿฌ์Šคํ„ฐ ๋‚ด์˜ nr-k8s-otel-collector Helm ์ฐจํŠธ ์„ค์น˜์™€ ์ผ๋ฅ˜ ์˜ต์ €๋ฒ„๋นŒ๋ฆฌํ‹ฐ๋ฅผ ํ™œ์„ฑํ™”ํ•˜๋Š” ๋ฐ ํ•„์š”ํ•œ ์ˆ˜์ง‘๊ธฐ ๋ฐฐํฌ๊ฐ€ ํฌํ•จ๋ฉ๋‹ˆ๋‹ค.

Kubernetes ๊ตฌ์„ฑ ์š”์†Œ๋ฅผ OpenTelemetry ์ˆ˜์ง‘๊ธฐ์— ํ†ตํ•ฉํ•˜์—ฌ ์ง€ํ‘œ, ์ด๋ฒคํŠธ, ๋กœ๊ทธ๋ฅผ ๋‰ด๋ ๋ฆญ์— ์ง์ ‘ ์ „์†กํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋Ÿฌํ•œ ํ…”๋ ˆ๋ฉ”ํŠธ๋ฆฌ ์‹ ํ˜ธ๋Š” Kubernetes Navigator, ๊ฐœ์š” ๋Œ€์‹œ๋ณด๋“œ, Kubernetes ์ด๋ฒคํŠธ ๋˜๋Š” Kubernetes APM ์š”์•ฝ ํŽ˜์ด์ง€ ์™€ ๊ฐ™์€ ๊ธฐ๋ณธ ์ œ๊ณต ํ™˜๊ฒฝ์„ ์ž๋™์œผ๋กœ ํ–ฅ์ƒ์‹œํ‚ต๋‹ˆ๋‹ค.

์–ด๋–ป๊ฒŒ ์ž‘๋™ํ•˜๋‚˜์š”?

nr-k8s-otel-collector Helm ์ฐจํŠธ๋ฅผ ๊ตฌํ˜„ํ•˜๋‹ค, ๋ฐฐํฌํ•˜๋‹ค ์ด๋Ÿฌํ•œ OpenTelemetry ์ˆ˜์ง‘๊ธฐ:

  • Deamonset Collector: ๊ตฌํ˜„ํ•˜๋‹ค, ๋ฐฐํฌํ•˜๋‹ค ๊ฐ ์ž‘์—…์ž ๋…ธ๋“œ์— ์žˆ์œผ๋ฉฐ ๋…ธ๋“œ์˜ ๊ธฐ๋ณธ ํ˜ธ์ŠคํŠธ์ธ cAdvisor, Kubelet ์—์„œ ์ง€ํ‘œ๋ฅผ ์ˆ˜์ง‘ํ•˜๊ณ  ์ปจํ…Œ์ด๋„ˆ์—์„œ ๋กœ๊ทธ๋ฅผ ์ˆ˜์ง‘ํ•˜๋Š” ์ผ์„ ๋‹ด๋‹นํ•ฉ๋‹ˆ๋‹ค.

  • ๋ฐฐํฌ ์ˆ˜์ง‘๊ธฐ: ๊ตฌํ˜„ํ•˜๋‹ค, ๋ฐฐํฌํ•˜๋‹ค ์ปจํŠธ๋กค ํ”Œ๋ ˆ์ธ ๋…ธ๋“œ์— ์žˆ์œผ๋ฉฐ Kube ์ƒํƒœ ์ง€ํ‘œ ๋ฐ ์ฟ ๋ฒ„๋„คํ‹ฐ์Šค ํด๋Ÿฌ์Šคํ„ฐ ์ด๋ฒคํŠธ์˜ ์ง€ํ‘œ๋ฅผ ์ˆ˜์ง‘ํ•˜๋Š” ์—ญํ• ์„ ๋‹ด๋‹นํ•ฉ๋‹ˆ๋‹ค.

    K8s OpenTelemetry diagram

์š”๊ตฌ ์‚ฌํ•ญ

Kubernetes ํ…”๋ฆฌ๋ฉ”ํŠธ๋ฆฌ ๋ฐ์ดํ„ฐ๋ฅผ ๋‰ด๋ ๋ฆญ์œผ๋กœ ๋ณด๋‚ด๋ ค๋ฉด OpenTelemetry ์ˆ˜์ง‘๊ธฐ๊ฐ€ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค. ์šฐ๋ฆฌ์˜ OpenTelemetry (NRDOT) ๋ฐฐํฌํŒ์€ ์ด๋ฏธ ์ฟ ๋ฒ„๋„คํ‹ฐ์Šค ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ์ž๋™์œผ๋กœ ๋ชจ๋‹ˆํ„ฐ๋งํ•˜๋„๋ก ์„ค์ •๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค. nr-k8s-otel-collector Helm ์ฐจํŠธ๋ฅผ ํ†ตํ•ด ๋ชจ๋“  ํ•„์ˆ˜ ๊ตฌ์„ฑ ์š”์†Œ๋ฅผ ๊ตฌํ˜„, ๋ฐฐํฌํ•˜์—ฌ ์ด๋ฅผ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค.

๋‹ค๋ฅธ OpenTelemetry ์ˆ˜์ง‘๊ธฐ๋กœ ์ „ํ™˜ํ•˜๋Š” ๊ฒฝ์šฐ ํ•„์š”ํ•œ ๋ชจ๋“  ์ฃผ์š” ๊ตฌ์„ฑ ์š”์†Œ๊ฐ€ ์žˆ๋Š”์ง€ ํ™•์ธํ•˜์„ธ์š”.

ํŒ

Kubernetes OpenTelemetry ์˜ต์ €๋ฒ„๋นŒ๋ฆฌํ‹ฐ๋ฅผ ์‚ฌ์šฉํ•˜๊ณ  ์‹ถ์ง€๋งŒ Helm ์ฐจํŠธ๋ฅผ ์„ค์น˜ํ•˜๊ณ  ์‹ถ์ง€ ์•Š์œผ์‹ ๊ฐ€์š”?

  1. ์ˆ˜์ง‘๊ธฐ์— ์ด์ „์— ๊ฐ•์กฐ ํ‘œ์‹œ๋œ ๊ตฌ์„ฑ ์š”์†Œ๊ฐ€ ํฌํ•จ๋˜์–ด ์žˆ๋Š”์ง€ ํ™•์ธํ•˜์„ธ์š”.
  2. ์ด ๋ฌธ์„œ ์— ์ œ๊ณต๋œ ์„ค์ • ์ง€์นจ์— ๋”ฐ๋ผ ์ˆ˜์ง‘๊ธฐ๋ฅผ ์ ์ ˆํžˆ ๊ตฌ์„ฑํ•˜์„ธ์š”.

OpenTelemetry๋กœ ์ฟ ๋ฒ„๋„คํ‹ฐ์Šค ํด๋Ÿฌ์Šคํ„ฐ๋ฅผ ์„ค์น˜ํ•˜์„ธ์š”

ํด๋Ÿฌ์Šคํ„ฐ์—์„œ OpenTelemetry๋ฅผ ์‹œ์ž‘ํ•˜๊ณ  ์‹คํ–‰ํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋‹จ๊ณ„๋ฅผ ๋”ฐ๋ฅด์„ธ์š”.

  1. Helm ์ฐจํŠธ ๊ฐ’ ํŒŒ์ผ์„ ๋‹ค์šด๋กœ๋“œํ•˜์—ฌ ํŠน์ • ์š”๊ตฌ ์‚ฌํ•ญ์— ๋งž๊ฒŒ ์กฐ์ •ํ•˜์„ธ์š”.

    • ํด๋Ÿฌ์Šคํ„ฐ ์ด๋ฆ„๊ณผ ํ•„์ˆ˜์ž…๋‹ˆ๋‹ค.

    • ์„ค์ • ์‹œ์œ„ ์˜ ์ „์ฒด ๋ชฉ๋ก์„ ํ™•์ธํ•˜์„ธ์š”.

  2. ๊ฐ’ ํŒŒ์ผ๊ณผ ํ•จ๊ป˜ Helm ์ฐจํŠธ๋ฅผ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค.

    bash
    $
    helm repo add newrelic https://helm-charts.newrelic.com
    $
    helm upgrade nr-k8s-otel-collector newrelic/nr-k8s-otel-collector -f your-custom-values.yaml -n newrelic --create-namespace --install
  3. ํŒจ๋“œ๊ฐ€ ์„ฑ๊ณต์ ์œผ๋กœ ํšŒ์ „๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•˜์‹ญ์‹œ์˜ค.

    bash
    $
    kubectl get pods -n newrelic --watch
  4. ์˜ฌ๋ฐ”๋ฅธ ์ฟผ๋ฆฌ๋ฅผ ์‹คํ–‰ํ•˜์—ฌ ๋‰ด๋ ๋ฆญ์ด ๋ฉ”ํŠธ๋ฆญ, ์ด๋ฒคํŠธ, ๋กœ๊ทธ ๋“ฑ ํ•„์š”ํ•œ ๋ฐ์ดํ„ฐ๋ฅผ ๊ฐ€์ ธ์˜ค๊ณ  ์žˆ๋Š”์ง€ ํ™•์ธํ•˜์„ธ์š”. ์ž์„ธํ•œ ๋‚ด์šฉ์€ ์ฟผ๋ฆฌ ๋นŒ๋” ์†Œ๊ฐœ๋ฅผ ์ฐธ์กฐํ•˜์„ธ์š”.

    FROM Metric SELECT * WHERE k8s.cluster.name='<CLUSTER_NAME>'
    FROM InfrastructureEvent SELECT * WHERE k8s.cluster.name='<CLUSTER_NAME>'
    FROM Log SELECT * WHERE k8s.cluster.name='<CLUSTER_NAME>'
  5. GKE AutoPilot์„ ์‚ฌ์šฉํ•˜๋Š” ๊ฒฝ์šฐ OpenTelemetry ์ˆ˜์ง‘๊ธฐ์˜ ํ˜ธํ™˜์„ฑ๊ณผ ์ ์ ˆํ•œ ๊ธฐ๋Šฅ์„ ๋ณด์žฅํ•˜๊ธฐ ์œ„ํ•ด values.yaml ํŒŒ์ผ์— ๋‹ค์Œ ์„ค์ •์„ ์ ์šฉํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

    privileged: false
    receivers:
    filelog:
    enabled: false
    daemonset:
    containerSecurityContext:
    privileged: false

OpenTelemetry๋กœ ์ฟ ๋ฒ„๋„คํ‹ฐ์Šค ํด๋Ÿฌ์Šคํ„ฐ ์ œ๊ฑฐ

OpenTelemetry ๋กœ ์ฟ ๋ฒ„๋„คํ‹ฐ์Šค ํด๋Ÿฌ์Šคํ„ฐ ๋ชจ๋‹ˆํ„ฐ๋ง์„ ์ค‘์ง€ํ•˜๋ ค๋ฉด ๋‹ค์Œ ๋ช…๋ น์„ ์‹คํ–‰ํ•˜์„ธ์š”.

bash
$
helm uninstall nr-k8s-otel-collector -n newrelic

๋ฐ์ดํ„ฐ ์ˆ˜์ง‘ ์ค„์ด๊ธฐ

LowDataMode ์˜ต์…˜์€ ๊ธฐ๋ณธ์ ์œผ๋กœ Kubernetes UI์— ํ•„์š”ํ•œ ์ง€ํ‘œ๋งŒ ์ˆ˜์ง‘ํ•˜๋„๋ก ํ™œ์„ฑํ™”๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค.

๋ฐ์ดํ„ฐ ์ˆ˜์ง‘์„ ๋” ์ค„์—ฌ์•ผ ํ•˜๋Š” ๊ฒฝ์šฐ ๊ฐ ๊ตฌํ˜„ํ•˜๋‹ค, ๋ฐฐํฌํ•˜๋‹ค ๊ตฌ์„ฑ ์š”์†Œ์— ๋Œ€ํ•œ nr-k8s-otel-collector ์ฐจํŠธ ๊ฐ’ ์—์„œ ์Šคํฌ๋ ˆ์ดํ•‘ ๊ฐ„๊ฒฉ์„ ๋Š˜๋ฆฝ๋‹ˆ๋‹ค.

๋ฉ”ํŠธ๋ฆญ

๋ฐ์ดํ„ฐ ์ฐพ๊ธฐ ๋ฐ ์‚ฌ์šฉ

๋ฐ์ดํ„ฐ๋ฅผ ์ฐพ๋Š” ๋ฐฉ๋ฒ•์— ๋Œ€ํ•ด ์ž์„ธํžˆ ์•Œ์•„๋ณด๋ ค๋ฉด ๋‹ค์Œ ๋ฌธ์„œ๋ฅผ ํ™•์ธํ•˜์„ธ์š”.

๋ฌธ์ œ์  ํ•ด๊ฒฐ

๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ•œ ์ˆ˜์ง‘๊ธฐ ํŒจ๋“œ์˜ ๋กœ๊ทธ๋ฅผ ํ™•์ธํ•ด ๋ณด์„ธ์š”. ๋‹ค์Œ ๋ช…๋ น์„ ์‹คํ–‰ํ•˜์„ธ์š”:

bash
$
kubectl logs <otel-pod-name> -n newrelic

nr-k8s-otel-collector Helm ์ฐจํŠธ์—์„œ verboseLog ๋ณด๊ณ ์„œ๋ฅผ true ๋กœ ์„ค์ •ํ•  ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.

์ผ๋ฐ˜์ ์ธ ์˜ค๋ฅ˜

GitHub ์ €์žฅ์†Œ์˜ ์ผ๋ฐ˜์ ์ธ ์˜ค๋ฅ˜ ์„น์…˜์„ ํ™•์ธํ•˜์„ธ์š”.

์ง€์›ํ•˜๋‹ค

์šฉ OpenTelemetry ์˜ต์ €๋ฒ„๋นŒ๋ฆฌํ‹ฐ์— ๋ฌธ์ œ๊ฐ€ ์žˆ๋Š” Kubernetes ๊ฒฝ์šฐ:

  • ๋น„์Šทํ•œ ๋ฌธ์ œ๊ฐ€ ์žˆ๋Š”์ง€ GitHub์˜ ๋ฌธ์ œ ์„น์…˜์„ ์‚ดํŽด๋ณด๊ฑฐ๋‚˜ ์ƒˆ ๋ฌธ์ œ๋ฅผ ์—ฌ๋Š” ๊ฒƒ์„ ๊ณ ๋ คํ•ด ๋ณด์„ธ์š”.