LLM observability with OpenLIT

While OpenTelemetry offers a powerful standard for collecting general application data (traces, metrics, logs), it lacks the ability to capture key performance indicators (KPIs) specific to AI models. Essential metrics like performance tracking, token usage, costs, and user interaction data in the form of LLM prompts and responses are crucial for effective LLM application monitoring and troubleshooting.

To bridge this gap in LLM observability, OpenLIT has emerged as a tailored solution. Built on the OpenTelemetry framework, OpenLIT is an open-source tool that integrates smoothly and extends its capabilities. It supports widely-used AI frameworks such as OpenAI, Anthropic, Pinecone, LangChain, among others. Additionally, it provides OpenTelemetry-based GPU monitoring capabilities out of the box.

Key benefits of OpenLIT:

• Advanced monitoring of LLM and VectorDB performance: OpenLIT automatically generates traces and metrics for in-depth performance and cost analysis of LLM and VectorDB usage, helping you optimize resource use and scale efficiently across various environments, such as production.

• Cost tracking for custom and fine-tuned models: Allows for precise cost tracking through a custom JSON file, enabling accurate budgeting, and alignment with project-specific needs.

• OpenTelemetry-native and vendor-neutral SDKs: OpenLIT is built with native support for OpenTelemetry, making it blend seamlessly with your projects. This vendor-neutral approach reduces barriers to integration, making OpenLIT an intuitive part of your software stack rather than an additional complexity.

OpenLIT empowers developers to leverage the strengths of OpenTelemetry while gaining the additional functionalities required for effective LLM monitoring and performance optimization.

Before you start

• Sign up for a New Relic account.
• Get the license key for the New Relic account to which you want to report data.

Instrument your LLM Model with OpenLIT

Follow these common setup steps for OpenTelemetry-based APM monitoring with New Relic.

OTEL_EXPORTER_OTLP_ENDPOINT = https://otlp.nr-data.net:443
OTEL_EXPORTER_OTLP_HEADERS = "api-key=YOUR_NEWRELIC_LICENSE_KEY"

import openlit
import os
from langchain_openai import ChatOpenAI

os.environ['OPENAI_API_KEY'] = 'OPENAI_API_KEY'
os.environ['OTEL_EXPORTER_OTLP_ENDPOINT'] = 'https://otlp.nr-data.net:443'
os.environ['OTEL_EXPORTER_OTLP_HEADERS'] = 'api-key=YOUR_NEWRELIC_LICENSE_KEY'

openlit.init()

def add_prompt_context():
  llm = ChatOpenAI(
      model="gpt-3.5-turbo",
      temperature=0)
  chain = llm
  return chain

def prep_prompt_chain():
  return add_prompt_context()

def prompt_question():
  chain = prep_prompt_chain()
  return chain.invoke("explain the business of company Newrelic and it's advantages in a max of 50 words")

if  __name__ == "__main__":
  print(prompt_question())