Add OpenTelemetry instrumentation for LiteLLM by whoIam0987 · Pull Request #88 · alibaba/loongsuite-python-agent

whoIam0987 · 2025-12-15T08:41:47Z

Description

Instrument Litellm with genai util.
Fixes # (issue)

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Modify releated unit tests.

Does This PR Require a Core Repo Change?

No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added
Documentation has been updated

CLAassistant · 2025-12-15T08:41:56Z

All committers have signed the CLA.

Copilot

Pull request overview

This pull request adds OpenTelemetry instrumentation for the LiteLLM library, which provides a unified interface to 100+ LLM providers. The instrumentation captures telemetry data for LLM operations including completions, embeddings, streaming, tool calls, and retry mechanisms.

Changes:

Added new instrumentation package for LiteLLM with support for sync/async completion and embedding APIs
Implemented streaming response wrappers with proper span lifecycle management
Added comprehensive test suite covering various LiteLLM features including tool calls, retries, and error handling

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
`__init__.py`	Main instrumentor class that wraps LiteLLM functions and manages telemetry handlers
`_wrapper.py`	Completion wrappers for sync/async calls with streaming support
`_embedding_wrapper.py`	Embedding operation wrappers for sync/async calls
`_stream_wrapper.py`	Stream response wrappers handling chunk accumulation and finalization
`_utils.py`	Utility functions for message conversion, provider parsing, and invocation creation
`version.py`, `package.py`	Package metadata and dependency declarations
`pyproject.toml`	Package configuration with build system and dependencies
`README.rst`	Documentation for installation, configuration, and usage
`test_*.py`	Comprehensive test suite covering completions, embeddings, streaming, tools, retries, and errors
`test-requirements.txt`	Test dependency specifications

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-05T13:55:37Z

...uite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/_embedding_wrapper.py

+
+        # Create invocation object


Duplicate comment. The comment "Create invocation object" appears twice consecutively on lines 63 and 65. Remove one of these duplicate lines.

Suggested change

# Create invocation object

Copilot · 2026-02-05T13:55:38Z

...uite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/_embedding_wrapper.py

+
+        # Create invocation object


Duplicate comment. The comment "Create invocation object" appears twice consecutively on lines 164 and 166. Remove one of these duplicate lines.

Suggested change

# Create invocation object

Copilot · 2026-02-05T13:55:38Z

...ite/loongsuite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/_wrapper.py

+
+        # For streaming, we need special handling


Duplicate comment. The comment "For streaming, we need special handling" appears twice consecutively on lines 85 and 87. Remove one of these duplicate lines.

Suggested change

# For streaming, we need special handling

Copilot · 2026-02-05T13:55:38Z

...ite/loongsuite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/_wrapper.py

+                stream_wrapper = AsyncStreamWrapper(
+                    stream=response,
+                    span=invocation.span,  # For TTFT tracking
+                    callback=lambda span,
+                    last_chunk,
+                    error: self._handle_stream_end_with_handler(
+                        invocation, last_chunk, error, stream_wrapper
+                    ),
+                )


Potential circular reference in lambda callback. The lambda function references stream_wrapper (line 366) before it's assigned (line 360). While Python closures can handle this due to lazy evaluation, this creates a circular reference that could potentially cause issues with garbage collection. Consider restructuring to avoid the circular reference, perhaps by creating the callback after the AsyncStreamWrapper is instantiated or using a different callback pattern.

Copilot · 2026-02-05T13:55:39Z

...ite/loongsuite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/__init__.py

+* ``ENABLE_LITELLM_INSTRUMENTOR``: Enable/disable instrumentation (default: true)
+* ``ARMS_LITELLM_INSTRUMENTATION_ENABLED``: Alternative enable/disable flag (default: true)


The documentation claims that the ENABLE_LITELLM_INSTRUMENTOR environment variable can be used to enable/disable instrumentation, but this variable is never referenced in the actual code. Only ARMS_LITELLM_INSTRUMENTATION_ENABLED is actually implemented. Either implement support for this environment variable or remove it from the documentation to avoid confusion.

Suggested change

* ``ENABLE_LITELLM_INSTRUMENTOR``: Enable/disable instrumentation (default: true)

* ``ARMS_LITELLM_INSTRUMENTATION_ENABLED``: Alternative enable/disable flag (default: true)

* ``ARMS_LITELLM_INSTRUMENTATION_ENABLED``: Enable/disable instrumentation (default: true)

Copilot · 2026-02-05T13:55:41Z

...ite/loongsuite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/_wrapper.py

+                suppress_token = context.attach(
+                    context.set_value(SUPPRESS_LLM_SDK_KEY, True)
+                )
+            except Exception:


'except' clause does nothing but pass and there is no explanatory comment.

Copilot · 2026-02-05T13:55:41Z

...ite/loongsuite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/_wrapper.py

+                                except Exception:
+                                    pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

pass

except Exception as decode_error:

# Ignore JSON parsing errors and fall back to the raw string,

# but log at debug level for diagnosability.

logger.debug(

"Failed to JSON-decode tool call arguments %r: %s",

arguments,

decode_error,

)

Copilot · 2026-02-05T13:55:42Z

...ite/loongsuite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/_wrapper.py

+            except Exception:
+                pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

pass

except Exception as handler_error:

# Swallow exceptions from telemetry failure reporting, but log them for diagnostics.

logger.debug(

"Error while reporting LLM failure in _handle_stream_end_with_handler: %s",

handler_error,

)

Copilot · 2026-02-05T13:55:42Z

...ite/loongsuite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/_wrapper.py

+                suppress_token = context.attach(
+                    context.set_value(SUPPRESS_LLM_SDK_KEY, True)
+                )
+            except Exception:


'except' clause does nothing but pass and there is no explanatory comment.

Copilot · 2026-02-05T13:55:42Z

...ite/loongsuite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/_wrapper.py

+                    context.set_value(SUPPRESS_LLM_SDK_KEY, True)
+                )
+            except Exception:
+                pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

pass

# Failed to attach suppression context; proceed without suppression.

logger.exception(

"Failed to attach suppression context for LiteLLM instrumentation"

)

github-actions bot assigned 123liuziming, Cirilla-zmh and ralf0131 Dec 15, 2025

github-actions bot requested review from 123liuziming, Cirilla-zmh and ralf0131 December 15, 2025 08:57

whoIam0987 added 2 commits January 27, 2026 17:13

Add OpenTelemetry instrumentation for litellm

d4be2d1

Add Litellm Instrument Unit Test

d8affed

whoIam0987 force-pushed the mingzhi/litellm branch from c26059b to d8affed Compare January 28, 2026 07:30

whoIam0987 changed the title ~~[WIP] Add OpenTelemetry instrumentation for LiteLLM~~ Add OpenTelemetry instrumentation for LiteLLM Feb 5, 2026

ralf0131 requested a review from Copilot February 5, 2026 13:50

Copilot started reviewing on behalf of ralf0131 February 5, 2026 13:50 View session

Copilot AI reviewed Feb 5, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add OpenTelemetry instrumentation for LiteLLM#88

Add OpenTelemetry instrumentation for LiteLLM#88
whoIam0987 wants to merge 2 commits intoalibaba:mainfrom
whoIam0987:mingzhi/litellm

whoIam0987 commented Dec 15, 2025 •

edited

Loading

Uh oh!

CLAassistant commented Dec 15, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Copilot AI Feb 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		* ``ENABLE_LITELLM_INSTRUMENTOR``: Enable/disable instrumentation (default: true)
		* ``ARMS_LITELLM_INSTRUMENTATION_ENABLED``: Alternative enable/disable flag (default: true)

-                                except Exception:
-                                    pass
+                                except Exception as decode_error:
+                                    # Ignore JSON parsing errors and fall back to the raw string,
+                                    # but log at debug level for diagnosability.
+                                    logger.debug(
+                                        "Failed to JSON-decode tool call arguments %r: %s",
+                                        arguments,
+                                        decode_error,
+                                    )

-            except Exception:
-                pass
+            except Exception as handler_error:
+                # Swallow exceptions from telemetry failure reporting, but log them for diagnostics.
+                logger.debug(
+                    "Error while reporting LLM failure in _handle_stream_end_with_handler: %s",
+                    handler_error,
+                )

-                pass
+                # Failed to attach suppression context; proceed without suppression.
+                logger.exception(
+                    "Failed to attach suppression context for LiteLLM instrumentation"
+                )

Conversation

whoIam0987 commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How Has This Been Tested?

Does This PR Require a Core Repo Change?

Checklist:

Uh oh!

CLAassistant commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

whoIam0987 commented Dec 15, 2025 •

edited

Loading

CLAassistant commented Dec 15, 2025 •

edited

Loading