Core API

ldai.client

class ldai.client.LDAIClient(client: LDClient)

The LaunchDarkly AI SDK client object.

__init__(client: LDClient)

agent_config(key: str, context: Context, default: AIAgentConfigDefault | None = None, variables: Dict[str, Any] | None = None) → AIAgentConfig

Retrieve a single AI Config agent.

This method retrieves a single agent configuration with instructions dynamically interpolated using the provided variables and context data.

Example:

agent = client.agent_config(
    'research_agent',
    context,
    AIAgentConfigDefault(
        enabled=True,
        model=ModelConfig('gpt-4'),
        instructions="You are a research assistant specializing in {{topic}}."
    ),
    variables={'topic': 'climate change'}
)

if agent.enabled:
    research_result = agent.instructions  # Interpolated instructions
    tracker = agent.create_tracker()
    tracker.track_success()

Parameters:

• key – The agent configuration key.

• context – The context to evaluate the agent configuration in.

• default – Default agent values. When not provided, a disabled config is used as the fallback.

• variables – Variables for interpolation.

Returns:

Configured AIAgentConfig instance.

agent_configs(agent_configs: List[AIAgentConfigRequest], context: Context) → Dict[str, AIAgentConfig]

Retrieve multiple AI agent configurations.

This method allows you to retrieve multiple agent configurations in a single call, with each agent having its own default configuration and variables for instruction interpolation.

Example:

agents = client.agent_configs([
    AIAgentConfigRequest(
        key='research_agent',
        default=AIAgentConfigDefault(
            enabled=True,
            instructions='You are a research assistant.'
        ),
        variables={'topic': 'climate change'}
    ),
    AIAgentConfigRequest(
        key='writing_agent',
        default=AIAgentConfigDefault(
            enabled=True,
            instructions='You are a writing assistant.'
        ),
        variables={'style': 'academic'}
    )
], context)

research_result = agents["research_agent"].instructions
tracker = agents["research_agent"].create_tracker()
tracker.track_success()

Parameters:

• agent_configs – List of agent configurations to retrieve.

• context – The context to evaluate the agent configurations in.

Returns:

Dictionary mapping agent keys to their AIAgentConfig configurations.

agent_graph(key: str, context: Context, default_ai_provider: str | None = None) → AgentGraphDefinition

` Retrieve an AI agent graph.

completion_config(key: str, context: Context, default: AICompletionConfigDefault | None = None, variables: Dict[str, Any] | None = None, default_ai_provider: str | None = None) → AICompletionConfig

Get the value of a completion configuration.

Parameters:

• key – The key of the completion configuration.

• context – The context to evaluate the completion configuration in.

• default – The default value of the completion configuration. When not provided, a disabled config is used as the fallback.

• variables – Additional variables for the completion configuration.

• default_ai_provider – Optional default AI provider to use for judge evaluation.

Returns:

The completion configuration with a tracker used for gathering metrics.

create_agent(key: str, context: Context, tools: Dict[str, Callable] | None = None, default: AIAgentConfigDefault | None = None, variables: Dict[str, Any] | None = None, default_ai_provider: str | None = None) → ManagedAgent | None

CAUTION: This feature is experimental and should NOT be considered ready for production use. It may change or be removed without notice and is not subject to backwards compatibility guarantees.

Creates and returns a new ManagedAgent for AI agent invocations.

Parameters:

• key – The key identifying the AI agent configuration to use

• context – Standard Context used when evaluating flags

• tools – ToolRegistry mapping tool names to callable implementations

• default – A default value representing a standard AI agent config result. When not provided, a disabled config is used as the fallback.

• variables – Dictionary of values for instruction interpolation

• default_ai_provider – Optional default AI provider to use

Returns:

ManagedAgent instance or None if disabled/unsupported

Example:

agent = client.create_agent(
    "customer-support-agent",
    context,
    tools={"get-order": fetch_order_fn},
    default=AIAgentConfigDefault(
        enabled=True,
        model=ModelConfig("gpt-4"),
        provider=ProviderConfig("openai"),
        instructions="You are a helpful customer support agent."
    ),
)

if agent:
    result = await agent.run("Where is my order?")
    print(result.output)

create_agent_graph(key: str, context: Context, tools: Dict[str, Callable] | None = None, default_ai_provider: str | None = None) → ManagedAgentGraph | None

CAUTION: This feature is experimental and should NOT be considered ready for production use. It may change or be removed without notice and is not subject to backwards compatibility guarantees.

Creates and returns a new ManagedAgentGraph for AI agent graph execution.

Resolves the graph configuration via agent_graph(), creates a provider-specific runner, and wraps it in a ManagedAgentGraph.

Parameters:

• key – The key identifying the agent graph configuration

• context – Standard Context used when evaluating flags

• tools – Registry mapping tool names to callables

• default_ai_provider – Optional provider override (‘openai’, ‘langchain’, …)

Returns:

ManagedAgentGraph instance, or None if the graph is disabled or unsupported

Example:

graph = client.create_agent_graph(
    "travel-assistant-graph",
    context,
    tools={
        "web_search_tool": my_search_fn,
        "get_weather": my_weather_fn,
    }
)

if graph:
    result = await graph.run("Find me restaurants in Seattle")
    print(result.output)

create_judge(key: str, context: Context, default: AIJudgeConfigDefault | None = None, variables: Dict[str, Any] | None = None, default_ai_provider: str | None = None) → Judge | None

Creates and returns a new Judge instance for AI evaluation.

Parameters:

• key – The key identifying the AI judge configuration to use

• context – Standard Context used when evaluating flags

• default – A default value representing a standard AI config result

• variables – Dictionary of values for instruction interpolation. The variables message_history and response_to_evaluate are reserved for the judge and will be ignored.

• default_ai_provider – Optional default AI provider to use.

Returns:

Judge instance or None if disabled/unsupported

Example:

judge = client.create_judge(
    "relevance-judge",
    context,
    AIJudgeConfigDefault(
        enabled=True,
        model=ModelConfig("gpt-4"),
        provider=ProviderConfig("openai"),
        evaluation_metric_key='$ld:ai:judge:relevance',
        messages=[LDMessage(role='system', content='You are a relevance judge.')]
    ),
    variables={'metric': "relevance"}
)

if judge:
    result = await judge.evaluate("User question", "AI response")
    if result and result.evals:
        relevance_eval = result.evals.get('$ld:ai:judge:relevance')
        if relevance_eval:
            print('Relevance score:', relevance_eval.score)

create_model(key: str, context: Context, default: AICompletionConfigDefault | None = None, variables: Dict[str, Any] | None = None, default_ai_provider: str | None = None) → ManagedModel | None

Creates and returns a new ManagedModel for AI conversations.

Parameters:

• key – The key identifying the AI completion configuration to use

• context – Standard Context used when evaluating flags

• default – A default value representing a standard AI config result. When not provided, a disabled config is used as the fallback.

• variables – Dictionary of values for instruction interpolation

• default_ai_provider – Optional default AI provider to use

Returns:

ManagedModel instance or None if disabled/unsupported

Example:

model = client.create_model(
    "customer-support-chat",
    context,
    AICompletionConfigDefault(
        enabled=True,
        model=ModelConfig("gpt-4"),
        provider=ProviderConfig("openai"),
        messages=[LDMessage(role='system', content='You are a helpful assistant.')]
    ),
    variables={'customerName': 'John'}
)

if model:
    response = await model.run("I need help with my order")
    print(response.content)

create_tracker(token: str, context: Context) → Result

Reconstruct a tracker from a resumption token.

Delegates to LDAIConfigTracker.from_resumption_token().

Parameters:

• token – A URL-safe Base64-encoded resumption token obtained from LDAIConfigTracker.resumption_token.

• context – The context to use for track events.

Returns:

A Result whose value is a new LDAIConfigTracker on success, or whose error describes the problem on failure.

judge_config(key: str, context: Context, default: AIJudgeConfigDefault | None = None, variables: Dict[str, Any] | None = None) → AIJudgeConfig

Get the value of a judge configuration.

Parameters:

• key – The key of the judge configuration.

• context – The context to evaluate the judge configuration in.

• default – The default value of the judge configuration. When not provided, a disabled config is used as the fallback.

• variables – Additional variables for the judge configuration.

Returns:

The judge configuration with a tracker used for gathering metrics.

ldai.tracker

class ldai.tracker.AIGraphTracker(ld_client: LDClient, variation_key: str, graph_key: str, version: int, context: Context)

Tracks graph-level metrics for AI agent graph operations.

Maintains an internal AIGraphMetricSummary that is updated as tracking methods are called. Retrieve it via get_summary().

__init__(ld_client: LDClient, variation_key: str, graph_key: str, version: int, context: Context)

Initialize an AI Graph tracker.

Parameters:

• ld_client – LaunchDarkly client instance.

• variation_key – Variation key for tracking.

• graph_key – Graph configuration key for tracking.

• version – Version of the variation.

• context – Context for evaluation.

get_summary() → AIGraphMetricSummary

Get the current summary of graph-level metrics.

Returns:

Summary of graph metrics tracked so far.

property graph_key: str

Graph configuration key used in tracking payloads.

track_duration(duration: int) → None

Track the total duration of a graph run.

Records at most once per graph tracker; further calls are ignored.

Parameters:

duration – Duration in milliseconds.

track_graph_metrics_of(metrics_extractor: Callable[[Any], AIGraphMetrics | None], func: Callable[[], Any]) → Any

Track graph-level metrics for a synchronous graph operation.

Times the operation, extracts AIGraphMetrics via the provided extractor, and fires graph-level tracking events (path, duration, success/failure, total tokens).

If the extracted AIGraphMetrics has a non-None duration_ms, that value is used instead of the wall-clock elapsed time.

Node-level metrics are not tracked by this method.

For async operations, use track_graph_metrics_of_async().

Parameters:

• metrics_extractor – Function that extracts AIGraphMetrics from the result

• func – Synchronous callable that runs the graph operation

Returns:

The result of the operation

async track_graph_metrics_of_async(metrics_extractor: Callable[[Any], AIGraphMetrics | None], func: Callable[[], Any]) → Any

Track graph-level metrics for an async graph operation (func is awaited).

Same event semantics as track_graph_metrics_of().

Parameters:

• metrics_extractor – Function that extracts AIGraphMetrics from the result

• func – Async callable that runs the graph operation

Returns:

The result of the operation

track_handoff_failure(source_key: str, target_key: str) → None

Track failed handoffs between nodes.

Parameters:

• source_key – The configuration key of the source node.

• target_key – The configuration key of the target node.

track_handoff_success(source_key: str, target_key: str) → None

Track successful handoffs between nodes.

Parameters:

• source_key – The configuration key of the source node.

• target_key – The configuration key of the target node.

track_invocation_failure() → None

Track an unsuccessful graph run.

Records at most once per graph tracker. track_invocation_success and track_invocation_failure share state; only one of the two can record per graph tracker, and subsequent calls are ignored.

track_invocation_success() → None

Track a successful graph run.

Records at most once per graph tracker. track_invocation_success and track_invocation_failure share state; only one of the two can record per graph tracker, and subsequent calls are ignored.

track_path(path: List[str]) → None

Track the path traversed through the graph during a graph run.

Appends to the summary’s path list and fires a $ld:ai:graph:path event.

May be called multiple times per Tracker; each call records the provided path segment and appends it to the summary so the full path can be built incrementally.

Parameters:

path – An array of configuration keys representing the sequence of nodes executed during graph traversal.

track_redirect(source_key: str, redirected_target: str) → None

Track when a node redirects to a different target than originally specified.

Parameters:

• source_key – The configuration key of the source node.

• redirected_target – The configuration key of the target node that was redirected to.

track_total_tokens(tokens: TokenUsage | None = None) → None

Track aggregated token usage across the entire graph run.

Records at most once per graph tracker; further calls are ignored.

Parameters:

tokens – Token usage data, or None when usage is unknown.

class ldai.tracker.FeedbackKind(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Types of feedback that can be provided for AI operations.

Negative = 'negative'

Positive = 'positive'

class ldai.tracker.LDAIConfigTracker(ld_client: LDClient, run_id: str, config_key: str, variation_key: str, version: int, context: Context, model_name: str, provider_name: str, graph_key: str | None = None)

Records metrics for a single AI run.

All events a tracker emits share a runId (a UUIDv4) so LaunchDarkly can correlate them in metrics views. See individual track methods for their specific semantics. Call create_tracker on the AI Config to start a new run. A resumption token preserves the runId, so events emitted by a tracker reconstructed in another process correlate with the original run.

__init__(ld_client: LDClient, run_id: str, config_key: str, variation_key: str, version: int, context: Context, model_name: str, provider_name: str, graph_key: str | None = None)

Initialize an AI Config tracker.

Parameters:

• ld_client – LaunchDarkly client instance.

• run_id – Unique identifier for this AI run.

• config_key – Configuration key for tracking.

• variation_key – Variation key for tracking.

• version – Version of the variation.

• context – Context for evaluation.

• model_name – Name of the model used.

• provider_name – Name of the provider used.

• graph_key – When set, include graphKey in all event payloads (e.g. config-level metrics inside a graph).

classmethod from_resumption_token(token: str, ld_client: LDClient, context: Context) → Result

Reconstruct a tracker from a resumption token.

This is used for cross-process scenarios such as deferred feedback, where a different service needs to associate tracking events with the original tracker’s runId.

Parameters:

• token – A URL-safe Base64-encoded resumption token obtained from resumption_token.

• ld_client – LaunchDarkly client instance.

• context – The context to use for track events.

Returns:

A Result whose value is a new LDAIConfigTracker bound to the original runId from the token on success, or whose error describes the problem on failure.

get_summary() → LDAIMetricSummary

Get the current summary of AI metrics.

Returns:

Summary of AI metrics.

property resumption_token: str

A URL-safe Base64-encoded JSON string that can be used to reconstruct a tracker in a different process (e.g. for deferred feedback).

The token contains runId, configKey, version, and optionally variationKey and graphKey (omitted when empty). modelName and providerName are not included.

track_duration(duration: int) → None

Manually track the duration of an AI run.

Records at most once per Tracker; further calls are ignored.

Parameters:

duration – Duration in milliseconds.

track_duration_of(func)

Automatically track the duration of an AI run.

An exception raised while the function runs will still record the duration. The exception will be re-thrown.

Parameters:

func – Function to track (synchronous only).

Returns:

Result of the tracked function.

track_error() → None

Track an unsuccessful AI generation attempt.

Records at most once per Tracker. track_success and track_error share state; only one of the two can record per Tracker, and subsequent calls are ignored.

track_feedback(feedback: Dict[str, FeedbackKind]) → None

Track user feedback for an AI run.

Records at most once per Tracker; further calls are ignored.

Parameters:

feedback – Dictionary containing feedback kind.

track_judge_result(judge_result: Any) → None

Track a judge result, including the evaluation score with judge config key.

May be called multiple times per Tracker; each call records the provided judge result.

Parameters:

judge_result – JudgeResult object containing score, metric key, and success status

track_metrics_of(metrics_extractor: Callable[[Any], LDAIMetrics | None], func: Callable[[], Any]) → Any

Track metrics for a synchronous AI operation.

This function will track the duration of the operation, extract metrics using the provided metrics extractor function, and track success or error status accordingly.

If the provided function throws, then this method will also throw. In the case the provided function throws, this function will record the duration and an error. A failed operation will not have any token usage data.

For async operations, use track_metrics_of_async().

When the extracted LDAIMetrics object has a non-None duration_ms field, that value is used as the measured duration instead of the wall-clock elapsed time.

Because each inner metric is at-most-once per Tracker, calling this twice on the same Tracker will run the inner block again but produce no additional metric events.

Parameters:

• metrics_extractor – Function that extracts LDAIMetrics from the operation result

• func – Synchronous callable that runs the operation

Returns:

The result of the operation

async track_metrics_of_async(metrics_extractor: Callable[[Any], LDAIMetrics | None], func: Callable[[], Any]) → Any

Track metrics for an async AI operation (func is awaited).

Same event semantics as track_metrics_of().

When the extracted LDAIMetrics object has a non-None duration_ms field, that value is used as the measured duration instead of the wall-clock elapsed time.

Because each inner metric is at-most-once per Tracker, calling this twice on the same Tracker will run the inner block again but produce no additional metric events.

Parameters:

• metrics_extractor – Function that extracts LDAIMetrics from the operation result

• func – Async callable or zero-arg callable that returns an awaitable when called

Returns:

The result of the operation

track_success() → None

Track a successful AI generation.

Records at most once per Tracker. track_success and track_error share state; only one of the two can record per Tracker, and subsequent calls are ignored.

track_time_to_first_token(time_to_first_token: int) → None

Manually track the time to first token of an AI run.

Records at most once per Tracker; further calls are ignored.

Parameters:

time_to_first_token – Time to first token in milliseconds.

track_tokens(tokens: TokenUsage) → None

Track token usage metrics.

Records at most once per Tracker; further calls are ignored.

Parameters:

tokens – Token usage data.

track_tool_call(tool_key: str) → None

Track a tool call for this configuration (standalone or within a graph).

May be called multiple times per Tracker; each call records a tool call event for the provided tool key.

Parameters:

tool_key – Identifier of the tool that was invoked.

track_tool_calls(tool_calls: Iterable[str]) → None

Track the tool calls made during an AI run.

Appends to the summary’s tool call list and fires a $ld:ai:tool_call event for each tool.

May be called multiple times per Tracker; each call records an event for every tool identifier provided.

Parameters:

tool_calls – Tool identifiers (e.g. from a model response).

class ldai.tracker.LDAIMetricSummary

Summary of metrics which have been tracked.

__init__()

property duration_ms: int | None

Duration of the AI operation in milliseconds.

property feedback: Dict[str, FeedbackKind] | None

property resumption_token: str | None

URL-safe Base64-encoded resumption token captured at tracker instantiation. Useful for deferred feedback flows where a downstream process needs to associate events with the original AI run.

property success: bool | None

property time_to_first_token: int | None

property tokens: TokenUsage | None

property tool_calls: List[str]

List of tool keys that were invoked during this operation.

class ldai.tracker.TokenUsage(total: int, input: int, output: int)

Tracks token usage for AI operations.

Parameters:

• total – Total number of tokens used.

• input – Number of tokens in the prompt.

• output – Number of tokens in the completion.

__init__(total: int, input: int, output: int) → None

input: int

output: int

total: int