A chain for evaluating ReAct style agents.

This chain is used to evaluate ReAct style agents by reasoning about the sequence of actions taken and their outcomes.

Hierarchy

  • AgentTrajectoryEvaluator
    • TrajectoryEvalChain

Constructors

Properties

llm: BaseLanguageModelInterface

LLM Wrapper to use

outputKey: string = "text"

Key to use for output, defaults to text

outputParser: TrajectoryOutputParser = ...
prompt: BasePromptTemplate

Prompt object to use

requiresInput: boolean = true
requiresReference: boolean = false
criterionName?: string
evaluationName?: string = ...

The name of the evaluation.

llmKwargs?: any

Kwargs to pass to LLM

memory?: any
skipInputWarning?: string = ...
skipReferenceWarning?: string = ...

Accessors

Methods

  • Parameters

    • inputs: ChainValues[]
    • Optionalconfig: any[]

    Returns Promise<ChainValues[]>

    Use .batch() instead. Will be removed in 0.2.0.

    Call the chain on all inputs in the list

  • Run the core logic of this chain and add to output if desired.

    Wraps _call and handles memory.

    Parameters

    • values: any
    • Optionalconfig: any

    Returns Promise<ChainValues>

  • Check if the evaluation arguments are valid.

    Parameters

    • Optionalreference: string

      The reference label.

    • Optionalinput: string

      The input string.

    Returns void

    If the evaluator requires an input string but none is provided, or if the evaluator requires a reference label but none is provided.

  • Evaluate a trajectory.

    Parameters

    • args: LLMTrajectoryEvaluatorArgs
    • OptionalcallOptions: unknown
    • Optionalconfig: any

    Returns Promise<ChainValues>

    The evaluation result.

  • Invoke the chain with the provided input and returns the output.

    Parameters

    • input: ChainValues

      Input values for the chain run.

    • Optionaloptions: any

    Returns Promise<ChainValues>

    Promise that resolves with the output of the chain run.

  • Format prompt with values and pass to LLM

    Parameters

    • values: any

      keys to pass to prompt template

    • OptionalcallbackManager: any

      CallbackManager to use

    Returns Promise<EvalOutputType>

    Completion from LLM.

    llm.predict({ adjective: "funny" })
    
  • Parameters

    • inputs: Record<string, unknown>
    • outputs: Record<string, unknown>
    • returnOnlyOutputs: boolean = false

    Returns Promise<Record<string, unknown>>

  • Parameters

    • input: any
    • Optionalconfig: any

    Returns Promise<string>

    Use .invoke() instead. Will be removed in 0.2.0.