LLM Extension

Creating AsyncLLMBaseExtension by using nman

Run this following command:

tman install extension default_async_llm_extension_python --template-mode --template-data package_name=llm_extension --template-data class_name_prefix=LLMExtension

Abstract APIs to implement

on_data_chat_completion(self, nyra_env: NyraEnv, **kargs: LLMDataCompletionArgs) -> None

This method is invoked when the LLM Extension processes a data completion request, specifically in scenarios where data is transmitted using the data protocol in streaming mode.

on_call_chat_completion(self, nyra_env: NyraEnv, **kargs: LLMCallCompletionArgs) -> any

This method is triggered when the LLM Extension processes a call completion request, applicable in scenarios where data is transmitted using the call protocol in non-streaming mode.

on_tools_update(self, nyra_env: NyraEnv, tool: LLMToolMetadata) -> None

This method is called when the LLM Extension receives a tool update request.

APIs

cmd_in: tool_register

This API facilitates the handling of tool registration requests. It accepts an array of LLMToolMetadata objects as input, which are subsequently appended toself.available_tools, ensuring their availability for future operations.

cmd_out: tool_call

This API is utilized to dispatch tool call requests. It can be integrated with any designated LLMTool Extension endpoint to retrieve the corresponding tool call results.