Skip to content

Conversation

@kyle65463
Copy link
Collaborator

Overview

  • Support passing kwargs to inference engine to make its usage more flexible
  • Move system_prompt from vectorQ config to initialization of inference engines
  • Refine comments of vectorQ policies and inference engines
  • Rename open_ai.py and lang_chain.py to openai.py and langchain.py to follow their naming convention
  • Rename create method of InferenceEngine to infer to follow the naming of VectorQ class
  • Rewrote tests for inference engines and add tests for using kwargs

Passing **kwargs for inference engine

With the new interface, the user can pass additional parameters to the underlying inference engine to get more flexibility, for example:

vectorq.infer(
   prompt="What is the capital of France?",
   response_format={...}, # an argument of OpenAI's API
)

Or overriding the model/temperature of the inference engine initialization:

inference_engine = OpenAIInferenceEngine(
  model_name="gpt-4.1-nano",
  temperature=0.0,
)
...
vectorq.infer(
  prompt="What is the capital of France?",
  model_name="gpt-4o", # override when doing inference
  temperature=1.0, # override when doing inference
)

@kyle65463 kyle65463 force-pushed the kyle/refactor-inference-engine branch from 17c9112 to fb772b7 Compare May 5, 2025 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants