extend gptel-send--handlers

[avishefi]

Hi,
I'm using the below code to extend the send handlers so something can be tweaked between tool calls during the fsm execution. Currently I'm using this for skills that the LLM can dynamically add to the context as the first assistant message contents right after the system prompt. Later on I plan on adding queued responses between tool calls.

Do you have this planned, @karthink? If relevant, I would be interested in opening a PR for this for gptel, let me know if this looks interesting and how would you approach this.

(defun as/my-gptel-send--skills-handler (fsm)
  "Replace <enabled_skills>...</enabled_skills> in FIRST assistant ('gptel 'response) block
with dynamic skills prompt AFTER tool results, before re-request."
  (let* ((info (gptel-fsm-info fsm))
         (data (plist-get info :data)))
    (when (and (plist-get info :tool-success)  ; ONLY post-tools
               (consp data))
      (let ((msgs (plist-get data :messages)))
        (and (vectorp msgs)
             (cl-loop for msg across msgs
                      for raw-content = (plist-get msg :content)
                      for content = (if (stringp raw-content) raw-content "") ;; tool-calls have nil content
                      for role = (plist-get msg :role)
                      when (and (string= role "assistant")
                                (string-match "<enabled_skills>[^<]*</enabled_skills>" content))
                      do (let ((new-content (replace-regexp-in-string
                                             "<enabled_skills>[^<]*</enabled_skills>"
                                             (gptel-skills-enabled)
                                             content t nil)))
                           (plist-put msg :content new-content)
                           (message "Replaced <enabled_skills>...</enabled_skills> with dynamic skills prompt")
                           (cl-return t))))))))

(defun as/my-gptel-send (&optional arg)
  "Like `gptel-send', but replaces <enabled_skills> in first assistant message
with dynamic skills prompt *after tool results*, before re-request."
  (interactive "P")
  (if (and arg (require 'gptel-transient nil t))
      (call-interactively #'gptel-menu)
    (gptel--sanitize-model)
    ;; Copy handlers from gptel-send--handlers
    (let* ((handlers (copy-alist gptel-send--handlers))
           (wait-handlers (copy-sequence (alist-get 'WAIT handlers))))
      ;; Insert custom fn at START of WAIT handlers (post-tool-results/pre-HTTP)
      (push #'as/my-gptel-send--skills-handler wait-handlers)
      (setf (alist-get 'WAIT handlers) wait-handlers)
      ;; Create FSM with custom handlers & start request
      (let ((fsm (gptel-make-fsm :handlers handlers)))
        (gptel-request nil
          :stream gptel-stream
          :transforms gptel-prompt-transform-functions
          :fsm fsm)
        (message "Querying %s..." (gptel-backend-name gptel-backend))
        (gptel--update-status " Waiting..." 'warning)))))

[karthink]

@avishefi Could you explain the motivation for this change? I'm assuming that the skills information is to provide information from Claude Skills folders, but I don't understand why this is provided only after a tool call and not with the prompt.

[avishefi]

Skills in general, not only Claude.

I have skills implemented as a tool that adds a specific skill content into the context when called. Since it is controlled by the model, it can be added during tool call rounds, and if that is not added to the context immediately after the tool call the model won't see the addition, it will only be added after the next turn after sending another request manually. Another consideration is prompt engineering - since burying skills inside the conversation may be less of a strong sign in long context conversations, one way to optimize this is to send the skills as the first assistant message right after the system prompt. This way, although the context changes and it defeats the prompt cache (not system cache which stays static), it does make the skill appear more relevant. I'm not entirely sure about this, I'm still experimenting with this.

Another use is prompt queuing, where prompts can be queued for either after the fsm finishes or in between tool calls to add context during in-flight fsm processing. This allows for longer periods of gptel <-> model autonomous operations. The naive case is queuing "continue" messages after each gptel-request finishes in order to make the interaction continue unattended, which can be implemented with gptel-post-response-functions or by passing a callback to gptel-request. In-flight messages add a way to send something to the model during extended gptel <-> model interactions without gptel-abort, hence need to be processed between tool calls.

I think the general design pattern here is a hook (or a set of hooks) into the fsm handlers, both the send and request handlers, so that it is extensible. Since both handlers are private variables (with "--"), it seems this requires either a way to extend the current handlers or add hooks inside the built-in handlers which provide for extensibility.

[avishefi]

Going through this again, the context engineering part is probably not a good idea since it will cause unexpected conversation turns if appending the skills descriptions as the first assistant message mid conversation. The skill description can be returned as the skill tool result making the content visible during multi-turn fsm <-> model interactions.

Still, the pattern of adding messages between tool calls through a hook or extending the fsm handlers seem relevant during long fsm <-> model interactions.

[avishefi]

Initial thoughts:

(defcustom gptel-fsm-transition-functions nil
  "Abnormal hook run on fsm transitions.

Each function receives fsm and next state.")

(defun gptel--fsm-next (machine)
  "Determine MACHINE's next state according to its transition table.

MACHINE is an instance of `gptel-fsm'"
  (let* ((current (gptel-fsm-state machine))
         (transitions (alist-get current (gptel-fsm-table machine))))
    (cl-loop
     with info = (gptel-fsm-info machine)
     for (pred . next) in transitions
     when (or (eq pred t) (funcall pred info))
     (unwind-protect
         (run-hook-with-args
          gptel-fsm-transition-functions
          machine next)
       nil)
     return next)))