Feature request: querying LLMs with visidata cells input

[GnarlyMshtep]

Feature request: querying LLMs with visidata cells as input

I often work with visidata to look at sheets which have lots of unstructured data (text). Currently, if I try to find pattern in the text, I would try to narrow my search with a combination of Regex search and manual inspection.

I think could be really useful if I could pick some number of cells e.g. within a certain column (the usual way they are selected in visidata) and forward them to an LLM (e.g. rendered as json and preceded by the additional query).

I am not sure how ambitious this but would be curious to hear from the team! To me this seems like a versatile command that will only be getting more useful.

Huge visidata fan! Thank you for the consideration.
Matan

[saulpw]

Thanks @GnarlyMshtep! I've been thinking about something like this idea myself. Can you provide a bit more context for the exact workflow? Like what would the "additional query" look like, and how would you read/parse the results?

[yarikoptic]

"Exactly the same" question came up in the scope of our fresh https://github.com/ReproNim/ABCD-visidata . In our case we have a column (or columns) with meaningful information which we would love to make LLM access/query to answer the question on which of those rows out of 90k of them are of interest to answer the question (e.g. "What is the relationship between volume of amygdala and how much youth was smoking on mondays after having a morning jin & tonic?") -- then we would have those rows selected, I click 'zm' to get the table presenting the data and potentially LLM also takes an initial stab at analytics there

[GnarlyMshtep]

Thanks @saulpw ! Sorry for the late response -- was working for a paper deadline. Let me try to throw some thoughts down:

Use case

The following is my most common visidata workflow:

I just generated a bunch of LM trajectories, along with traces I generated some summary statistics, and something feels unexpected (e.g. the task success rate is unexpectedly low). I logged all the results in a structured .jsonl file (dataclass -> json) and I am using vd to navigate the trace and 'vibe check' what could be going on. To do that, I might filter / sort by a certain statistic and then read a few trajectories.

This workflow is nice because I am reading exactly the text inputted and outputted by the LMs, but its not scalable. Sometime I would have a specific question which I would want to ask on all the trajectories, like, "did the LM do X?", especially where the behavior X is rare. It would be nice if I could just select to run all the text from a column on this question from inside visidata and get an answer.

Note that there are applications, e.g. https://docent.transluce.org/, which are dedicated exactly for transcript discovery and analysis. I am thinking of these questions as being smaller, more narrow scope, and quick iteration then using e.g. docent. A docent integration could be very interesting tho!

An example MVP features implementation

• provide an API key in a config file and model of choice
• There is a template (that you can edit) which takes in a {natural_lang_predicate} ("did LM do X?") and some text to be analyzed {text_to_be_analyzed}. The template asks the model to analysis and score which indicates to what extent (from 0 to 1) the LM thinks the behavior exists.
• Inside visidata select a column containing text field (by placing cursor on it?) and with some keyboard shortcut trigger thenatural_lang_predicate prompt. Enter the natural_lang_predicate.
• Confirm that # many text field with total # chars will be sent to model of choice.
• model of choice is invoked # many text field text fields time parallel to analyze text_to_be_analyzed.
• visidata parses analysis and score in response and adds column to the sheet for each.
• user can interact with the score, e.g. by sorting, then reading analysis, etc.

How does this sound? Anything I should clarify? Not 100% sure this would end up being useful for many but I think I would use this!

[saulpw]

I got Claude to implement this (and provided some suggestions along the way). Please test the PR (you'll have to import visidata.experimental.llm in your visidatarc). Let me know how it goes and we can fix and/or merge it!

[GnarlyMshtep]

Sorry for the late response. Could u explain what are the steps to testing this? I am guessing I need to clone this commit, add to my visirc, and then llm will me in = function menu?

How can I e.g. configure the default prompt template, pick the LLM, etc?