Chatting with LLMs in a terminal

The Large Language Models is a technology and a tool, which in the hands of the people who know what they are doing can be very useful. I experienced such usefulness multiple times at work. The added value is real. However, a big deterrent for the adoption of the LLMs as a productivity tool for me was the UI and the lack of good integration with my keyboard-centric, terminal-based workflow. To not fall behind in the adoption I made my own interactive shell for chatting with self-hosted LLMs.

Self-hosting

I consider to utilize the LLM as a worktool primarily with the self-hosted open-weights models. Why? - Simply I do not want my workflow and my tools to depend on a will of some vendor. Self-sufficiency is my long term strategy. My client currently supports only Ollama as a backend, but this part is not that important.

Features

In order to make the chats with LLM feel less cumbersome in the terminal, the following features are supported:

• Automatic scrolling in tmux copy-mode to the beginning of the LLM response
• Formatting LLM response as a markdown with syntax highlighting (via Rich)
• Converting LaTex formulas in LLM response to Unicode (via TeXicode)
• User input with hotkeys (I use vi but emacs is supported), history and tab completion for commands (via python-prompt-toolkit)
• Multi-line input by using your text editor (from the $EDITOR environment var)
• Saving/loading conversations
• Running in a non-interactive mode for scripting

Demo

Programming questions:

Math questions:

Further development and afterthoughts

I should probably work next on implementing support for the Retrieval-augmented generation (RAG) and adding agentic capabilities, but currently the biggest obstacle in making this setup more useful for me is the lack of RAM, the models that I can self-host are just too dumb on my current hardware :`-)

There is an opinion that with the wide adoption of LLMs we will see a rise of mediocre products, particularly in software. I do not think that it would be a new state of affairs, to be honest, the software has been “mediocre” for a long time. On the other hand no one prevents you as a skillful programmer to employ LLMs to help make the software better. For example this program was written on Python and requires many megabytes of dependencies to run, another programmer could rewrite it in C/C++ with almost no 3d party dependencies which would make it a leaner, thus, a better piece of software, and I do believe that in this process, employing the LLMs can provide a productivity gain (perhaps I can even try doing it myself when this setup is finished).

The project page on Github: https://github.com/redahe/xllmshell/