Chaz: An LLM <-> Matrix Chatbot

Chaz is a Matrix bot that connects you to a large number of LLM models. It is a replacement for the Chat interfaces that you might be used to when working with LLMs.

Code repository

Instead of dealing with each company’s custom interfaces, this just moves your chats into Matrix with all the features of your favorite Matrix client.

(Chaz is written in Rust, and while building this I also built a very basic Matrix Bot Framework in Rust, headjack)

By plugging into Matrix, we get a few things for free:

  1. Use any Matrix client you want, and any Matrix tooling since they are just Matrix chats.
  2. Let anyone you want use it (there’s an allow_list). For example, run a bot yourself and share access to your API keys with your less technical friends. Or access to your internal company models/keys.
  3. Share a chat session with anyone. Just add them into the Room.

Originally I wanted to allow users to use their own API keys, so you could run this as a simple service and just point people to it instead of making everyone run this themselves. There are obvious security implications with this though since whoever controlled the bot would have access to the keys, and I don’t want to deal with that now.

I have setup a demo account if you want to test it out a little, but long term this is something you’ll need to run yourself. You’ll only be able to send a few messages to the demo account, and you won’t be able to use it in large rooms. It’s also running a system prompt that is more for amusement than practicality.

The demo is @chaz-demo:jackson.dev

chaz-demo example conversation

Why Link to heading

A few weeks ago I was trying to play with different LLMs. Running some locally with ollama or llamafile, using ChatGPT or Gemini, or other APIs like Together.ai or Groq. All of those have different interfaces, either custom made or only available over API.

But it’s just a Chat interface, and I have already been moving all of my chats into Matrix.

AIChat was the missing piece that tied it together for me. It’s a command line interface that lets you setup a lot of different backends and select them on the command line. Pushing ollama and ChatGPT into the same interface.

So I had an idea, it’d be easy to write a simple Matrix bot, with a mostly text interface, and use AIChat as the backend to unify everything into Matrix.

Why is it called Chaz? Link to heading

Several months ago I saw someone on Hackernews mention that they use a system prompt requesting that the models refer to themselves as Chaz in the 3rd person. I’ve been using it myself and I find it hilarious.

I think it was a reference to Chazzzzz from the show Disenchantment.

Results Link to heading

After a bit of work, I now have Chaz. It is pretty rough, but I already use it as a mostly fully capable replacement for the other interfaces. You can:

  1. Send the chats to any backend setup in AIChat.
  2. Send images to the image models.
  3. Switch between backends in the middle of a session.
  4. Rename the Room/Topic using a model.
  5. Setup a role/system prompt to whatever you want.

It very quickly became a workable replacement for the Chat interfaces I had been using.

Matrix Bot Framework for Rust Link to heading

I wrote this in Rust but there doesn’t seem to exist a Matrix bot framework for Rust. So I ended up splitting out headjack into it’s own project.

It handles the bare minimum necessary to deal with the Matrix protocol with a lean towards helping write bots. A very simple bot could be written on top of headjack with minimal effort

Usage Link to heading

Chaz supports commands (currently everything prefixed with a ‘.’ to match AIChat, but that may change)

.help

Available commands:
- .print - Print the conversation
- .send - <message> - Send this message without context
- .model - <model> - Select the model to use
- .list - List available models
- .clear - Ignore all messages before this point
- .rename - Rename the room and set the topic based on the chat content
- .help - Show this message

Any message that isn’t a command causes the entire conversation (up to the last .clear command) to be send to AIChat for completion.

Future Link to heading

Here are some things I want to do with this, most of which just require some typing and not building anything novel:

  1. Login as your own Matrix account to provide suggested responses to messages. (I’m not sure the cleanest way to setup that interface).
    1. This one is definitely better when it’s builtin to your client, but we may be able to fake this one ok.
    2. Ideally I’d want to be able to edit my drafts from the bot, but I don’t think Matrix syncs drafts.
    3. e.g. Give it access to your own account, so you can ask Chaz “Help me draft a reply in room ‘Fizx Gang’, I want to explain XYZ”.
  2. Possibly use the Room Topic to store the configuration, for display/editing of the settings.
  3. Send to multiple backends in a single session.
  4. Respond to Reactions (thumbs up/down/etc). Maybe by regenerating responses, selecting the response you want to continue with when using multiple backends, etc.
  5. Support both sending and receiving files, to hookup to image generation backends.
  6. Replace AIChat as the backend, as using that as the “API” is fairly limiting.
  7. LLamafile support.
  8. Support for changing the model settings from the chat. Temperature, Grammar, Tokens requested, etc.