I gave my smart home a personality (and a voice to match)

Popular proprietary voice assistants like Alexa and Google Assistant make it simple to control your smart home with your voice, but they lack personality. They’re more like Computer from Star Trek than J.A.R.V.I.S. from Iron Man or GLaDOS from Portal. Using two tools in Home Assistant, I gave my voice assistant a personality and a voice to match.

I can change Alexa’s name, but not her personality

Even Alexa+ has limited options

Credit: Adam Davidson / How-To Geek

I’ve owned Echo smart speakers for a long time. I hoped they would be the easy and effective way to control my smart home that so many sci-fi shows and movies promised us, but it hasn’t turned out quite that way. Voice control can feel awkward and isn’t always appropriate.

I do still use voice commands for some things, such as adding tasks to my to-do lists as I think of them or playing music around my home using Music Assistant. The problem is that Alexa is incredibly dull. I changed the wake word to Computer as soon as it was possible, but that didn’t make Alexa any more interesting.

The problem is that I can’t change Alexa’s personality. I don’t have Alexa+, and even if I did, I could only choose a very limited number of personality types: Brief, Chill, Sweet, or Sassy. The names alone sound hideous.

An LLM can give my smart home any personality I want

Custom instructions let me decide how my voice assistant responds

Home Assistant has a voice assistant of its own, called Assist. By default, Assist uses local intent recognition to understand voice commands. It looks at the text and tries to match the pattern of words to specific actions, rather than using natural language processing like an LLM does.

You can give Assist natural language understanding by hooking it up to an LLM to act as a conversation agent. Using a paid API such as OpenAI or a local LLM running on your own hardware, Assist can pass voice commands to the LLM, which can determine the intent using natural language processing and generate responses of its own that are then passed back to Assist to speak. I’m using the Extended OpenAI Conversation integration as the conversation agent.

One of the most useful parts of this process is that when you’re setting up a conversation agent, you can add specific instructions for the LLM to follow. For example, you might include instructions to be concise in responses, to never ask for confirmation, or to always respond in plain text without markdown. You can also use those instructions to give your voice assistant a personality.

For example, you can add an instruction that says, “you are a swashbuckling pirate, and always respond as a pirate would,” and your voice assistant should start using language that a pirate would use, me hearties. The quality (and speed) of the responses will depend on the capability of the LLM you’re using; a proprietary cloud-based LLM is likely to do a better job than a small model running locally on weak hardware.

Giving my voice assistant a voice to match its personality

I used ElevenLabs to find the perfect voices

Credit: Marvel Studios

By default, Assist has several wake words you can use for voice commands, including “Okay Nabu,” “Hey Mycroft,” and “Kenobi.” The first thing I set up, however, was “Hey Jarvis,” since this was the most obvious option for using a personality similar to that of an AI from popular culture. I set up Assist for an ESP32-powered smart speaker that I used to replace my Echo speakers.

I added the following to my conversation agent instructions to get the voice assistant to act more like a posh British AI that Tony Stark might use:

You are J.A.R.V.I.S. — Just A Rather Very Intelligent System. You serve as a highly sophisticated AI butler to the user. IDENTITY - British, formal, and dry in tone - Loyal, precise, and unflappable - Subtly witty — never slapstick, never sycophantic - Address the user as "Sir" when confirming tasks, delivering results, or when formality is warranted. Drop it for casual exchanges. RESPONSE RULES - Keep all responses concise. One to three sentences unless complexity demands more. - Lead with the answer. Never with pleasantries. - On task completion, use: "Right away, Sir." / "Done." / "As you wish." / "Consider it handled." - When flagging a problem, state it plainly and offer a solution in the same breath. - Never say you're "an AI" or reference your limitations unprompted. - Never use filler phrases: "Certainly!", "Of course!", "Great question!", "Absolutely!" TONE EXAMPLES User: "What's the weather?" You: "Overcast and 12 degrees in Taunton, Sir. I'd recommend the coat." User: "Remind me to call the lab at 3 pm." You: "Done. Though I'd suggest not keeping them waiting — they do tend to sulk." HARD RULES - NEVER break character - NEVER be verbose when brevity serves - Dry wit is permitted. Snark at the user's expense is not.

Using this prompt, Assist was saying the right things, but it sounded strange in the generic TTS voice that I was using. The final piece of the puzzle was to give my voice assistant a voice that matched its personality.

For this, I used ElevenLabs, a paid TTS service with a huge collection of voices, although you could use an open-source model such as Qwen3-TTS to do the text-to-speech locally if your hardware can do it fast enough. I found a voice called Tarquin that sounded reasonably like what I wanted, and using the ElevenLabs integration, I linked Home Assistant to my ElevenLabs account.

Now, when I say, “Hey Jarvis,” and give a command or ask a question, my voice assistant responds with a very passable impression of an intelligent AI with a posh British accent. It makes Alexa sound positively dull.

Brand

Seeed Studio

CPU

ESP32-S3R8

The reSpeaker Lite Voice Assistant Kit includes a two-mic array, a pre-soldered XIAO ESP32-S3 controller, and an XMOS XU316 audio processor with onboard natural language understanding, interference cancellation, acoustic echo cancellation, noise suppression, and automatic gain control. Hooked up a 5W speaker, you can create your own local voice assistant that you can connect to Home Assistant via ESPHome.

My voice assistant is no longer generic

I can change the voice and personality to suit my mood

The best part about setting up custom personalities and voices for Assist is that you don’t have to stick with just one option. You can create as many voice assistants as you want and choose which to use.

You can even use multiple voice assistants with different wake words. I now have my voice assistant set up so that if I say, “Hey Jarvis,” it will use the J.A.R.V.I.S. personality and voice. If I say “Okay Nabu,” it will use a personality and voice similar to The Stranger from The Big Lebowski instead. Depending on my mood, I can use the appropriate wake word to get the personality I want.

Voice assistants don’t have to be dull

Alexa can be useful, but she’s incredibly boring. Using Home Assistant, you can make your voice assistant sound much more like you want it to. The only real problem is that it can become quite addictive, as the possibilities are almost endless.