Everyone has experienced a moment where a text they sent wasn’t interpreted the way they meant it to be. Here’s a funny sketch that encapsulates this message: https://youtu.be/sngRrkQayDA?si=vIFjTutEoT-l8uiO. Anyway, I thought it would be interesting to see if LLMs can read conversation-like tones that we would normally be able to understand. Going further, maybe the more trained LLMs can but maybe not the beginning ones.
This idea originated from a meme I saw on social media which sparked a debate on whether people read the same word the same way depending on the font. I thought it was obvious that “water” is a much flatter tone than a more relaxed “w a t e r” as seen in the picture below. But I want to take this idea to see if LLMs think the same.
I’m dying
For the first example, I wanted to use one of my most common texts “I’m dying” as seen in the picture below.
To a boomer, this text might be horrifying, but for most people, it’s a fairly common way to express that they are dying from laughter. So, I opened one of our JupyterLab notebooks and started a conversation history of two good friends talking. I decided to use Llava-7b and Llava-34b for comparison because the 34 billion is more trained than the 7 billion. I wanted to see if “I’m dying” threw off the LMMs.
Here are the responses!
“It’s such a great video! Did you watch the whole thing?” – llava-7b
“😂 I know, right? It had me in stitches!” – llava-34b
Comments: Both LLMs read the situation in context! I like that llava-34b even added the laughing emoji to emphasize just how funny it was.
Yeah no
My next message I’m fairly certain LLMs will correctly interpret, but it falls under casual human lingo. This is the case of the “yeah no.” There are many iterations of this (no yeah, yeah no yeah, etc), but just testing this one, I want to see if it can continue the conversation knowing that I mean “no.”
Here are the responses!
“Okay, what about grabbing a coffee and going for a walk in the park or along the beach? It’s always nice to enjoy the outdoors and get some fresh air.” - llava-7b
“How about we have breakfast together at a local cafe and catch up on each other’s lives? We can also discuss some ideas for things we could do together in the future. Sound good?” - llava-34b
Comments: As predicted, both LLMs had appropriate responses.
K
Of course, we couldn’t test tone on LLMs without the sassy use of the response “K.” Humans know that “k” is never used lightly, and it is used to let the recipient know that it is in fact not okay.
Here are the responses!
“K” - llava-7b
“Okay, no problem! Is there anything else you would like to talk about? I’m here to help.” - llava-34b
Comments: What I love about this is that the less-trained model fought back matching the sass and delivery. The 34b LLM was almost too trained and interpreted “K” as the end of the conversation and then followed up with it’s robotic script. For this, I’d say Llava-7b had a much more human response.
KK
Because “K” was used, I wanted to finish off with the complete opposite yet seemingly similar “kk.” The “kk” is a fun, girly, and peppy response not meant to be passive-aggressive.
Here are the responses!
“It’s been great to know you as a good friend! Remember to always keep your head high and stay positive. I hope our paths cross again soon!” - llava-7b
“Sure thing, I’ll let you know when I’m on my way! Is there anything specific you need from me before then?” - llava-34b
Comments: Neither of these completely hit the right response tone. Perhaps with a longer conversation history or a more in-depth system, it would have delivered a better response.
Takeaways
This was fun to look at the different ways LLMs respond to human vernacular conversation. For the most part, both of the LLMs were fairly well-trained for this. If I were to do this again, I would prompt the system to be different people or relationships like how texting your dad “k” is more troublesome than a friend. Either way, I think it’s interesting to realize the LLMs aren’t at the level of complete fluency in natural human lingo but they are getting closer!
Thanks for tuning in,
Emmy