Case Study - Making AI Chat Indistinguishable from Human Conversation

Why AI text is instantly recognizable (and it is not just the vocabulary)

Most people can spot AI text within seconds. Ask them why, and they will say something vague like "it sounds too polished" or "the words are weird." But when we dug into this for a client building a conversational product, we found the problem runs much deeper than vocabulary choices.

AI text has a rhythm problem. Human writing is messy, inconsistent, full of half-finished thoughts and sudden topic shifts. AI writing is metronomic. Every sentence lands at roughly the same length. Every paragraph follows the same shape. The overall effect is like listening to a drummer who never misses a beat: technically perfect, but uncanny.

We spent three months cataloguing exactly what gives AI away in chat contexts, running detection experiments, attempting fine-tuning on real conversation data, and ultimately developing a practical toolkit for making LLM output feel genuinely human. This case study documents what we found.

The five detection signals we identified

Through systematic analysis of AI and human chat transcripts, we isolated five primary signals that humans (and automated detectors) rely on.

1. Vocabulary distribution. AI models reach for formal, low-frequency words that real people never use in casual conversation. Words like "delve," "crucial," "facilitate," "I'd be happy to help," and "Great question!" are reliable AI markers. Human chat is full of filler, slang, and repetition.

2. Burstiness. This turned out to be the strongest signal. Human sentence length varies wildly within a single message. AI produces remarkably uniform output. Measure the standard deviation of sentence lengths across a conversation, and the difference is striking.

3. Message structure. AI defaults to one long, comprehensive reply. Humans send multiple short messages in rapid succession, often splitting a single thought across three or four bubbles.

4. Tonal consistency. AI maintains the same register throughout an entire conversation. Humans shift constantly, mixing formal phrasing with slang, enthusiasm with sarcasm, all within a few lines.

5. Punctuation habits. AI overuses em dashes, semicolons, and perfect comma placement. In real chat, humans rarely use em dashes at all. They use ellipses, leave out commas, and break grammar rules freely.

Here is what the difference looks like in practice:

AI response:
"That's a great question! I'd be happy to help you with that.
The key consideration here is understanding the underlying
framework, which facilitates a more nuanced approach to the
problem. Let me break this down for you."

Human response:
"oh yeah I looked into that"
"basically the framework is kinda weird"
"like you have to set it up manually??"
"idk if thats normal or not"
"but yeah it works once you figure it out lol"

We tried fine-tuning on WhatsApp exports. Here is what happened.

The obvious idea: if you want an AI to sound like a specific person, train it on their messages. We exported 2,000+ WhatsApp messages from a volunteer's chat history, cleaned the data, and ran LoRA fine-tuning on Llama 3.

The results were disappointing.

The model memorized specific phrases and emoji patterns rather than learning the underlying conversational style. It would drop the volunteer's most common expressions into completely wrong contexts. With fewer than 1,000 training examples, the model exhibited catastrophic overfitting, repeating the same three or four sentence fragments regardless of input.

We also discovered that high-perplexity tokens in chat data (emoji sequences, misspellings, internet slang) damaged model parameters in unpredictable ways. The fine-tuned model became less coherent on general topics without meaningfully improving on style matching.

Fine-tuned output (trying to sound like the volunteer):
"haha ja genau 😂 also ich würd sagen haha ja genau
das ist mega so haha ja"

Actual volunteer message:
"ja bin mir nicht sicher ehrlich gesagt"
"hast du das schon mal probiert?"

The conclusion: fine-tuning on chat data does not work well without thousands of high-quality examples per speaker. For most use cases, few-shot prompting with carefully selected example messages is both cheaper and more effective.

What actually makes AI chat more human-like

Based on our experiments, these techniques produced measurable improvements in human evaluation scores.

Strategic imperfection. Instruct the model to occasionally hesitate, self-correct, or trail off. Not randomly, but in places where humans naturally do so: when uncertain, when changing direction, or when searching for the right word.

Message splitting. Break single responses into multiple short messages. A response that would be one paragraph becomes three or four chat bubbles. This alone improved naturalness scores by 25%.

Varied sentence length. Explicitly prompt for a mix of very short and medium-length sentences. Avoid the AI tendency toward uniform 15 to 20 word sentences.

Vocabulary filtering. Post-process outputs to replace known AI vocabulary markers. Swap "crucial" for "important" or "really matters." Remove "I'd be happy to" entirely. Strip semicolons and em dashes. This can be done with a simple output filter without touching the model.

Register matching. Analyze the user's messages and mirror their formality level. If they write in fragments with no capitalization, respond the same way.

Before (standard LLM):
"I appreciate you sharing that! There are several important
considerations to keep in mind. First, you'll want to ensure
that the configuration aligns with your specific requirements."

After (with our techniques applied):
"yeah that makes sense"
"so the main thing is getting the config right"
"I messed around with it for a while and honestly"
"just matching the settings to what you actually need works best"

The watermarking reality in 2026

The EU AI Act requires AI-generated content to be machine-detectable by August 2026. Google's SynthID and similar systems from Anthropic embed statistical watermarks by biasing token selection during generation. The tokens themselves look normal, but their distribution carries a detectable signal.

Here is the fundamental tension: these watermarks are trivially removable. Running watermarked text through a paraphrasing model destroys the statistical signal completely. Even simple operations like translating to another language and back, or asking a different model to rewrite the text, strip the watermark.

For builders of conversational AI products, this means watermarking is a compliance checkbox, not a technical barrier. The real challenge remains building AI that communicates naturally enough that detection becomes a moot point, not because you are evading it, but because the conversation genuinely serves the user.

Results and practical recommendations

Human evaluators scored our optimized outputs 40% higher on naturalness compared to baseline LLM responses in blind A/B testing. The most impactful individual technique was message splitting (25% improvement alone), followed by vocabulary filtering (15%) and strategic imperfection (12%).

Our practical recommendations for teams building conversational AI:

1. Do not fine-tune on small chat datasets. Use few-shot prompting with 5 to 10 representative messages instead.
2. Split responses into multiple messages. This single change has the largest impact on perceived naturalness.
3. Build a vocabulary blocklist. Maintain a list of known AI markers and filter them from output.
4. Measure burstiness. Track the variance in your model's sentence lengths. If the standard deviation is below 5 words, the output will read as robotic.
5. Match the user's register. Analyze incoming messages and adjust formality dynamically.
6. Accept imperfection. The goal is not perfect grammar. The goal is communication that feels real.

If you are building a product where AI needs to sound human, or if you are running into detection issues with your current approach, get in touch. We have worked through these problems and can help you skip the expensive dead ends.

What we did