Today, it is easy to assume that emotion-aware AI began with ChatGPT and other large language models. Ask an LLM how you are feeling, and it can infer sadness, excitement, uncertainty, or joy from your words with surprising nuance.

But years before language models became part of everyday conversation, AI artists were already building systems that responded to human emotion—not by understanding language, but by interpreting faces.

One such project was Project Interwoven Expressions, an interactive installation created for Zeitgeistbot that combined real-time facial emotion detection with generative typography and AI-driven interaction.

The project explored a simple question:

What happens when artwork reacts to your emotional state rather than your physical presence?

The answer relied not on transformers or large language models, but on one of deep learning’s earlier breakthroughs: Convolutional Neural Networks, or CNNs.

Before Large Language Models

When most people hear “AI” today, they think of systems built on transformer architectures—models trained on enormous collections of text to predict the next token in a sequence.

Computer vision followed a different evolutionary path.

Throughout the 2010s, CNNs became the dominant architecture for recognizing visual information. They excelled at detecting patterns inside images, including:

• Faces
• Eyes
• Objects
• Handwritten numbers
• Medical imagery
• Facial expressions

Rather than processing sequences of words, CNNs learned spatial relationships between pixels. Small filters scanned an image layer by layer, gradually identifying increasingly complex features—from edges and corners to noses, mouths, eyebrows, and eventually entire facial expressions.

For emotion recognition, this was exactly what was needed.

Teaching a Computer to Recognize Emotion

Humans rarely think about how we recognize emotion.

Within milliseconds, we notice subtle changes such as:

• Raised eyebrows
• Tightened lips
• Wrinkled noses
• Smiles
• Changes in eye shape

CNNs learned these same visual relationships statistically.

Instead of relying on hard-coded rules such as, “If the mouth curves upward, the person is happy,” thousands of labeled examples taught the network to associate facial configurations with emotional categories.

Typical outputs included probabilities for emotions such as:

• Happy
• Sad
• Angry
• Fearful
• Surprised
• Disgusted
• Neutral

Rather than making absolute decisions, the network estimated confidence across multiple emotions simultaneously.

The Interactive Pipeline

Project Interwoven Expressions transformed those probabilities into artistic input rather than analytical output.

The interaction looked roughly like this:

Webcam
  ↓
Face Detection
  ↓
CNN Facial Expression Model
  ↓
Emotion Probabilities
  ↓
Generative Artwork
  ↓
Dynamic Typography and Visual Patterns

Instead of storing identity or attempting psychological profiling, the installation used the detected emotional state as a live creative parameter.

A smile might influence typography differently than surprise.

A neutral expression might leave the visual field relatively calm.

The artwork existed in constant negotiation between viewer and machine.

Why CNNs Were the Right Tool

Large language models are extraordinary at interpreting language.

Facial expression recognition, however, is fundamentally a vision problem.

CNNs offered several important advantages:

• Fast enough for real-time webcam interaction
• Lightweight enough to run inside a browser
• Effective at extracting spatial visual features
• Supported by a mature ecosystem of pretrained emotion models

Libraries such as TensorFlow.js, ml5.js, and browser-based facial expression classifiers allowed artists to deploy neural networks without requiring cloud infrastructure or expensive GPUs.

This made installations responsive, private, and immediate.

AI Before It Became Conversational

One of the most interesting aspects of Interwoven Expressions is that the AI was not trying to talk.

It was not generating essays.

It was not answering questions.

It was not pretending to understand the viewer.

Instead, AI functioned almost invisibly—as a perceptual layer.

The neural network observed visual signals, estimated emotional expression, and handed those measurements to another creative system that generated evolving typographic forms.

The installation combined emotion detection with GPT-generated visual language, creating a dialogue between computer vision and generative text before multimodal AI became mainstream.

AI as perception rather than conversation.

Before Multimodal Was a Buzzword

Today, we talk about multimodal AI as though it is entirely new.

Modern systems can combine:

• Vision
• Language
• Audio
• Video
• Reasoning

Project Interwoven Expressions explored a similar idea from an artistic perspective years earlier.

The installation merged:

• Computer vision and CNN-based emotion detection
• Generative typography
• Language generation
• Interactive installation design

Rather than asking AI to imitate human intelligence, the project asked AI to become another participant in the creative process.

Looking Back

The history of AI often gets rewritten around whichever technology is currently dominating the headlines.

Today, that technology is the large language model.

Before transformers transformed language, CNNs transformed vision.

They gave computers the ability to perceive patterns that previously required human observation, enabling applications ranging from medical imaging and autonomous vehicles to facial expression recognition and interactive art.

Project Interwoven Expressions sits at that intersection.

It reminds us that AI’s evolution was not a single leap toward conversation. It was a collection of specialized breakthroughs in vision, language, speech, and planning that eventually converged into the multimodal systems we use today.

Long before AI could discuss emotion, it first had to learn how to recognize the subtle movements of a human face.

And sometimes, that recognition became art.