Category: AI, Human-Robot Interaction, Machine Learning, Neural Networks, Deep Learning

Hello everyone! Welcome to MetroForm's Lab, a blog documenting a unique project. I'm excited to share the beginning of a new journey in developing Mika-X's emotional intelligence capabilities. Mika-X is an advanced humanoid companion concept, and currently, she only has a "body" (as you can see in the Projects section). That was the easy part. Now, it's time to give her "mind". This project aims to give her the ability to understand and respond to human emotions in real-time.

If you are new to AI, neural networks and machine learning, you might find this blog useful for your own projects. If you have any questions, feel free to drop me a line.

Human-Robot Interaction (HRI) is a cornerstone of my vision at Metroform Studio. While Mika-X's physical design might be polished and detailed, true companionship (assuming, of course, she would be one day manufactured) requires emotional intelligence. My goal is to create a system that can:

• Recognize human emotions through facial expressions
• Analyze emotional content in voice patterns
• Combine multiple inputs for accurate emotion detection
• Generate contextually appropriate responses

This isn't just about creating an impressive AI/ML portfolio piece, while I'm working on my IBM AI Developer Professional Certificate — it's also about creating more meaningful connections between humans and machines.

Technical Foundation: Setting Up My AI Laboratory

Before diving into the exciting parts, I needed to establish a solid technical foundation. I'm using Python as my primary programming language, along with several specialized libraries:

TensorFlow/Keras: For building and training the neural networks
OpenCV: For facial detection and image processing
Librosa: For audio analysis and feature extraction
MediaPipe: For advanced facial landmark detection

One of the first challenges was setting up a stable development environment. If you've worked with machine learning projects before, you know the pain of dependency conflicts... After going back and forth several times, I finally settled on using Anaconda to manage the environment, which turned out to be a lifesaver for keeping all my packages playing nicely together. One of the benefts of Anaconda is that it continously compares all dependencies and upgrades/downgrades them accordingly, in order to ensure compatibility.

Data: The Foundation of Emotional Intelligence

AI systems are only as good as the data they're trained on. For this project, I'm currently working with two primary datasets:

FER-2013 (Facial Expression Recognition): A comprehensive dataset containing over 35,000 facial images labeled with seven basic emotions (happy, sad, angry, surprised, fearful, disgusted, and neutral).

RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song): A collection of voice recordings by professional actors expressing various emotions.

Data preprocessing was a crucial step. I had to:

• Normalize image sizes and pixel values
• Extract meaningful audio features from voice recordings
• Split our data into training and testing sets
• Apply data augmentation to improve model robustness

I achieved this with two scripts: data_preprocessing.py (for images) and voice_preprocessing.py (for audio).

The Architecture: Dual-Modal Emotion Recognition

I decided to use a dual-modal system that analyzes both facial expressions and voice patterns separately before combining the results. Here's my high-level architecture:

1. Facial Emotion Recognition Pathway:

• Face detection using Haar cascades
• Feature extraction using Convolutional Neural Networks (CNNs)
• Emotion classification with a specialized model architecture

2. Voice Emotion Recognition Pathway:

• Audio recording and preprocessing
• Feature extraction (MFCCs, chroma, spectral features)
• Emotion classification using a neural network

3. Fusion and Response Generation:

• Combining results from both pathways
• Confidence-weighted decision making
• Contextually appropriate response generation

Challenges and Learnings

Every project comes with some challenges, and this one is no exception:

Data Quality: Real-world emotion data is messy and the commonly available data sets are of questionable quality. People express emotions differently, and labeling is subjective. Also, a lot of the sample images in FER-2013 are misleading or simply useless, not showing the emotion clearly (see the examples below). Some are too blurry, too dark, duplicated, partially obscured, badly framed, or covered with watermarks. I will definitely need to augment the image data set with additional, higher quality images (I will cover the techniques in the following posts).
Computational Resources: Training neural networks requires significant processing power. My workstation is currently looking at me like 😱 "Dude, seriously?". Oh, well.
Real-time Processing: Analyzing both video and audio in real-time presents technical challenges. For every single frame from the camera, the code will need to perform several steps (read, convert color, resize, run the model, draw boxes). Will see about the speed when the camera and microphone come into play...😳

What's Next?

I'm currently in the model training phase, with pretty good initial results. In our next post, I'll share details about my model architectures, training processes, performance metrics I'm achieving, and how I intend to finetune the training sets. Stay tuned as I continue building Mika-X's emotional intelligence. Cheers!

- Adam