Lookalike: Celebrity Face Matching Application - Research Release Documentation

1. Introduction

Lookalike is a web-based application designed to match user-uploaded photos to celebrity lookalikes using facial recognition technology. The application leverages deep learning models for face embedding generation and similarity search to identify the closest celebrity matches. This documentation provides a comprehensive overview of the development process, data preparation, embedding generation, matching formula, and deployment details.

The project was developed by M Zaid and hosted on Hugging Face Spaces with the frontend deployed on Cloudflare Pages.

2. Application Development

This application was built using Python for the backend and HTML/JavaScript for the frontend. The backend processes image uploads, generates face embeddings, and performs similarity searches. The frontend, provides an intuitive interface for users to upload photos and view their celebrity matches.

2.1 Backend Development

The backend was developed using FastAPI, a modern Python web framework, to handle asynchronous image uploads and API requests. Key libraries include:

DeepFace (v0.0.86): For face embedding generation using the ArcFace model.
FAISS: For efficient similarity search using a precomputed index.
TensorFlow (v2.15.0): For running the ArcFace model
Pandas and NumPy: For data handling and numerical operations.
Requests: For uploading user images to imgbb for hosting.

3. Data Preparation

The dataset for Lookalike was sourced and prepared to include a comprehensive set of celebrity images, names, and precomputed embeddings. The dataset is also hosted on Hugging Face.

3.1 Data Collection

The dataset contains 346,000 rows of celebrity data, including names, images, and URLs. Example entries include celebrities like Terence Bernie Hines and Tom Hines. The data was collected from public sources, such as the IMDB-Face dataset, and curated to ensure a diverse range of celebrities.

3.2 Data Structure

The dataset includes the following files:

IMDB-Face-with-embeddings.csv: A CSV file with columns `name`, `image`, and `url` for each celebrity.
celebrity_embeddings_normalized.npy: A NumPy array of precomputed face embeddings.
celebrity_index_cosine.faiss: A FAISS index for efficient similarity search using cosine similarity.

4. Embedding Generation

Face embeddings were generated using the DeepFace library with the ArcFace model, which provides high-accuracy facial recognition embeddings. The process involved the following steps:

4.1 Image Processing

Each celebrity image from the dataset was processed to extract a 512-dimensional face embedding. The images were loaded using PIL (Python Imaging Library), converted to RGB format, and passed to DeepFace.

4.2 Embedding Extraction

The DeepFace `represent` function was used with the ArcFace model:

                embedding = DeepFace.represent(image, model_name='ArcFace', enforce_detection=False)[0]['embedding']
            

The `enforce_detection=False` parameter ensures embeddings are generated even if face detection fails. Each embedding was normalized using L2 normalization to prepare it for cosine similarity search:

embedding = embedding / np.linalg.norm(embedding)

4.3 FAISS Index Creation

The normalized embeddings were stored in a NumPy array (`celebrity_embeddings_normalized.npy`) and used to create a FAISS index (`celebrity_index_cosine.faiss`) optimized for cosine similarity search. FAISS was chosen for its efficiency in handling large-scale similarity searches.

5. Matching Formula

The matching process involves generating an embedding for the user-uploaded image and comparing it against the precomputed celebrity embeddings using cosine similarity.

5.1 User Image Embedding

When a user uploads an image, it is processed similarly to the dataset images using DeepFace’s ArcFace model. The resulting 512-dimensional embedding is normalized:

                user_embedding = DeepFace.represent(np.array(img), model_name='ArcFace',
                enforce_detection=False)[0]['embedding']

                user_embedding = np.array(user_embedding).astype('float32')

                user_embedding /= np.linalg.norm(user_embedding)

5.2 Similarity Search

The normalized user embedding is queried against the FAISS index to find the top 5 closest matches. FAISS uses cosine similarity, where a higher score indicates a closer match. The similarity score is converted to a percentage:

                user_embedding = user_embedding.reshape(1, -1)

                D, I = index.search(user_embedding, k=5)

                similarity = round(float(score * 100), 2)

Here, `D` contains the similarity scores, and `I` contains the indices of the top 5 matching celebrities. The scores range from 0 to 1, where 1 indicates an identical match.

6. Conclusion

Lookalike successfully demonstrates the application of facial recognition technology for celebrity matching. By leveraging DeepFace, FAISS, and a robust dataset, the application provides accurate and efficient matching results. Future improvements could include GPU support for faster processing, additional models for embedding generation, and enhanced security for API access.