AI Virtual Makeup Try-On
Beauty brands face a dilemma: how to let customers try on cosmetics without physical contact? Naive 2D image overlays look unnatural—they ignore reflections, skin texture, and facial expressions. We solve this with a hybrid architecture: landmark-based detection for real-time and AI generative models for high-quality photos. Our experience in Computer Vision (5+ years, 20+ projects in the beauty segment) shows that this approach reduces cosmetic returns by 30% thanks to more accurate shade matching. As a result, clients get not just a try-on tool, but one that increases cart conversion by 25% and cuts return rates.
For real-time scenes, we use MediaPipe Face Mesh [^1]—a solution from Google that detects 468 facial keypoints with under 5 ms latency. For high-quality photos, we apply Stable Diffusion Inpainting with a mask of application zones.
Comparing Approaches: Landmark-Based vs AI-Generative
| Characteristic | Landmark-Based (MediaPipe) | AI-Generative (Stable Diffusion) |
|---|---|---|
| Speed | < 5 ms (real-time) | 5–15 sec (offline) |
| Overlay accuracy | High on contours | High on texture |
| GPU requirements | None (WebGL in browser) | Yes (GPU class T4+) |
| Realism | Medium (smoothed) | High (accounts for lighting) |
| Application scope | Live streams, AR filters | Photo try-on in app |
Why Hybrid Approach is More Effective Than Pure AI?
For real-time scenes, latency is critical: MediaPipe offers a 100x speed advantage over generative models. At the same time, positioning accuracy for lips, eyelids, and cheekbones is high enough that jagged edges can be smoothed with bilinear filtering. We guarantee a stable 30 FPS frame rate even on mid-range mobile devices. For photos, where quality trumps speed, we use Stable Diffusion XL Inpainting with prior face segmentation—the model receives a mask of application zones and generates texture accounting for skin relief and incident light. This produces a living makeup effect, not a flat overlay. Processing time is 5–15 seconds per image, acceptable for user photos in an app.
How We Ensure Real-Time at 30 FPS?
Key is using WebGL for rendering and an optimized WASM build of MediaPipe. We avoid CPU-GPU data copying by directly accessing textures via WebGL2. As a result, on a mid-range smartphone (Snapdragon 865+) we get stable 30 FPS with under 5 ms latency. Additionally, we apply bilinear filtering for edge smoothing and adaptive bitrate for the video stream.
MediaPipe Approach (Real-Time)
import mediapipe as mp
import cv2
import numpy as np
from PIL import Image
class RealTimeMakeupAR:
def __init__(self):
self.face_mesh = mp.solutions.face_mesh.FaceMesh(
static_image_mode=False,
max_num_faces=1,
min_detection_confidence=0.5,
min_tracking_confidence=0.5
)
LIPS_INDICES = [61, 185, 40, 39, 37, 0, 267, 269, 270, 409, 291, 375, 321, 405, 314, 17, 84, 181, 91, 146]
UPPER_LID_L = [362, 382, 381, 380, 374, 373, 390, 249, 263, 466, 388, 387, 386, 385, 384, 398]
CHEEKS_L = [36, 31, 228, 229, 230, 231, 232, 233, 244, 245, 188, 174, 177, 215, 213, 192]
def apply_lipstick(self, frame, color, opacity=0.6):
rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
results = self.face_mesh.process(rgb)
if not results.multi_face_landmarks:
return frame
landmarks = results.multi_face_landmarks[0]
h, w = frame.shape[:2]
lip_points = np.array([
[int(landmarks.landmark[i].x * w),
int(landmarks.landmark[i].y * h)]
for i in self.LIPS_INDICES
], dtype=np.int32)
overlay = frame.copy()
cv2.fillPoly(overlay, [lip_points], color[::-1])
return cv2.addWeighted(frame, 1 - opacity, overlay, opacity, 0)
How AI Generative Try-On Works?
For photos, where quality matters more than speed, we use Stable Diffusion XL Inpainting with prior face segmentation. The model receives a mask of application zones (lips, eyelids, cheekbones) and generates texture accounting for skin relief and incident light. This produces a living makeup effect, not a flat overlay. Processing time is 5–15 seconds per image, acceptable for user photos in an app.
AI Generative Approach (High Quality)
from diffusers import StableDiffusionXLInpaintPipeline
import torch
from PIL import Image
import io
class AIPhotoMakeupTryOn:
def __init__(self):
self.pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
"diffusers/stable-diffusion-xl-1.0-inpainting-0.1",
torch_dtype=torch.float16
).to("cuda")
self.face_parser = FaceParser()
def apply_makeup_ai(self, photo, makeup_style, intensity=0.7):
img = Image.open(io.BytesIO(photo)).convert("RGB")
face_mask = self.face_parser.get_makeup_zone_mask(img)
result = self.pipe(
prompt=f"beautiful makeup, {makeup_style}, natural skin texture",
negative_prompt="unnatural, heavy, clown, overdone",
image=img,
mask_image=face_mask,
strength=intensity,
num_inference_steps=30,
guidance_scale=8.0
).images[0]
buf = io.BytesIO()
result.save(buf, format="PNG")
return buf.getvalue()
Pipeline Tuning Details
To boost realism, we fine-tune the model on a dataset of 5000 professional beauty photos with diverse skin tones and lighting. We use LoRA adapters, which speeds up training and reduces model size. Inference runs on GPU class T4 using TensorRT for latency optimization.Development Stages of Virtual Try-On
| Stage | Duration | Result |
|---|---|---|
| Requirements analysis and reference gathering | 3–5 days | Technical specification, interface prototypes |
| Architecture selection and model preparation | 5–7 days | Choice of landmark-based or AI generative approach |
| Detection and rendering implementation | 7–10 days | Working prototype in browser or app |
| Integration with product catalog | 3–5 days | API connection, shade synchronization |
| Testing and optimization | 5–7 days | Device testing, performance tuning |
| Deployment and documentation | 2–4 days | Rollout, team training |
What is Included in the Work
Our certified experience in Computer Vision allows us to deliver turnkey projects. As a result, you receive:
- documentation: architecture diagram, API description, integration guide
- source code with comments and CI/CD pipeline
- team training on system operation
- support during industrial operation (SLA of at least 99.9%)
Product Architecture
Web/Mobile App
├── Camera/Photo mode
│ ├── Real-time: MediaPipe WASM + WebGL (< 5ms)
│ └── Photo: AI inpainting API (10–15 sec)
├── Product catalog (lipstick, eyeshadow shades)
├── Shade picker + color mixer
└── Share / Download result
Timelines: browser real-time AR makeup (MediaPipe) — 3–4 weeks. AI photo try-on API — 1–2 weeks. Full mobile app with catalog — 3–4 months.
Browser Real-Time Implementation
import * as faceMesh from '@mediapipe/face_mesh';
import * as drawingUtils from '@mediapipe/drawing_utils';
const faceMeshInstance = new faceMesh.FaceMesh({locateFile: (file) => {
return `https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh/${file}`;
}});
faceMeshInstance.setOptions({
maxNumFaces: 1,
minDetectionConfidence: 0.5,
minTrackingConfidence: 0.5
});
function renderLipstick(ctx, landmarks, color, opacity) {
const lipIndices = [61, 185, 40, 39, 37, 0, 267, 269, 270, 409, 291, 375];
ctx.globalAlpha = opacity;
ctx.fillStyle = color;
ctx.beginPath();
lipIndices.forEach((idx, i) => {
const lm = landmarks[idx];
if (i === 0) ctx.moveTo(lm.x * canvas.width, lm.y * canvas.height);
else ctx.lineTo(lm.x * canvas.width, lm.y * canvas.height);
});
ctx.closePath();
ctx.fill();
ctx.globalAlpha = 1.0;
}
Contact us for a pilot project — get a demo within 5 days. Request a consultation — we'll select the optimal architecture for your tasks.
[^1]: MediaPipe Face Mesh — library for detecting and tracking facial keypoints.