AI Inpainting in Mobile Apps: From Mask to Result
You take a great photo on your iPhone, but a passerby sneaks into the frame. In the app, you want to erase them with a couple of taps — that’s a job for AI inpainting. Technically, the solution consists of three stages: create a mask, send the image to the model API, apply the result. Each stage demands precision, or you’ll get artifacts or slow loading. Let’s break down how to implement this practically, with code and configs.
Inpainting is a technology that replaces a region of an image based on its context. In mobile apps, we use cloud APIs (DALL·E 2, Stable Diffusion) to generate content. But the key challenge is drawing the mask on the device. The quality of the mask determines 80% of the result. An inaccurate mask causes the model to replace too much or leave object traces. So the first task is building an intuitive drawing tool.
Why Mask Quality Determines the Inpainting Result?
A mask is a binary image where white pixels denote the editing area. The model uses it as a hint. If the mask extends beyond the object, the model “fill in” adjacent elements, creating unnatural transitions. If the mask is too narrow, parts of the removed object remain. Hence, precise drawing with zoom and undo is essential.
How to Implement Mask Drawing on iOS?
On iOS, we use CAShapeLayer and UIBezierPath. On touch, we draw circles along the movement path, merging them into one path. The mask becomes a black‑and‑white image: black background — unchanged area, white strokes — region of interest.
class MaskDrawingView: UIView {
private var maskLayer = CAShapeLayer()
private var path = UIBezierPath()
private var brushSize: CGFloat = 30
override func draw(_ rect: CGRect) {
UIColor.black.setFill()
UIRectFill(rect)
UIColor.white.setFill()
path.fill()
}
override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) {
guard let touch = touches.first else { return }
let point = touch.location(in: self)
let circle = UIBezierPath(arcCenter: point, radius: brushSize / 2, startAngle: 0, endAngle: .pi * 2, clockwise: true)
path.append(circle)
setNeedsDisplay()
}
func getMaskImage() -> UIImage {
UIGraphicsBeginImageContextWithOptions(bounds.size, false, 0)
layer.render(in: UIGraphicsGetCurrentContext()!)
let image = UIGraphicsGetImageFromCurrentImageContext()!
UIGraphicsEndImageContext()
return image
}
}
For smoothness, we use UIGraphicsImageRenderer and optimize rendering via displayLayer. On iPad with Apple Pencil, the mask is more precise — we can increase the path resolution.
How to Implement Mask Drawing on Android?
On Android, we use a custom View with Canvas and Paint. We draw circles on a separate bitmap, which is then passed as a mask. It’s important to adjust anti‑aliasing and brush size according to screen density.
class MaskDrawingView(context: Context) : View(context) {
private val maskBitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
private val canvas = Canvas(maskBitmap)
private val paint = Paint().apply {
color = Color.WHITE
style = Paint.Style.FILL
strokeWidth = brushSize
isAntiAlias = true
}
override fun onTouchEvent(event: MotionEvent): Boolean {
when (event.action) {
MotionEvent.ACTION_MOVE -> {
canvas.drawCircle(event.x, event.y, brushSize / 2, paint)
invalidate()
}
}
return true
}
}
For performance, we use Bitmap.Config.HARDWARE on Android 8+ and avoid unnecessary copies.
How to Integrate DALL·E 2 Inpainting?
DALL·E 2 only accepts PNG with an alpha channel, where transparent pixels denote the editing area. So the mask is embedded into the image’s alpha channel. The maximum request size is 4 MB. We send multipart/form‑data:
func inpaint(image: UIImage, mask: UIImage, prompt: String) async throws -> UIImage {
guard let imageData = image.pngData(), let maskData = mask.pngData() else {
throw InpaintError.invalidImage
}
var request = URLRequest(url: URL(string: "https://api.openai.com/v1/images/edits")!)
request.httpMethod = "POST"
request.setValue("Bearer \(apiKey)", forHTTPHeaderField: "Authorization")
let boundary = UUID().uuidString
request.setValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")
var body = Data()
// image (must be PNG, RGBA, max 4 MB)
body.appendMultipart(boundary: boundary, name: "image", filename: "image.png", contentType: "image/png", data: imageData)
// mask (PNG, RGBA, transparency = editing area)
body.appendMultipart(boundary: boundary, name: "mask", filename: "mask.png", contentType: "image/png", data: maskData)
// prompt
body.appendMultipart(boundary: boundary, name: "prompt", data: prompt.data(using: .utf8)!)
// size (must match input image size)
body.appendMultipart(boundary: boundary, name: "size", data: "1024x1024".data(using: .utf8)!)
body.append("--\(boundary)--\r\n".data(using: .utf8)!)
request.httpBody = body
let (data, _) = try await URLSession.shared.data(for: request)
let response = try JSONDecoder().decode(ImageResponse.self, from: data)
// Download result
let (imageData2, _) = try await URLSession.shared.data(from: URL(string: response.data[0].url)!)
return UIImage(data: imageData2)!
}
DALL·E 2 limitation: only accepts PNG with alpha channel (RGBA). The mask is passed via transparency: transparent pixels = editing area. Not a black‑and‑white mask like in SD, but an alpha channel. Maximum file size is 4 MB.
What Alternatives Exist? Stable Diffusion Inpainting
Stable Diffusion inpainting via Replicate accepts base64 for image and mask. The strength parameter controls the depth of replacement: 1.0 — full replacement, 0.5 — soft blending with original content. This gives more flexibility than DALL·E 2.
val body: [String: Any] = [
"version": "...", // SD inpainting model
"input": [
"prompt": prompt,
"image": "data:image/jpeg;base64,${imageBase64}",
"mask": "data:image/png;base64,${maskBase64}",
"num_inference_steps": 25,
"guidance_scale": 7.5,
"strength": 0.99
]
]
Comparison of inpainting models:
| Model | Input Format | Max Size | Notes |
|---|---|---|---|
| DALL·E 2 | PNG with alpha channel | 4 MB | High quality, strict format requirements |
| SD via Replicate | JPEG/PNG base64 | 10 MB | Flexible, adjustable strength, lower cost per request |
How to Handle Images of Different Sizes?
Before sending to the API, the image is resized to the required size (e.g., 1024×1024 for DALL·E 2). We use aspect fill to maintain proportions and crop excess. After generation, the result is overlaid onto the original at the mask coordinates. For this, we preserve the original dimensions and preview position.
func resizeAndCrop(_ image: UIImage, to size: CGSize) -> UIImage {
UIGraphicsBeginImageContextWithOptions(size, false, 1.0)
let aspectFill = max(size.width / image.size.width, size.height / image.size.height)
let newSize = CGSize(width: image.size.width * aspectFill, height: image.size.height * aspectFill)
let origin = CGPoint(x: (size.width - newSize.width) / 2, y: (size.height - newSize.height) / 2)
image.draw(in: CGRect(origin: origin, size: newSize))
let result = UIGraphicsGetImageFromCurrentImageContext()!
UIGraphicsEndImageContext()
return result
}
How to Ensure Smooth UX During Loading?
An API request takes 1 to 5 seconds depending on the model and image size. We show a loading animation with progress (for SD, we can track via polling). On iOS, use SwiftUI ProgressView; on Android, CircularProgressIndicator. After receiving the result, we apply a smooth cross‑fade transition.
What Our Integration Work Includes
- Analysis of your requirements and selection of the optimal API (DALL·E 2, Stable Diffusion, or a combination).
- Development of a mask drawing UI with undo/redo, brush size adjustment, and zoom.
- Integration of the chosen API with error handling, retries, and caching.
- Image transformation: resizing, cropping, and overlay of the result.
- Documentation and training for your team.
- 3 months of post‑release technical support.
Contact us for a technical assessment of your project — we will help select the appropriate API and implement the integration in the shortest possible time. We work with a result guarantee and adherence to deadlines. Get a consultation — send us a description of your task, and we will evaluate it as part of a free audit.
Implementation Timeline
- Basic integration (one API + mask drawing screen) — 5 to 8 days.
- Full editor (undo/redo, multi‑API, overlay) — 3 to 4 weeks.
Timelines are refined after analyzing your project.