ControlNet: why generation doesn't hold composition
A text prompt sets the meaning of the frame, but it barely controls its geometry. The angle, pose, and position of an object within the scene are described in words approximately — and on each run they come out different. The image turns out convincing, while the composition is random.
The reason lies in the nature of generation itself: the picture is assembled from noise, and text only shifts the result toward the desired content.
Nothing holds the object boundaries, horizon line, or gaze direction — so sample-based generation gives "similar, but not the same." Controlling composition through the prompt remains probabilistic: change the wording of the hint or the seed — and the angle drifts apart.
The trade-off of such freedom is manual frame finishing. The designer selects one successful variant out of dozens, cleans artifacts, and draws in missing details.
An unstable result does not scale: in a series of layouts in a unified style, each frame has to be corrected separately, and a repeated run does not reproduce the previous one.
ControlNet closes exactly this gap. An additional spatial signal is fed into the model — a contour, pose skeleton, depth map, or boundary markup — and geometry is held separately from content.
Composition stops being guessed: generation follows the given sample, and edits are reduced to style rather than rebuilding the scene. Below we break down how this works and what it gives in practice.
What sample-guided generation gives
Generation from a text description gives a different result from run to run: composition floats, the object leaves the frame, and the batch has to be re-selected from scratch.
Sample-guided control removes this uncertainty — ControlNet transfers the geometry of the reference into generation: contour, pose, horizon line, frame grid. Variability remains where it is needed — in light, material, and style.
The effect of the transition is measured not by "beauty of the image," but by the share of accepted frames and the time per batch.
Therefore, before implementation we record baseline values on a sample of tasks, and after we compare the result on the same sample — numbers, not impressions.
- Share of frames accepted on the first pass: grows by multiple, because geometry is set by the sample, not guessed by the model from the description.
- Number of iterations until approval: one or two edits per frame instead of the cycle "generated — rejected — rewrote the description."
- Time per batch: reduced by fewer re-generations and no manual angle finishing.
- Defect class by geometry: angle violations, cropped objects, and broken proportions go away; only meaningful style edits remain.
- Repeatability of the result: one sample and one description produce a series matching in composition — the batch is assembled without rebuilding the scene.
- Composition accuracy: framing and object position match the approved sample, so new scenes are assembled from already accepted references.
We record these six checks in the report for each batch — this shows where sample-guided control had an effect and where the task is solved by editing the sample itself.
ControlNet modes: comparison of approaches and limitations
ControlNet adds an additional control input to the generative model: composition stops being a lottery and is set by a conditioning map.
The mode determines exactly what is held — object geometry, skeleton, scene volume, or zone boundaries — and this affects which artifacts will have to be removed in post-processing.
The wrong mode choice is more damaging than an unsuccessful prompt: the image turns out technically clean, but incorrect in construction.
Modes are chosen from the task, not by the principle "looks more impressive." Below are the alternatives between which one actually has to choose on a project.
| Mode (conditioning map) | When it is suitable | Mode limitations |
|---|---|---|
| Object contour | Precise geometry is needed: product, mockup, drawing, logo on a surface | Pulls noise and texture from the source; on highlights, fur, and hair the contour breaks, and double lines appear |
| Pose map | Character in a given position, series of identical angles, "person in frame" shooting | Sets the skeleton, but not volume or clothing; hands, feet, and intersections of arms with the body break most often |
| Depth map | Volume and order of planes: interior, complex scene, background replacement while preserving perspective | Does not hold small details or precise boundaries; closely standing objects merge into one plane |
| Area boundaries | Repainting and replacing materials by masks, retouching by zones, fill control | Accuracy equals mask quality; on gradients and translucency the boundary "floats" |
Modes are not mutually exclusive: a pose map plus a depth map simultaneously hold both the person's position and the room's perspective.
The weight of each map is selected so that the control signal does not overpower the text prompt — otherwise the image becomes rigid and repeats the source.
Separately, we check that the mode does not break small details at the output resolution: some defects are visible only in a crop.
The common limitation of all modes — they control composition, but not identity. If a specific person or recognizable product is needed, separate identity fixation and palette control are added to the conditioning map.
Below we break down how this is assembled into one process and where the boundary lies between controlled generation and regular generation.
Implementation stages: from task analysis to handover
We structure the order of work so that the result is reproducible, not dependent on a lucky run. First comes the process audit: we analyze how source materials are prepared and scenes are assembled, where rework arises, and how many iterations go into one approved frame.
Based on the results, we fix the control scheme — which parameters are set rigidly and which remain at the operator's discretion.
- Process audit — the path from task statement to approved frame, bottlenecks, and typical deviations from expectations.
- Control scheme — ControlNet layer map: what sets composition, what sets pose and perspective, and what remains to the generator.
- Setup for the product range — training on the client's objects and materials: angles, lighting, typical categories.
- Control sample — a run on a held-out set of scenes with comparison of the result to the reference before launch into the flow.
- Operating regulations — instructions for operators: source preparation, permissible deviations, actions on defects.
- Acceptance and handover — access, configurations, trained models, documentation, and training of the client's team.
Each stage ends with an artifact that can be checked: layer map, set of test scenes, instruction, configurations. We hand over not only the result but also the ability to reproduce it — afterward the client's team works independently.
Configurations, profiles and regulations at the output
Development ends not with a launched model, but with a set of artifacts with which the client's team works further without us. We hand over for ControlNet-based generation: from profiles for specific shooting scenarios to the criteria by which the result is accepted.
- Sets of configurations for task types. Separate profiles for product photography on a white background, a complex scene with pose preservation, and interior redrawing: fixed weights of control layers, trigger thresholds, order of steps. A repeated run on the same input gives a comparable frame.
- Connection scheme. What prepares the control signal, which model builds the base, in what sequence they are connected, and at what step the operator's manual editing is plugged in.
- Access policies and roles. Who changes weights and profiles, who only launches generation, who works with source materials. Core edits pass review by a second engineer, and change history is preserved.
- Operator instruction. Step-by-step launch scenario and analysis of typical failures: composition drifted, artifacts appeared at object boundaries, shooting angle changed.
- Reference set of inputs and outputs. Several dozen fixed pairs on which, after each update, a regression check is run and the result is compared to the reference.
- Acceptance rules. Checklist: composition match with the source, contour cleanliness, stability on repeated runs, whether we fit within the agreed processing time.
- Versioning regulations. How profile versions are numbered, where the reason for a change is recorded, and how to roll back to the previous configuration build.
Such a set removes dependence on one specialist: changes are made consciously, not by trial and error. Describe the task — we will return with an estimate and a plan.
Case: ControlNet in catalog product photography
One client's catalog contains several thousand items that need to be shown in a unified composition: the same angle, scale, product position in the frame, and similar light and shadow.
Before implementation, batches were assembled manually, and a noticeable share of frames went to reshoot: the product ended up shifted to the edge, shadows did not match the reference, and scale "floated" between batches.
We assembled a composition control framework on ControlNet. For each product type, we fixed a reference layout — object contour, depth map, and frame boundaries — and fed it as a control signal for generation, not as a wish in the text description.
Metrics before and after
The share of frames sent to reshoot decreased from about 30% to below 5%. We prepare a batch of one hundred items in one working day instead of three or four, and batch stability holds: the position, scale, and placement of the product match from the first card to the last.
New catalog items fit into the same framework without rebuilding the scene and without reconfiguring for each SKU.
An honest limitation of the scenario: gloss, transparent, and mirrored surfaces are poorly described by a control contour — for them we left a separate route with manual finishing.
Frames for reshoot there do not go entirely, but their share stopped affecting the batch delivery deadline.
How does ControlNet differ from simple re-generation?
Repeated generation is the same prompt run again: the starting noise changes, and with it the entire frame. The styling is recognizable, but object positions, angle, and perspective drift.
The mode is good when composition is not important — when you need to collect variants of texture and lighting for selection.
Fixing the seed removes variability altogether: the output is the same frame, and edits go only through the prompt text. But this does not give geometry control — the seed determines what the noise will be, not where the object will stand. The wording "shift left" will work approximately and differently from run to run.
ControlNet solves a different task: it is an external control scheme that accepts an additional input — depth map, contour, pose markup — and mixes it into generation at intermediate steps. The condition is set before rendering, not selected after a dozen attempts.
Hence the difference in predictability. With an external scheme, the frame framework is set by the input map: on repeat, object boundaries, the horizon line, and the place for a product or text are preserved; only texture and details change. Composition is approved once, and then execution is iterated.
Comparison of approaches in practice: repeated generation — when a search for mood and style is needed; an external scheme — when there is an approved layout, a pose reference, or a strict requirement for element placement.
They are usually combined: the map sets geometry, and its influence strength is weakened toward the end of generation so that the material does not look stenciled.
Technical doubts and frequently asked questions
Technical doubts about ControlNet arise not in a demo, but when it is time to put the solution into production: will it withstand a flow of similar tasks, will there be enough data, will an already debugged process have to be rebuilt. Below are four questions our engineers hear most often.
How accurately are pose and composition repeated?
The control map sets geometry: skeleton position, object contours, depth. The model follows it independently of the rest of the description, so deviation by key points between runs is measured in single-digit percent.
If variability is needed, the condition weight is lowered by a parameter — there is no need to retrain the model for this.
How much training data will be needed?
For accurate pose repetition, no training is needed at all — a control map is enough. An own dataset is required when a style or a specific object is fixed: here several dozen approved samples suffice.
We start with a small set and add data only where the similarity metric does not reach the threshold.
Won't batches diverge in style and color?
Uniformity holds because all samples in a batch pass through one model version and one set of control maps. The reference is fed to the input, not described in words, so hue, light, and angle do not "float" from frame to frame.
Generation parameters are fixed in the configuration — the result is reproducible on a repeated launch.
Is it compatible with an already configured process?
The module connects as a separate processing step and does not require changing the input data format. We hand over configurations, schemes, and launch policies so that your team can maintain the solution without us.
Describe the task — we will return with an estimate and an implementation plan.Let's discuss the task: what is needed for an estimate and plan
To return with a task estimate and an implementation plan, rather than general wording, we need engineering context: how the process is structured now and which requirements for the result are critical.
Below is what to describe in the first message; this is enough to break the work into stages and give real timelines.
- Task and result. Which process we automate and what we consider success: composition stability, speed of material preparation, or the share of results accepted without edits.
- Source data. Examples of current images and references, dataset volume, whether there is markup of pose, depth, segmentation — this determines which control maps we build.
- Composition requirements. What must be strictly observed (object position, perspective, background style), and where variability is acceptable. ControlNet sets the framework, but the set of maps is selected for your scene.
- Integration and infrastructure. Where generation is embedded — editor, service, content preparation pipeline — where launch is assumed, and what data restrictions apply.
- Regulations and acceptance. Mandatory quality checks, data isolation requirements, and who on the client side accepts the result.
- Priorities and work timelines. What is needed first, whether there is a fixed launch date, and what can be moved to the second wave.
After the analysis, we send an implementation plan: stages, scope of work, control points, and a timeline estimate for each. Describe the task in free form — we will return with a plan and clarifying questions.
