AI Agent for Desktop Application Automation

Desktop applications without an API are every automation engineer's classic headache. Legacy ERP, CAD programs, bank clients, 1C in thick client mode. Selenium can't see them, there's no REST API. We solve this with an AI agent that either uses Computer Use (screenshot + mouse/keyboard control) or accesses the accessibility API (pywinauto, UI Automation) — a more reliable option for Windows applications. In this article, we'll break down how to build such an agent, the pitfalls, and how much it saves.

Why an AI Agent Instead of RPA?

Classical RPA systems (UiPath, Blue Prism) require rigid UI markup and break on any update. An LLM-based AI agent understands the UI at the semantic level: it doesn't need a predefined selector — it analyzes the element tree and decides on an action. This reduces maintenance costs by 60% over the long term — 3x more resilient to UI changes than RPA. Typical projects start at $5,000 and pay back within 3 months. We rely on the Windows UI Automation API and pywinauto documentation.

How We Build Such an Agent?

At the core is a pywinauto + Anthropic API bundle. pywinauto accesses the Windows UI Automation API — the same technology used by screen readers. Elements are located by accessibility attributes (AutomationId, Name, ControlType), not by pixels. Significantly more reliable than screenshots.

from anthropic import Anthropic
import pywinauto
from pywinauto.application import Application
from pywinauto.findwindows import ElementNotFoundError
import json
import subprocess
import time

client = Anthropic()


class DesktopAppAgent:
    """AI agent for Windows desktop app automation"""

    def __init__(self, app_path: str = None, app_title: str = None):
        self.app_path = app_path
        self.app_title = app_title
        self.app = None
        self.main_window = None

    def launch_or_connect(self):
        """Launches app or connects to a running instance"""
        try:
            if self.app_title:
                self.app = Application(backend="uia").connect(title_re=f".*{self.app_title}.*")
            elif self.app_path:
                self.app = Application(backend="uia").start(self.app_path)
                time.sleep(2)  # wait for init
        except pywinauto.findwindows.ElementNotFoundError:
            if self.app_path:
                self.app = Application(backend="uia").start(self.app_path)
                time.sleep(2)

        self.main_window = self.app.top_window()

    def get_ui_tree(self, max_depth: int = 4) -> dict:
        """Gets UI element tree"""
        def extract_element(element, depth=0):
            if depth > max_depth:
                return None
            try:
                info = {
                    "name": element.window_text()[:100] if element.window_text() else "",
                    "control_type": element.element_info.control_type,
                    "automation_id": element.element_info.automation_id or "",
                    "enabled": element.is_enabled(),
                    "visible": element.is_visible(),
                    "rect": str(element.rectangle()),
                }
                children = []
                for child in element.children():
                    child_info = extract_element(child, depth + 1)
                    if child_info and (child_info["visible"] or child_info["enabled"]):
                        children.append(child_info)
                if children:
                    info["children"] = children[:20]  # max 20 children
                return info
            except Exception:
                return None

        return extract_element(self.main_window)

    def find_and_interact(self, instruction: str) -> str:
        """LLM determines which element is needed and what to do"""
        ui_tree = self.get_ui_tree()

        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=512,
            messages=[{
                "role": "user",
                "content": f"""Analyze the UI tree and return a JSON with the action:
{{
  "action": "click|type|select|get_value|find",
  "automation_id": "element ID if any",
  "name": "element name",
  "control_type": "element type",
  "value": "value for type/select"
}}

Instruction: {instruction}

UI tree:
{json.dumps(ui_tree, ensure_ascii=False)[:4000]}

Only JSON."""
            }],
        )

        try:
            text = response.content[0].text
            action = json.loads(text[text.find("{"):text.rfind("}") + 1])
            return self._execute_ui_action(action)
        except Exception as e:
            return f"Parsing error: {e}"

    def _execute_ui_action(self, action: dict) -> str:
        """Executes action on UI element"""
        try:
            # Search by automation_id or name
            element = None

            if action.get("automation_id"):
                element = self.main_window.child_window(
                    auto_id=action["automation_id"]
                )
            elif action.get("name"):
                element = self.main_window.child_window(
                    title=action["name"],
                    control_type=action.get("control_type"),
                )

            if not element:
                return "Element not found"

            act = action.get("action", "click")

            if act == "click":
                element.click_input()
                return f"Clicked on {action.get('name', action.get('automation_id'))}"

            elif act == "type":
                element.set_edit_text(action.get("value", ""))
                return f"Typed: {action.get('value', '')}"

            elif act == "select":
                element.select(action.get("value", ""))
                return f"Selected: {action.get('value', '')}"

            elif act == "get_value":
                return element.window_text() or element.get_value()

        except ElementNotFoundError:
            return f"Element not found: {action}"
        except Exception as e:
            return f"Error: {type(e).__name__}: {e}"

        return "Action executed"


class DesktopWorkflowAgent:
    """High-level agent for executing tasks in a desktop app"""

    TOOLS = [
        {
            "name": "get_ui_state",
            "description": "Gets the current UI tree state of the application",
            "input_schema": {"type": "object", "properties": {}},
        },
        {
            "name": "interact_with_element",
            "description": "Interacts with a UI element (click, type, select)",
            "input_schema": {
                "type": "object",
                "properties": {
                    "automation_id": {"type": "string"},
                    "action": {"type": "string", "enum": ["click", "type", "select", "get_value"]},
                    "value": {"type": "string"},
                },
                "required": ["action"],
            },
        },
        {
            "name": "wait",
            "description": "Waits for a state change in the application",
            "input_schema": {
                "type": "object",
                "properties": {
                    "seconds": {"type": "number", "default": 1.0},
                    "wait_for_element": {"type": "string"},
                },
            },
        },
        {
            "name": "keyboard_shortcut",
            "description": "Presses a key combination (Ctrl+S, Alt+F4, etc.)",
            "input_schema": {
                "type": "object",
                "properties": {
                    "shortcut": {"type": "string", "description": "E.g., Ctrl+S, Alt+Tab, F2"},
                },
                "required": ["shortcut"],
            },
        },
    ]

    def __init__(self, desktop_agent: DesktopAppAgent):
        self.agent = desktop_agent

    async def run(self, task: str) -> dict:
        messages = [{"role": "user", "content": task}]
        steps = 0

        while steps < 40:
            response = client.messages.create(
                model="claude-sonnet-4-5",
                max_tokens=1024,
                system="You are a desktop app automation agent. Use tools sequentially.",
                tools=self.TOOLS,
                messages=messages,
            )

            tool_results = []
            done = False

            for block in response.content:
                if hasattr(block, "text") and block.text:
                    if "done" in block.text.lower() or "completed" in block.text.lower():
                        done = True

                elif block.type == "tool_use":
                    result = ""
                    inp = block.input

                    if block.name == "get_ui_state":
                        result = json.dumps(self.agent.get_ui_tree(max_depth=3), ensure_ascii=False)[:3000]

                    elif block.name == "interact_with_element":
                        result = self.agent._execute_ui_action(inp)

                    elif block.name == "wait":
                        time.sleep(inp.get("seconds", 1.0))
                        result = "Waited"

                    elif block.name == "keyboard_shortcut":
                        import pyautogui
                        keys = inp["shortcut"].replace("+", " ").split()
                        pyautogui.hotkey(*[k.lower() for k in keys])
                        result = f"Pressed {inp['shortcut']}"

                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": result,
                    })

            if done or response.stop_reason == "end_turn":
                return {"success": True, "steps": steps}

            messages.append({"role": "assistant", "content": response.content})
            messages.append({"role": "user", "content": tool_results})
            steps += 1

        return {"success": False, "steps": steps}

Which Approach to Choose: pywinauto vs Computer Use?

We use both approaches: pywinauto as primary, Computer Use as fallback. If the app has an accessible UI model, we prefer pywinauto; otherwise, Computer Use.

When is Computer Use Needed as a Fallback?

Computer Use helps when the app uses custom graphics or doesn't provide an accessibility tree — for example, old CAD systems or data visualizers. In such cases, the agent takes a screenshot of the area and passes it to the LLM for analysis. However, it's 2x slower and less reliable: p99 latency is 3x higher and the model may misjudge coordinates. Therefore, we use Computer Use only where pywinauto is powerless.

Practical Case: Automating 1C:Accounting from Our Practice

Task: Monthly generation of 40 reports in 1C for 12 legal entities. The process took 3 working days for two accountants.

Approach: pywinauto for 1C desktop client (version 8.3). UI Automation works with 1C through COM objects and accessibility API.

Results:

• 40 reports × 12 entities: 3 working days → 4 hours (nightly run) — 80% reduction in processing time
• Manual input errors: reduced to 0
• Challenge: 1C periodically changes automation_id on updates — we added fallback search by element name

Key point when working with 1C: the app uses a custom engine; not all elements are visible through standard UI Automation. Part of the automation is implemented directly through 1C's COM interface. Our experience shows that a hybrid scheme (pywinauto + COM) yields the best result.

What's Included in the Work?

1. UI tree analysis — identifying accessible elements, creating an automation map.
2. Agent development for a specific workflow — implementing action sequences with fallback logic.
3. LLM integration — selecting the model (Claude or GPT), setting up prompts for stable JSON command generation.
4. Batch processing + monitoring — scheduled runs, error alerting.
5. Documentation — architecture description, run instructions, maintenance.

Estimated Timelines

We guarantee agent stability throughout the entire operational period — if the application updates, we adjust selectors and prompts. We've been working with desktop automation for over 5 years, completed more than 20 projects for banks, retail, and logistics.

Get a consultation for your scenario — we'll estimate the project in 1–2 days. Contact us, and we'll suggest the optimal turnkey solution.

LLM Development: Fine-Tuning, RAG, Agents, and Production Deployment

Using GPT‑4 or Claude 3.5 Sonnet through a public API is not a solution — it's just a tool. When the requirement is to "make it like ChatGPT, but on our data," there is a real engineering challenge behind it: from prompt engineering to training a 70B model on your own infrastructure. End-to-end LLM solution development is a complex stack, and we have been doing it for over 5 years. During this time, we have completed over 20 projects in generative AI: from RAG systems for legal departments to custom support agents. Where exactly your task falls depends on data, latency requirements, budget, and how critical confidentiality is.

A typical situation: the client has already tried ChatGPT, but results are unstable — sometimes accurate, sometimes hallucinating. Or they need integration into a corporate portal while complying with security policies. Let's break down each layer of the stack in detail — from RAG to production deployment.

Why Do RAG Systems Break and How to Fix It?

RAG (Retrieval-Augmented Generation) looks simple: find relevant documents, put them in context, get an answer. In practice, it fails in several places.

Chunking without overlap. Classic mistake: chunk_size=512, overlap=0. If the answer lies across two chunks, retrieval won't find either with sufficient confidence. Solution: overlap 15–25% of chunk_size, or better yet, sentence-aware splitting with spaCy or NLTK instead of naive character splitting.

Poor embedder. text-embedding-ada-002 is good for general use, but on legal or medical texts, specialized models like E5-large-v2, BGE-M3, or fine-tuned sentence-transformers on domain data outperform it. Recall@5 differences can be 15–25%.

No re-ranking. Vector search optimizes for speed, not relevance. A cross-encoder re-ranker (ms-marco-MiniLM-L-6-v2, bge-reranker-large) after initial retrieval improves top-3 accuracy with acceptable latency (+50–150ms). This is often more impactful than improving the embedding model.

Hybrid search. Dense vectors alone work poorly on exact queries: names, SKUs, codes. BM25 (sparse) finds exact matches but misses semantics. Hybrid via RRF (Reciprocal Rank Fusion) is the optimal compromise. Qdrant, Weaviate, and pgvector 0.7+ support hybrid search natively.

When to Fine-Tune Instead of Prompt Engineering?

Prompt engineering solves ~70% of LLM adaptation tasks for a domain. The remaining 30% require fine-tuning. Three indicators: the model ignores a specific output format even with detailed prompting; the task requires deep knowledge of specialized vocabulary (medicine, law); you need to significantly reduce token costs by replacing a large model with a smaller specialized one.

LoRA and QLoRA are the standard for SFT. LoRA adds trainable low-rank matrices to attention layers. A typical configuration for Llama-3 8B: r=64, lora_alpha=128, target_modules=["q_proj","v_proj","k_proj","o_proj"] yields ~0.8% trainable parameters, training on one A100 40GB. QLoRA adds 4-bit quantization (NF4) and allows fine-tuning 70B models on two A100 40GB, though speed drops by half compared to bf16.

DPO instead of RLHF. Direct Preference Optimization requires only (chosen, rejected) pairs, not scalar reward signals. DPOTrainer from the trl library (Hugging Face) implements it in a few dozen lines.

Common mistake. A dataset of 500 examples, 5 epochs, validation loss 0.8 — seems fine. But on test, the model degrades on general instructions. Cause: catastrophic forgetting. Solution: add 10–20% general instruction-following examples (Alpaca, FLAN) to the training set to preserve original capabilities.

How to Choose a Base Model: 8B or 70B?

For most tasks, fine-tuning an 8B model is sufficient. 70B is needed when deep reasoning is required or the 8B baseline does not reach the required quality even after fine-tuning. Inference cost for Llama-3 8B via vLLM on A100 is efficient; the exact cost depends on volume.

What Does PagedAttention Bring to Production?

vLLM is the first choice for serving open-source models. PagedAttention is the key technical innovation: KV-cache is managed like virtual memory in an OS, without fragmentation. This yields 2–4x higher throughput compared to naive HuggingFace Transformers inference. The vLLM documentation confirms that continuous batching and PagedAttention are the standard for high-load LLM services.

Typical numbers on A100 80GB for Llama-3 8B (bf16): 400–600 req/s, P50 latency 200–400ms, P99 latency 600–900ms at concurrency 64. For 70B on two A100 with tensor parallelism: 80–120 req/s, P99 latency 1.5–2.5s. AWQ or GPTQ quantization reduces memory consumption by 2x with quality loss within 1–3%.

Multi-Agent Systems

Agents are LLMs with access to tools: search, code execution, API calls, database interaction. Common patterns:

• ReAct (Reason + Act): the model reasons → chooses a tool → observes the result → reasons again. LangChain and LlamaIndex implement it out of the box.
• Multi-agent orchestration: multiple specialized agents with a coordinator on top. Example: coordinator → researcher (search + summarization) → coder (code generation and execution) → critic (verification). Tools: AutoGen (Microsoft), CrewAI, custom implementation on LangGraph.

In production, agent systems are non-deterministic. Essential: guardrails, step limits, logging of each step, human-in-the-loop for critical actions.

How We Work: Stages, Timeline, Deliverables

What Is Included

We deliver:

• Technical documentation (model card, configs, deployment instructions)
• Access to infrastructure (code repository, trained weights)
• 1 month of post-deployment support (consultations, bug fixes)
• Customer team training (2–3 sessions on system operation)

Timeline: basic RAG prototype — 1–2 weeks. Fine-tuning with customer data — 3–6 weeks (including data preparation). Production system with monitoring and retraining — 2–4 months. Cost is calculated individually based on data volume, model complexity, and infrastructure requirements.

We guarantee the quality of the final model with performance benchmarks and ongoing monitoring. Our engineers have hands‑on experience with dozens of production LLM systems.

Want to evaluate your project? Leave a request — we will prepare a preliminary summary within 1–2 business days. Or get a consultation on choosing the approach: RAG, fine-tuning, or hybrid — we will tell you what works best for you. Contact us to discuss your LLM development needs. Schedule a free consultation today.