Analytical agents on TaskWeaver: where operations break
An analytical agent on TaskWeaver, written for one report, lives only until the first new question. Logic, sources, and export format are hardcoded directly into the code cell: to add a metric or change a slice, the scenario is rebuilt rather than extended.
After a few iterations there are already five working versions, and no one knows which one is considered current.
Worse still with calculation reproducibility. Cells execute in a live session; call order and intermediate tables are not recorded anywhere—the result depends on when data was pulled and what remained in memory from the previous step.
The same query run a day later produces a different figure, and the dispute is not about the method but about whose run is "correct."
The third failure is understanding someone else's scenario. There is no schema: which plugin reads what, where queries are cached, by what rules empty results are discarded. Understanding a finished report takes days, and it breaks at the very first change in source structure.
All three failure points lie not in the model but in operations—and they appear after agent implementation, when the first report is delivered.
We examine how this is closed: contracts at each step's output, fixed input slices, and scenario versioning, so that a rerun produces the previous result.
What a managed agent layer provides: measurable effects
The effect of a managed agent layer is verified by measurement, not by the impression from a demo.
We record baseline values for four quantities before start and repeat the measurement after a month of work on real data—otherwise "it became more convenient" can neither be confirmed nor disputed. This is what the acceptance checklist looks like for an environment with TaskWeaver.
| What we measure | Before implementation | After | How we calculate |
|---|---|---|---|
| Scenario assembly speed | 2–3 weeks: approvals and manual code for each report | 2–4 days from task description to first run | Calendar time to launch on production data |
| Share of requests without an engineer | 20–30%, the rest goes into the development queue | 70–80% the analyst closes independently in the interface | Labeling of requests over a month |
| Failure class | Silent output of a plausible but incorrect figure | The agent returns a failure and writes the reason to the log | Share of answers with a confirmed error |
| Reporting automation | 3–5 person-days per regular report | Assembly on a schedule, a person checks only outliers | Time from period close to finished report |
To keep the checklist from becoming a formality, three things are needed before start:
- baseline measurement: without it, there is no way to prove the shift;
- a fixed set of test queries, including deliberately unsolvable ones;
- a person responsible for acceptance on the business process side, not only on the development side.
Reducing failures is more important here than speeding up: a scenario that honestly stops and asks for data saves more time than one that confidently produces an incorrect result.
TaskWeaver deployment options: environment, team, hybrid
The TaskWeaver deployment scheme determines not launch speed but which data the agent is allowed to see at all and who is responsible for the result of its calculations.
The difference between modes lies not in the engine itself but in where data, models, and query execution logs are stored.
Before choosing, we fix three parameters: the class of processed data, the number of people working with the agent simultaneously, and load predictability. They determine whether to keep computations inside the perimeter or move part outside.
Comparison of modes
| Variant | When suitable | Limitations |
|---|---|---|
| Isolated environment | Personal data, trade secrets, requirement not to let queries leave the perimeter | Models and dependencies are updated according to regulations, not automatically; the ceiling on parallel requests is set by local resources |
| Team server | Shared scenarios, several analysts, need for a unified agent library and role-based separation | Single perimeter: access to individual datasets is configured more finely, and the cost of a permissions error is higher |
| Hybrid scheme | Sensitive data remains local, heavy computations are moved to an external environment | Two environments to synchronize; explicit control is needed over which fields and fragments leave the perimeter |
What to consider during transition
Transition between modes is feasible without rewriting scenarios: we describe the agent declaratively and move the binding to models and storages into environment configuration.
This allows starting with a team server and then moving the sensitive block into an isolated environment without stopping analysts' work.
Implementation stages: from scenario audit to environment handover
The work order is the same for any scenario: first we analyze what already exists in data and reports, then design the schema, and only then configure.
This order removes the main risk—building the environment around a hypothesis rather than real sources, only to discover at the first check that some fields do not reconcile. Below is the implementation framework for TaskWeaver that we follow on every project.
- Scenario and source audit. We collect the list of questions the team regularly asks of the data and look at where the figures come from. The output is a prioritized list of scenarios and a source map.
- Environment schema design. We break each scenario into steps: query parsing, access to tables and files, calculation, response format. We separately fix where the agent calculates itself and where it must call an external tool.
- Environment and access setup. We deploy the environment, connect sources, grant rights, and enable logging. Every access to data remains in the log—you can see what the answer is based on.
- Calculation verification on historical data. We run analytical steps on past periods and compare with reference reports. We investigate discrepancies at control points down to the cause rather than masking them with rounding.
- Behavior verification on failures. Unavailable source, empty selection, ambiguous query: we define in advance whether the agent clarifies, returns a partial answer, or refuses. We describe boundaries explicitly.
- Documentation and environment handover. We hand over schemas, configurations, failure handling rules, and an operations manual, train the team, and support the first period of independent work.
Handover package: configurations, access policies, regulations
The TaskWeaver handover package is not a folder of files but a description of how the system lives in operation. We assemble it so that your team can redeploy the environment, add a scenario, or investigate an incident without contacting us.
- Configuration files — agent launch parameters, connections to data sources, limits on execution time and sample size, with comments on every non-obvious value.
- Environment schema — which services the system consists of, where network boundaries run, how data gets inside, and what does not leave.
- Access and role separation policies — who adds a scenario, who sees raw data, who receives only final analytics. Rights are described by roles, not by people, so they survive staff changes.
- Scenario library — verified analytical queries with descriptions of input data, limitations, and examples of correct results.
- Support regulations — update order, actions when a source is unavailable, who is notified and by what deadline.
- Documentation — instructions for the analyst and administrator, including a log of known limitations and workarounds.
- Control examples — reference questions and expected answers: after a configuration change, you immediately see that agent behavior has not changed.
The package remains in your environment. With it, you can bring up the system from scratch, transfer support to another team, or expand the scenario library on your own.
Case: analytical environment for reporting before and after
Before: manual assembly and discrepancies between sources
The client's analytics service assembled regular reporting manually: exports from several accounting systems were consolidated into tables, and duplicates and discrepancies between sources were cleaned by eye.
The consolidated report took 2–3 days to prepare, each time from scratch—the calculation logic lived in the analyst's head and in scattered files. Any change to the methodology meant a complete recalculation and the risk of not reconciling with the previous period.
What was changed and what resulted
Regular reporting was moved to an agent layer: TaskWeaver was taken as the basis because it separates step planning and data work—the agent builds a plan, calls the necessary functions, and returns not only a number but also the chain of calculations.
Sources were described as separate tools, and reconciliation rules and the metric dictionary were fixed in configuration rather than in scripts.
- Preparation of the consolidated report: 2–3 days → 3–4 hours, including verification.
- Share of reports without manual edits: from ~30% to 90%.
- Connecting a new source: weeks → days, without rewriting logic.
- Verifiability: every metric has a step log—a discrepancy is investigated step by step rather than by a full recalculation.
- Shift handover: the methodology is described in configuration; a new analyst reaches independent work in a matter of days.
We do not disclose the client's data—we provide only the metrics agreed with their department.
How TaskWeaver differs from a custom script-based agent
A custom-built agent is a pre-written chain of steps: the developer anticipates possible scenarios and separates them with branches. While there are two or three queries, the scheme is predictable. But each new data source adds a branch, and routing logic grows faster than it pays off.
TaskWeaver separates what is usually mixed in scripts—plan and code. The agent's task planner builds the plan as a data structure: steps, dependencies, expected result.
Each step gets its own code and executes separately, so the plan can be checked, approved, and adjusted step by step without rewriting the entire scenario.
The agent's tools are described declaratively: connecting a new storage or reference dictionary means adding one tool with a description and parameters, and the planner decides where to apply it. In the custom-built variant, this is a new script plus edits to routing and surrounding code.
Execution is isolated: step code runs in a separate session; its variables—intermediate tables and results—live within one task and are not carried further.
An error is localized to a specific step rather than bringing down the whole chain, and is visible in the log next to the code.
The selection threshold is simple: for a narrow, stable task, a set of scripts is simpler and less expensive to maintain. When combinations of sources and phrasing exceed ten, the approach with a planner and reusable tools wins.
Common technical doubts: point-by-point answers
Before a pilot of an analytical agent on TaskWeaver, clients ask the same set of questions: what to do with already written calculations, how to limit access to data, how to make sure the agent calculates correctly, and whether the infrastructure will withstand query growth. We answer each briefly.
We have our own calculation scripts—will they have to be rewritten?
No. Existing calculations are wrapped in tools that the agent calls as ordinary functions: formulas and logic remain yours; only the launch method changes.
We migrate one calculation at a time and compare output with a reference run—a discrepancy is visible on the very first test.
How does the agent get access to sensitive data?
Under a separate service account: read-only and only the necessary views, without exporting raw arrays in the response. If data must not leave the perimeter, the entire calculation remains inside the perimeter—only aggregates leave.
How are calculation errors caught?
Numbers come from code execution, not from model-generated text. On top—control checks: reconciliation of totals with an independent source, acceptable value bounds, and a repeat run on the same input data.
What about load as the number of queries grows?
The agent runs as a service with a limit on concurrent tasks: heavy calculations go into a queue, and results are reused by input parameters. A peak does not hit the storage, and response time remains predictable.
Let's discuss your task: what to send for assessment
To assess an implementation, you do not need a finished project or a detailed technical specification—initial data about the task is enough.
We work with analytical agents based on TaskWeaver, so it is important to understand not the infrastructure but the scenario itself: what decisions the agent must make and based on what data.
What is sufficient to send from the client side:
- task description: what decisions are currently made based on analysis and what is slow or expensive about it;
- 2-3 real examples of input data—exports, tables, reports, emails, scans in the form in which they arrive, without preliminary cleaning;
- examples of correct answers that the analyst prepares manually: we use them to fix acceptance criteria;
- list of data sources, their owners, and required access—read or write;
- environment restrictions: which data must not leave the perimeter and where the solution is allowed to be deployed;
- a client specialist who will confirm the correctness of the agent's answers on the validation set;
- how often the analysis is run and in what form the result is needed—report, table, message in a work channel.
The discussion proceeds as a task breakdown: we clarify scenarios, boundaries, and quality requirements, after which we send an implementation assessment, work plan, and team composition.
If some data is missing, we will suggest how to collect it in the first stage; this is a normal situation.
