Setting up Neptune.ai: why experiments are not reproducible
Setting up Neptune.ai begins not with launching a server, but with answering a question: why yesterday's training run no longer repeats.
A typical picture: metrics improved, the model was accepted, and a week later an attempt to retrain the same configuration yields a different result. Parameters were edited in a notebook, the dataset was swapped along the way, the environment was updated in the background.
As a result, experiment tracking comes down to scattered logs: some in the console, some in spreadsheets, some in chat, and the chain of decisions can no longer be restored.
The root of the problem is that three things are not fixed: artifact versions, hyperparameters, and the environment. A dataset without a checksum, a checkpoint without a link to a commit, dependencies without a list — and model reproducibility rests on the engineer's memory.
The scale of the problem is industry-wide: according to Survey on Machine Learning Reproducibility, most specialists have encountered a situation where someone else's or their own past result could not be repeated without contacting the author.
The cost of such uncertainty is measurable in engineering terms.
Comparing two runs takes hours of manual matching instead of one query; rolling back to a successful configuration is impossible because it is unknown what exactly it was; finding the cause of degradation turns into archaeology through logs.
When versions of data, code, and parameters are stored next to metrics, every run becomes a reproducible point rather than a memory.
We break down how this is addressed: what exactly Neptune.ai setup fixes, how to link artifacts to runs, and what checks should be put in place so that an experiment can be repeated months later.
Bringing order to experiments: reproducibility and iteration speed
After setting up experiment tracking — on Neptune.ai or a similar tracker — it is not reporting that changes, but the speed of work. Finding the configuration that gave the best result a month ago takes minutes instead of hours of chat and digging through folders.
The share of reproducible runs stops depending on who performed them: parameters, data version, and environment are fixed together with the result.
Comparing runs and data versions
Each run writes metrics by training steps, a set of parameters, and a link to the dataset version — by content, not by file name. Runs land on the same charts, and the discrepancy is immediately visible: where the model diverged and which parameter pulled it up.
Previously, a successful result could not be linked to the data on which it was obtained, and half of the iterations were spent re-searching what had already been covered.
Model and artifact registry
A finished model enters the registry together with weights, configuration, and metrics and lives through stages — draft, review, production, archive.
You can see who moved the version further and when, and on which metrics that decision was made; returning to the previous one is one operation instead of restoring from backups.
The output is that the time to find a working configuration drops from days to hours, reproducibility becomes a rule rather than luck, and there are no lost runs — each is tied to data and code.
This is the main effect: the team spends time on new hypotheses, not on restoring already-tested ones.
Neptune.ai setup options: cloud, local perimeter, hybrid
Neptune.ai setup options differ not by interface, but by what data the team has the right to move outside the perimeter.
Run metadata, model artifacts, and datasets can be separated across different perimeters — then tracking remains end-to-end, and data isolation does not turn into manually moving files between environments.
| Option | When it fits | Limitations |
|---|---|---|
| Cloud scheme | The team works distributed, and data does not fall under internal restrictions on external storage | Access control and coordination with the security service are needed; uploading large artifacts runs into the communication channel |
| Fully local perimeter | Sensitive datasets, regulator requirements, or a closed perimeter with no external access | Backup and storage updates remain on the customer's side, and capacity for peak runs is planned in advance |
| Hybrid scheme | Some experiments may be run in the cloud, while some must remain inside | Two points of failure and synchronization between perimeters: a regulation is needed for what exactly goes outside |
Where the data boundary runs
The boundary is set not by artifact type, but by sensitivity class: metrics and run parameters remain in the cloud, datasets and model weights — in local storage.
An agent inside the perimeter sends only aggregated indicators outside, so loss of observability is minimal, and export of raw data is excluded technically rather than by policy.
Implementation stages: from process review to handover of regulations
Setting up Neptune.ai begins not with installing the service, but with reviewing how the team trains models today.
Until it is clear which runs are significant and where the connection between code, data, and result is lost, automation will produce a beautiful but useless feed of charts. Implementation stages proceed in an order where each next step relies on the previous one.
How the audit is conducted
Engineers look at repositories, run logs, and ways of recording parameters and metrics. Separately, we check what happens to intermediate weights and datasets — that is where reproducibility is most often lost. The result is a map of runs and a list of gaps.
- Audit of training processes: who launches training, which parameters they change, where they store results.
- Design of the tracking scheme: which metrics, parameters, and files are linked to a run, how projects are named.
- Setting up tracking: recording training curves and system indicators, grouping runs, comparison.
- Connecting artifact storage: model weights and configurations are stored with versions in one place.
- Testing the scheme on a real run: we run training and verify that the data is recoverable.
- Acceptance: the team goes through a checklist from comparing runs to rolling back to a previous model version.
- Handover of regulations: instructions, naming rules, procedure for working with experiment history.
Verification and acceptance
There is one criterion: any run is reproduced and explained by the stored data. If a metric is in the report but is not linked to the code and data version, the work is not accepted.
Configurations, schemes, and regulations: what remains with the team
As a result of the setup, the team receives not just a working server, but a set of artifacts: from them the perimeter can be raised again, handed over to another engineer, and checked without the implementation author. Below is what exactly is recorded in writing and in what form it goes to the customer.
Documentation and regulations
- Tracking configuration files in Neptune.ai — connection parameters, the set of logged metrics, rules for grouping runs by projects and tags.
- Experimental perimeter scheme: the data path from training to run comparison, failure points, and places where manual control is needed.
- Run templates for typical tasks — training, fine-tuning, model comparison — with pre-filled tags and parameter sets.
- Documentation: launch instructions, project structure, and naming rules — a new engineer is brought into the perimeter in a day, not a week.
- Maintenance regulation: update procedure, retention period for old runs, actions when metrics diverge, and escalation points.
Access policies and roles
- Roles and rights: read, run, and administration are separated — an experimenter cannot delete someone else's project or rewrite history.
- Access policies: tokens, their lifetime, binding to service accounts, and revocation procedure when an employee leaves.
- Action log: who changed the configuration and when — analyzing a metric discrepancy does not turn into a search for someone to blame.
The set is designed so that the team then works independently: expands the set of metrics and adds projects without rebuilding the perimeter from scratch.
Case study: how the search for a working model configuration was shortened
Implementation case study: the customer's team trained a recommendation model, and experiments were conducted in notebooks, spreadsheets, and chat. We deployed a unified run registry on their side and put work with it on a regular basis.
What it was before
The parameters of each run were stored with the author: some in a spreadsheet, some in chat, some in their head.
Finding a working configuration took up to two days — logs were matched manually, and understanding which combination produced the required quality was possible only by re-running.
Fewer than half of the experiments remained reproducible: data and environment versions were not fixed. The share of duplicates reached one third — the same hypothesis was tested two or three times without knowing about the previous result.
What it became after
Runs were linked to versions of data, code, and environment in Neptune.ai. Finding a working configuration dropped from two days to three hours: the needed run is found by filtering by parameters and metrics, and comparison is done along identical axes.
The share of reproducible runs grew from 45% to 95% — the result is raised from the registry together with the environment.
Duplicate runs decreased threefold, manual notes almost disappeared: parameters and metrics are written automatically, and only conclusions on hypotheses remain in the registry.
What to do if experiments are already run in another tool?
A full one-time migration is rarely needed: while active series of runs are underway, the risk of breaking links between runs is higher than the benefit from speed.
Therefore, we first build coexistence of perimeters — the old tool remains working, Neptune.ai is raised alongside, and both write to an agreed entity scheme.
The basis of parallel work is a unified naming model: project, series, run, step, artifact set.
We assign each run a stable key from the dataset name, configuration fingerprint, and parameters, so a record from one perimeter is unambiguously projected into the other, and comparison of metrics between sites remains correct.
History migration proceeds in two passes. First, import of results by metadata — project structure, links of runs to steps, comments, and conclusions. Then heavy data: metric curves, charts, model checkpoints.
Repeated import updates the record rather than creating duplicates, so an interrupted export can be completed from the same place.
Active research does not stop: unfinished runs are written to completion in the old perimeter, while new ones start in the new one.
After convergence, we compare a control sample of experiments — if the key metrics match, we merge the perimeters into one, and leave the former in history read mode.
After that, it is enough to maintain unified naming rules for new runs — otherwise, in a couple of months, history will again diverge across two unrelated databases.
Technical doubts: access, load, data security
Doubts about access, load, and storage come up before the start, not after the first incident. We collected four questions that engineering teams ask most often.
How to differentiate access between departments?
We assign roles at the workspace and project level: a data engineer sees the runs of all teams, an analyst sees only their own. External integrations and build systems receive read keys; write keys are short-lived and rotated on a schedule.
Will metric collection slow down training?
Metrics and artifacts are sent asynchronously: logs are buffered and sent in batches from a background thread, and training does not wait for them. Overhead stays within fractions of a percent of epoch time; logging is disabled with one flag in the run configuration.
Where is experiment data stored?
The option is fixed at the start: your perimeter on your own capacity or dedicated cloud storage. Metadata is stored in a database, heavy artifacts — in object storage with access via signed links; retention period and cleanup are set by a separate rule.
What about support after handover?
We hand over documentation, deployment schemes, and a maintenance regulation: key rotation, version updates, recovery from backup. For the first period, we stay in touch and handle incidents together with your team; after that — support on request.
Let's discuss the task and come back with an implementation plan
An accurate estimate of the scope of work becomes possible only after we see your current model training picture: where runs are launched, who compares results and how, what is already set up in Neptune.ai, and what lives in spreadsheets and chat.
Therefore, we start discussing the task with inputs from the customer, not with a cost calculated before the audit.
What is useful to prepare for the conversation:
- current training process: how runs are launched — manually or from an orchestrator, where artifacts and metrics end up;
- volume of run history — how many experiments and over what period need to be migrated so that comparison is meaningful rather than selective;
- data storage requirements: artifact lifetimes, placement region, who has access to sensitive datasets;
- team composition — how many engineers and analysts will work in the system and with what rights;
- desired deployment perimeter: cloud, customer's closed perimeter, or hybrid, and what network restrictions exist;
- current visualization and reporting tools, so as not to duplicate what already works;
- system operating mode — a one-time history migration or ongoing tracking of new experiments.
In response, we send an estimate of the scope of work and an implementation plan: sequence of stages, scheme of perimeters and storages, composition of roles and access permissions, acceptance points.
The next step is a review based on your data, after which the labor effort and order of work are clear. Describe the task — we will come back with an estimate and plan.
