JupyterHub setup: where GPU quotas break
Default JupyterHub setup looks harmless: one image, a node with accelerators, each engineer spins up their own server and works. While there is one team, the scheme holds.
As soon as a second or third group connects to the environment, the shared ML development environment turns into a competition zone: model training grabs all available accelerators and holds them until the last epoch, while the next researcher sees that the resource is busy.
Then the familiar degradation begins. Resource limits for teams are not set, so a notebook session with a forgotten hyperparameter sweep consumes accelerator memory entirely.
GPU sharing between teams rests on verbal agreements in chat, the task queue on accelerators is not built — runs either fail due to lack of memory or wait days for release. Engineers idle, and the experiment plan shifts not by hours but by days.
A separate loss is manual transfer of results between tools. Artifacts, metrics, and launch parameters remain in local notebook directories, they are transferred by hand, and part disappears on server restart.
Comparing two runs between teams is no longer possible, and reproducing someone else's result even less so. As a result, the same hypothesis is tested again because the previous report was not found.
We break down how this is closed: quotas for accelerators per team and per user, a priority queue instead of "first come" capture, and a unified experiment tracker where the run result goes automatically.
What a controlled environment gives: measurable effects
A controlled development environment differs in that each of its properties is verified without manual administrator actions. Below are five points that we close during JupyterHub setup, and how the effect is measured.
Ready environment in minutes
A workplace comes up from a pre-built profile with the required library branch, rather than being assembled on the spot from raw dependencies. Environment startup time drops from hours to minutes, and a new engineer starts working on day one.
Resource isolation between teams
Accelerator quotas are calculated by groups: one team does not occupy others' cards and does not bring down others' training. A class of failures due to node overload disappears.
Transparent resource consumption
You can see who holds how many accelerators and where idle capacity is. The impact of cluster idling stops being unknown — unused quotas are visible and can be redistributed.
Experiments without manual transfer
Parameters, metrics, and artifacts go to the tracker directly from the notebook along with the code version. Experiment accounting does not depend on whether the researcher remembered to transfer results by hand.
Predictable recovery
Environment definitions and quotas are described declaratively and stored in the repository. After a failure, the node returns to the same state reproducibly.
Deployment options: comparison of approaches
The choice of option is determined by the load profile: how many people work simultaneously, how often heavy training jobs are launched, and whether data isolation between teams is needed.
The same JupyterHub on a shared node and on separated resources behaves differently — on a shared node, one training run crowds out everyone else, and MLflow metrics go with them.
| Option | When it fits | Limitations |
|---|---|---|
| Shared node | small team, peaks are rare, everyone works with the same data | shared resource: heavy training crowds out colleagues, quotas are set manually, no data isolation |
| Dedicated environment for teams | multiple teams with different access rights and their own datasets | higher maintenance overhead: environments are updated separately, environments diverge over time |
| Scheduler with load separation | uneven load, there is a queue of training jobs and interactive sessions | requires configured priorities and limits, otherwise the queue blocks interactive work |
| Hybrid resource scheme | interactive kept on a permanent environment, training goes to temporary resources | observability is harder: logs and metrics need to be consolidated from two places |
In practice, we start with a shared node and strict quotas: this is fast and predictable while the team is small. As soon as a second team or job queue appears, we move to separation — by environments or through a scheduler.
Analyzing the load profile is the main step: it determines where you will be in six months, not today.
JupyterHub setup: stages from audit to handover
The order of work almost does not depend on team size — only the amount of resources to allocate differs. Below is the sequence in which we conduct implementation: each step relies on data from the previous one, so a skipped audit comes back as reworking the quota scheme.
- Audit of ML team load. We look at how many people work simultaneously, which tasks require accelerators, and how long compute sessions live. Output is a consumption profile with peak hours, not average monthly figures.
- Quota and role scheme. We distribute resources by groups and roles: researcher, engineer, platform administrator. We tie quotas to the task profile, not to a person — a new team gets an environment without manual settings edits.
- Environment deployment. We deploy JupyterHub, prepare images for typical stacks and separately for accelerator tasks, and enable automatic shutdown of idle sessions. Forgotten notebooks stop holding resources in the queue.
- Connecting the experiment tracking system. We link the environment to the run storage: parameters, data versions, and artifacts enter the shared log automatically. A result is reproduced from a record, not from the author's memory.
- Testing under peak load. We reproduce simultaneous startup of the maximum number of sessions, observe scheduler and storage behavior, and measure response time. Limits are adjusted before handover, not after the first failure.
- Handover and team training. We provide documentation, launch templates, analysis of typical errors, and a regulation: how to request resources, who changes quotas, what to do when a node fails.
What we hand over: configurations and regulations
JupyterHub setup ends not with launching the service, but with handing over to the customer's team everything needed for independent operation.
We put artifacts in the repository and format them so a new engineer can figure them out in a day, and changes go through review rather than edits directly on the server.
- Environment configurations — deployment manifests, proxy and storage settings, domain and certificate binding; deployment is repeated with one command.
- Resource allocation schemes — profiles with GPU and memory quotas, priorities between teams, per-user limits, and release of idle resources.
- Access policies and roles — administrator, researcher, observer, integration with corporate directory, granting access to secrets and external data.
- Environment templates — images with pinned library versions for training and for model inference, instructions for adding your own package.
- MLflow integration — tracking server address, artifact storage, linking experiments to the author, rules for moving a model to the production environment.
- Environment documentation — component map, where to look at logs and metrics, typical failures, headroom for number of users.
- Incident escalation regulation — criticality levels, response time, decision owner, procedure for rolling back to the previous configuration.
We close the handover with scenario runs: restore a node from backup, revoke a departed employee's access, bring up training after a failure. They pass without us — the environment is accepted.
Case: how we reduced environment downtime for AI teams
The customer is a product team with four areas: model research, data preparation, implementation, and support of running services.
Everyone shared one pool of machines with accelerators, and access was granted manually: a request in chat, waiting for a slot, setting up the environment for a specific task.
The downside of such a scheme is downtime. A new engineer waited days for a working environment, and a hung session could hold an accelerator for days while others queued.
Another class of failures was non-reproducibility: environments were assembled manually, and an experiment result was not repeated on another machine.
What we changed
We deployed JupyterHub as a single entry point: one address, login via corporate account, ready environment profiles for typical tasks. We introduced roles — researcher, engineer, administrator — with different rights and image sets, and accelerator quotas for each area.
Idle sessions release resources automatically, and runs and metrics are written to the shared experiment tracker.
Metrics before and after
Time from request to working environment dropped from two or three days to minutes. Downtime due to waiting for resources fell by about five times, and the share of runs with saved metrics grew from a third to almost all.
The "accelerator occupied by someone else's session" failure class disappeared: quotas and auto-release removed both the queue and manual bargaining for machines.
For the administrator, this is a role change: instead of a dispatcher manually handing out machines, they configure policies and images. Teams get a predictable start, and the area lead gets a visible load picture for each profile.
MLflow is already deployed: how to connect the environment?
A deployed experiment tracking server is a ready source of truth: it contains runs, metrics, and artifacts that the team's conclusions rely on. Therefore, the development environment is not rebuilt or moved, but connected to the running environment as another participant.
Nothing changes in the current scheme: the server stays in place, storage is the same, accesses are the same.
Connection is done at the environment image level: the server address and access token are fixed in environment variables and substituted at startup. The notebook writes runs to the same place the rest of the team sends them, not to a local file that no one will find later.
If the server lives in an internal environment, a private route is added — the environment sees the service without exposing it externally.
History is not lost: connection does not affect storage, new runs are appended to already accumulated ones, and previous records remain available for comparison. Project names and experiment markup are preserved, so a past run can be opened and continued directly in the notebook.
Compatibility is checked before rollout — with a test run on one notebook, verifying metrics and artifacts in the shared dashboard.
Tool compatibility is covered by library versions: the image pins the same branches that engineers already use, and login goes through a single account. The usual workflow remains — notebooks launch as before, only now they are linked to shared experiment tracking.
The result is that the development environment and tracking system work as one environment, without data transfer or re-accumulation of results.
Technical doubts: analysis of frequent questions
Doubts at the implementation stage usually concern not launch but operation: what happens when several teams with different loads come to one cluster, who gets accelerators first, and what happens to training that has been running for two days.
Below is an analysis of each point; all of this is solved by policies in the configuration, not by manual administrator intervention.
We check on a pilot with a real load profile: two or three teams, nodes of different classes, and experiment history from the previous environment. If behavior diverges from expected, we adjust policies before rolling out to the entire fleet.
How are accelerators divided if there are fewer machines than people who want them?
A quota per group plus queue priority: short sessions pass ahead, long-running ones wait and receive a notification. The scheduler does not give one team the entire fleet, even if its sessions started first.
Nodes of different classes are described by separate launch profiles, so the team chooses a suitable one itself instead of occupying the most powerful by default.
Does a hung notebook hold an accelerator?
Lifetime is tied to kernel activity, not to start time: idle hours close the session, and the working folder remains on disk. Additionally — a limit on simultaneous sessions per user and a warning with a delay before shutdown, so no one loses computation results.
How is role-based access arranged?
Groups from the corporate directory are pulled automatically: the administrator manages images and quotas, the researcher launches their sessions, the observer sees metrics without access to data.
Secrets are issued at the profile level and do not end up in the notebook; actions are recorded in the log.
Is experiment history transferred?
Runs from MLflow move together with parameters, metrics, and artifacts. The server address is substituted into the environment, so past runs open from new notebooks without code changes — comparison of results before and after the transition remains correct.
Let's discuss the task: we'll put together a plan for your load
To discuss the task substantively and come back with an estimate of implementation timelines, rather than generalities, we need initial data to start.
We will go through some points together on a call, and you will prepare some in advance — the accuracy of the work plan for the environment depends on this.
- Load profile. How many engineers work simultaneously and which tasks prevail: model training, fine-tuning, or short experiments. From here — node configuration and quota policy.
- Accelerators and access. Which machines are allocated for compute, whether there is free capacity, and who grants access to the cluster. We need a contact of an engineer who will confirm network restrictions.
- Experiment tracking. Whether you keep a log of runs, model versions, and artifacts, and whether integration with the storage already in use is needed. If history has been accumulated — we decide whether to transfer it or start over.
- Rights and roles. Who is allowed which resources, whether quotas by project are needed, who administers users, and how access is revoked when an employee leaves.
- Data. Where datasets are stored, what volume they are, and whether they can be moved between environments. This determines the scheme for mounting storage into JupyterHub working environments.
- Project scope. Who makes decisions on the customer side, the desired launch window, and what we consider the result of the first iteration.
Based on the conversation, we will come back with an estimate of timelines and a work plan for the environment: stages, areas of responsibility, verification procedure. We fix the agreed scope of work in writing before the start — this way the first launch passes without rework.
