# Which workloads live on a narrow link

> Training large models over wide-area networks is a working class of architectures. A 12B model was trained across four data centres at 0.84 Gbit/s.

Source: https://naasson.com/en/blog/workloads-on-a-narrow-link/
Published: 2026-05-07
Language: en
Direction: Lab by Naasson (https://naasson.com/en/lab/)
Publisher: Naasson — https://naasson.com

---
The objection to a data centre in the mountains always sounds the same: without a fast link there is nothing to compute there. The objection was valid until a class of architectures appeared that is designed specifically for slow links between nodes.

## What does not work, and why

Training a frontier model on a single campus needs connectivity on the order of four hundred gigabits per second between tens of thousands of accelerators. Tensor parallelism exchanges activations on every forward and backward pass, which is only acceptable inside a node, on the chip-to-chip interconnect. Pipeline parallelism tolerates tens of gigabits, but it is still a job for a single campus.

Mixture-of-experts models additionally need all-to-all exchange between experts, and published measurements show it can take almost half of the execution time. Such models are not split across data centres at all.

We do not claim this class of work. Training the largest models end to end is a job for a single campus, and a mountain site does not compete for it.

## What does work

A separate class of algorithms synchronises not on every step but once every few hundred steps. What is exchanged is not gradients but the accumulated change in weights, and traffic drops by two orders of magnitude.

<Stat items={[
  { value: "0.84 Gbit/s", label: "between sites in a published experiment" },
  { value: "12B", label: "parameters in the model trained this way" },
  { value: "1–10 Gbit/s", label: "provided by the satellite stage of our site" }
]} />

Published results in this area: training with eight workers matched the fully synchronous version while exchanging five hundred times less data. Later variants added staggered synchronisation of parameter subsets, continued training during exchange, and quantisation of transmitted data, cutting link requirements by another two orders of magnitude in total.

In one of the most recent published experiments, a twelve-billion-parameter model was trained across four data centres, and bandwidth requirements were reduced to 0.84 gigabits per second between eight sites, while keeping a high share of useful work even at noticeable failure rates.

Our first connectivity stage is satellite, one to ten gigabits per second. It meets these requirements. The second stage, fibre, exceeds them by a wide margin.

## Where most of the market is

A substantial share of commercial demand for compute, measured in GPU hours, is not training the largest models. It is fine-tuning, reinforcement learning, batch inference, synthetic data generation, scientific computing and rendering.

<Callout type="tip" title="What these workloads have in common">
They are all weakly sensitive to external bandwidth. Data is brought in once, and after that the workload is bound by the accelerators themselves. Link latency is not on the critical path.
</Callout>

Fine-tuning a large model with low-rank adaptation takes hours on a single accelerator, with weights of a few tens of gigabytes when quantised. Bringing that volume in over satellite is a question of time, not of possibility.

## What stays outside the first stage

Inference with a guaranteed response time below fifty milliseconds. Here physics is on the side of the objection: satellite latency cannot be fixed with algorithms.

We assign this class of workloads to the second connectivity stage and do not promise it earlier. The difference between "not possible now" and "never possible" matters, but so does the difference between "possible after fibre" and "possible today".

## Why this is a technical justification, not a sales argument

We cite publications and specific measured values, not general arguments about distributed training. A client planning to place a workload should be able to verify the claim and, more importantly, verify whether it applies to their own workload.

The answer "it depends on your communication profile" sounds evasive. It is also the only correct one: a site's suitability is determined not by its link, but by how much of that link your job needs.