Lab: A Mixed-Platform Cluster

Validated on: written from the documentation cited above; not yet validated on hardware on any track. The backend each host contributed, the split proportions and the failure behaviour observed will be recorded here when the validation pass has run this lab on the reference cluster.

Objective

The previous lab used a cluster for the thing everyone builds one for: memory. This one is about the thing that makes llama.cpp’s RPC backend unusual, which is that the machines do not have to match. By the end you will have run one model across two different GPU backends, and you will have four measurements that a uniform cluster cannot teach you.

• The same model, same cluster, split by memory and then split by speed, so you can see which choice the default makes and what changing it is worth.
• The cost of one slow host, measured rather than assumed, by adding a deliberately slow one.
• What the client actually does when a machine disappears in the middle of a generation.
• A written topology: which machine held which layers, on which backend, over which link.

The model is Qwen3-30B-A3B at Q4_K_M, a single file of about 18.6 GB. On two machines of 16 GB each it needs the pair, which keeps the exercise honest: this is not a model you could have run on either machine alone.

⌂Single-machine path

One machine. Run one ggml-rpc-server restricted to the CPU device beside your ordinary GPU, split a smaller model between them, and you have a genuinely heterogeneous cluster on one box. Use Qwen3-8B at Q4_K_M, which fits the 8 GB floor.

This path is better than it sounds for this particular lab, because the interesting measurement here is what a slow participant does to the whole run, and a CPU device beside a GPU is the most dramatic version of that experiment you can arrange. What it cannot show you is a network, so the failure-behaviour task uses a stopped process rather than a pulled cable. Say which you did in the notebook.

Architecture

Any two tracks will do. The reference cluster’s most instructive combination is one CUDA host and one Vulkan host, because the two backends were compiled from the same source for different silicon and they are not equally fast per gigabyte held.

Two backends, one model, plus an optional slow third

• clientclientWhichever machine you sit at. Runs llama-server, holds the GGUF, contributes its own device.
• workersecond GPU hostA different track from the client: CUDA, Vulkan or Metal.
• workerdeliberately slow hostA CPU-only rpc-server, added in task 6 and removed afterwards.

• client ⟷ connected to second GPU hostwired Ethernet from Part 18
• client ⟷ connected to deliberately slow hostthe same link, carrying the same few kilobytes per token

The third host is added deliberately in task 6 to measure what a slow participant costs. It is not part of the baseline.

What has to match across the cluster is the model file and the llama.cpp build number. What does not have to match is the operating system, the processor architecture or the backend. That is the whole point of this lab, and it is the capability that no other clustering tool in this course offers.

Requirements

Two machines on different tracks, each with at least 16 GB, both with a llama.cpp build carrying -DGGML_RPC=ON, both on the network from Part 18’s lab, and the environment file from this part’s first lab. About sixty minutes, all attended. One download of about 18.6 GB, to the client only.

Qwen3-30B-A3B and Qwen3-8B are both Apache-2.0 licensed and neither is gated. The model reference records the licence for each.

Track S — NVIDIA DGX Spark

A Spark is the natural client here: it has the memory to hold the larger share and the CUDA build is the one most likely to be the faster participant. Pair it with the Ryzen AI Max+ box on Vulkan for the most interesting comparison in the reference cluster, or with a Mac on Metal.

Two Sparks are a uniform cluster, not a mixed one. If that is all you have, do this lab with one Spark and the CPU device of the other, which is the single-machine experiment stretched across a real network and is still worth doing.

Track X — AMD Ryzen AI Max+ 395

This is the track that makes the memory-versus-speed question real. A Ryzen AI Max+ has a great deal of memory and an integrated GPU that is not as fast per gigabyte as a discrete card or a Spark, so the default proportional split will hand it a large share of the layers and the cluster will move at its pace.

That is not a fault to fix; it is the measurement this lab is for. Run the default split, measure, then take layers away from this machine with --tensor-split and measure again. Remember the GPU-visible cap when deciding how much memory this host can really offer.

Build with -DGGML_VULKAN=1 -DGGML_RPC=ON. If you also built the HIP path in Part 6, run the whole lab twice, once with each backend on this host; that comparison is genuinely useful and is not published reliably anywhere.

Track M — Apple silicon

Metal joins a mixed cluster with no extra build flags beyond -DGGML_RPC=ON. A Mac is a good second host: the unified memory means the share you give it is the share it can take, with none of the caps the other tracks have.

A laptop on battery will thermally limit part-way through and produce two different answers to the same measurement. Plug it in before starting, and if the numbers drift, say so in the notebook rather than averaging them.

Track N — NVIDIA desktop or laptop

A desktop card is the fastest participant per gigabyte in most home clusters and the most constrained in capacity, which makes it the clearest illustration of why memory-proportional splitting is not always right: the default will give your fastest device the smallest share.

Task 5 is where you take that back. Build with -DGGML_CUDA=ON -DGGML_RPC=ON. Inside WSL2, remember that the virtual machine’s memory limit, not Windows’ memory, is what this host can offer.

Tasks

Working directory and terminal roles

Prepare the course execution workspace once before this procedure. It includes this part's scripts, data and shared Python helpers. In the client or training terminal, select this directory:

RunnableAll tracks

select this part’s execution directory

export LABS_ROOT="${LABS_ROOT:-$HOME/llm-course/labs}"
export LAB_DIR="$LABS_ROOT/part-19-llama-cpp-rpc"
cd "$LAB_DIR"
pwd
test -f "probe-rpc-devices.sh"

Expected result: pwd ends in part-19-llama-cpp-rpc and the file check returns successfully. If it does not, finish workspace preparation before continuing. Activate the environment in the requirements for your track. Bare script and data filenames below are relative to this directory; paths to earlier experiments must point at the artefacts you actually retained.

Keep each foreground server in a separate terminal and send requests from this terminal. Reapply lesson-specific environment variables in each new shell. Stop at the first failed checkpoint and retain its output; the execution guide explains how to distinguish missing files, endpoint failures and capacity problems.

1. Get the model and the environment onto the client

RunnableAll tracks

one file, on the client only

hf download unsloth/Qwen3-30B-A3B-GGUF --include "*Q4_K_M*" \
    --local-dir ~/models/unsloth/Qwen3-30B-A3B-GGUF

Copy rpc.env from the previous lab and change MODEL to point at the file you just downloaded, and RPC_HOSTS to the single second host you are using here. Everything else carries over.

2. Start the second host, and confirm which backend it brought

RunnableAll tracks

on the second host

. ./rpc.env
bash start-rpc-server.sh --background

The server’s own banner names the devices it found, and that is the first piece of evidence. The second is the client’s view:

RunnableAll tracks

on the client: what does each host offer?

. ./rpc.env
bash probe-rpc-devices.sh mixed-topology.md

Read mixed-topology.md. You should see your local device first, then the remote one, with different backend names beside them. If both say the same thing, one of the two builds is not what you think it is, and Part 6’s install lesson has the procedure for proving which.

=Expected result

A mixed cluster is the one case where the device list is genuinely interesting rather than bookkeeping. A CUDA device and a Vulkan device in the same list, serving one GGUF file, is the thing this part exists to show.

3. Measure each machine on its own first

You cannot split by speed without knowing each machine’s speed, and you cannot explain a cluster result without a baseline for each participant. Use the Part 6 methodology on a model that fits each machine individually.

RunnableAll tracks

on each machine separately, not through RPC

~/llama.cpp/build/bin/llama-bench \
    -m ~/models/unsloth/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf \
    -p 512 -n 128 -r 3 -ngl 999 -fa on

Record both generation rates. They are the inputs to task 5 and the explanation for everything that follows.

4. Split by memory, which is the default

RunnableAll tracks

on the client: let llama.cpp choose

. ./rpc.env
TENSOR_SPLIT= bash run-split.sh

In another terminal, once it has loaded:

RunnableAll tracks

measure the default split

python3 measure-split.py \
    --iface "$CLUSTER_IFACE" \
    --lab part-19/lab-a-mixed-platform-cluster \
    --tensor-split "default, by free memory" \
    --note "CUDA client, Vulkan host" \
    --labbook labbook.md

Read the client’s load log and write down how many layers each device received. On a mixed cluster this is usually where the surprise is: the machine with the most memory is not necessarily the machine you want holding most of the model.

5. Split by speed, and measure the difference

Feed the planner both figures for each device: its usable memory and the decode rate you measured in task 3.

RunnableAll tracks

two plans from the same cluster

python3 plan-tensor-split.py \
    --weights-gb 18.6 --kv-gb 0.8 --layers 48 \
    --device client-cuda:22:38 \
    --device host-vulkan:90:12 \
    --plan both

The memory plan gives the Vulkan host four fifths of the layers because it has four fifths of the memory. The speed plan gives it about a quarter, because it generates at about a quarter of the rate. Those are very different clusters.

Take the speed plan’s split string, restart the client with it, and measure again.

RunnableAll tracks

on the client: the split you chose

. ./rpc.env
pkill -f llama-server
TENSOR_SPLIT=0.76,0.24 bash run-split.sh

✕Common mistake

Splitting by speed only works while every share still fits. Give a 24 GB card three quarters of a model it cannot hold and the allocation fails, or worse, spills into system memory across PCIe and the host becomes the slowest participant instead of the fastest. plan-tensor-split.py flags any device asked to hold more than it has; heed the flag rather than the intention.

The right split is the fastest one among those that fit, which is usually neither extreme.

Measure again with the same command as task 4, changing only --tensor-split and the note. Two rows, one variable.

6. Add a slow host and watch every token pay for it

Now make the point that a cluster runs at the pace of its participants. Start a second rpc-server on the client itself, restricted to the CPU device and bound to the loopback address, and add it to the split. Keeping it local means the comparison isolates the slow device rather than adding a second network path at the same time.

RunnableAll tracks

on the client: a deliberately slow participant

. ./rpc.env
CLUSTER_ADDR=127.0.0.1 RPC_PORT=50053 RPC_DEVICE=CPU \
    bash start-rpc-server.sh --background

RunnableAll tracks

on the client: include it, with a small share

. ./rpc.env
pkill -f llama-server
RPC_HOSTS="$RPC_HOSTS,127.0.0.1:50053" TENSOR_SPLIT=0.6,0.3,0.1 bash run-split.sh

Ten per cent of the layers on a CPU device is a small share by any measure. Measure the cluster again and compare against task 5.

=Expected result

The damage is out of proportion to the share. A tenth of the layers on a device that is an order of magnitude slower adds roughly a tenth of a very large number to every token, and the run slows far more than a tenth. This is the single most useful intuition in the part: in a layer split the machines take turns, so the total time per token is a sum over participants, and one slow participant is not diluted by the fast ones.

This is also the fault the challenge page hands you, arriving by accident rather than on purpose: a host whose GPU backend failed to load falls back to its CPU device and joins the cluster looking perfectly healthy.

7. Take a host away in the middle of a generation

Failure behaviour is part of a system’s specification, and this one is undocumented, so measure it rather than guess.

Start a long generation, and while it is running, stop one of the RPC servers.

RunnableAll tracks

on the client: ask for a long answer

python3 measure-split.py \
    --iface "$CLUSTER_IFACE" \
    --lab part-19/lab-a-mixed-platform-cluster \
    --n-predict 512 \
    --note "host removed mid-generation" \
    --print-only

RunnableAll tracks

on the second host, while that is running

pkill -f ggml-rpc-server

Record exactly what happened, in this order:

1. What the client printed, verbatim.
2. Whether the request returned partial text, an error, or hung until a timeout.
3. Whether llama-server stayed up and served a second request afterwards, or had to be restarted.
4. Whether restarting the RPC host was enough to recover, or whether the client had to be restarted too.

!Important

There is no recovery mechanism in this backend and the README’s own description of it as a fragile proof of concept covers this. Treat every host in an RPC cluster as a hard dependency of the client: if any of them stops, the run stops. That is a decisive argument against putting an RPC cluster behind anything that people depend on, and it is why Part 23’s operations part treats a single-machine service as the default.

Restart the host and confirm the cluster comes back before moving on.

8. Write the topology down

The artefact of this lab is not a rate. It is a description of a cluster that somebody, possibly you in six months, can reconstruct. Part 18’s record-topology.py already wrote one notebook line per machine describing its interfaces, its maximum transmission unit and its measured links; what this lab adds is which backend each machine contributed and which layers it held, so run that script again on each machine if anything about the network has changed since you did Part 18’s lab.

RunnableAll tracks

probe-rpc-devices.sh

#!/usr/bin/env bash
# Purpose: ask every rpc-server in the cluster what device it is offering, one host at a
#          time, and write the device order the client will use into one short report
# Platform: all (run it on the client; the hosts may be any mixture of tracks)
# Minimum memory: 8 GB on the client; the hosts need only enough for their own share
# Assumes: llama.cpp built with -DGGML_RPC=ON on the client so llama-bench can register RPC
#          devices; one ggml-rpc-server running on every host in RPC_HOSTS; the environment
#          from env-example.txt has been sourced
#
# Usage: . ./rpc.env && bash probe-rpc-devices.sh [report.md]
#
# Environment:
#   RPC_HOSTS   host:port,host:port                      (required)
#   LLAMA_BIN   directory holding llama-bench            (default: found on PATH)
#
# Why one host at a time: --list-devices with every host registered prints one flat list,
# and nothing in that list says which host a device came from. Registering one host per
# run and recording the order is the only way to know what --tensor-split is addressing.

set -euo pipefail

REPORT="${1:-rpc-devices.md}"

die() { echo "probe-rpc-devices: $*" >&2; exit 1; }

if [ -n "${LLAMA_BIN:-}" ]; then
  BENCH="$LLAMA_BIN/llama-bench"
else
  BENCH="$(command -v llama-bench || true)"
fi
[ -n "$BENCH" ] && [ -x "$BENCH" ] || die "llama-bench not found; set LLAMA_BIN"

# --- Part 18 compatibility --------------------------------------------------------------
# Part 18's .env lists every machine in CLUSTER_PEERS, space separated, as names without
# ports. When RPC_HOSTS is not set, build it from those names in the order they appear,
# appending RPC_PORT to each. Set RPC_HOSTS yourself whenever the order matters, when only
# some peers take part, or when a peer listens on a different port: the order is what
# --tensor-split addresses.
RPC_PORT="${RPC_PORT:-50052}"
if [ -z "${RPC_HOSTS:-}" ] && [ -n "${CLUSTER_PEERS:-}" ]; then
  read -r -a PART18_PEERS <<< "$CLUSTER_PEERS"
  for peer in "${PART18_PEERS[@]}"; do
    RPC_HOSTS="${RPC_HOSTS:+$RPC_HOSTS,}${peer}:${RPC_PORT}"
  done
  echo "    RPC_HOSTS built from Part 18's CLUSTER_PEERS: $RPC_HOSTS"
fi

[ -n "${RPC_HOSTS:-}" ] || die "set RPC_HOSTS (or Part 18's CLUSTER_PEERS); see env-example.txt"

{
  echo "# RPC cluster device probe"
  echo
  echo "- client: $(hostname -s), $(uname -s) $(uname -m)"
  echo "- probed: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "- RPC_HOSTS: \`$RPC_HOSTS\`"
  echo
} > "$REPORT"

echo "==> local devices, which always come first in the split order"
{
  echo "## Local devices (position 1 onwards in --tensor-split)"
  echo
  echo '```'
} >> "$REPORT"
"$BENCH" --list-devices 2>&1 | tee -a "$REPORT"
echo '```' >> "$REPORT"
echo >> "$REPORT"

IFS=',' read -r -a HOSTS <<< "$RPC_HOSTS"
POSITION=1
for hp in "${HOSTS[@]}"; do
  host="${hp%%:*}"
  port="${hp##*:}"
  echo "==> $hp"
  {
    echo "## RPC host $POSITION: \`$hp\`"
    echo
  } >> "$REPORT"

  if (exec 3<>"/dev/tcp/$host/$port") 2>/dev/null; then
    echo "    reachable"
    echo "- TCP connect: succeeded" >> "$REPORT"
  else
    echo "    NOT reachable: nothing is listening on $hp" >&2
    echo "- TCP connect: **failed** — no server listening, or the wrong interface" >> "$REPORT"
    echo >> "$REPORT"
    POSITION=$((POSITION + 1))
    continue
  fi

  # Round-trip time on the path the RPC traffic will take. A number in the tens of
  # milliseconds here means Wi-Fi or a router, and this part's challenge page is about that.
  if command -v ping >/dev/null; then
    rtt="$(ping -c 4 "$host" 2>/dev/null | tail -n 1 || true)"
    [ -n "$rtt" ] && echo "- round trip: \`$rtt\`" >> "$REPORT"
  fi

  {
    echo
    echo '```'
  } >> "$REPORT"
  "$BENCH" --rpc "$hp" --list-devices 2>&1 | tee -a "$REPORT"
  {
    echo '```'
    echo
  } >> "$REPORT"
  POSITION=$((POSITION + 1))
done

{
  echo "## Split order"
  echo
  echo "The proportions given to \`--tensor-split\` are read in this order: every local"
  echo "device listed above first, then the RPC hosts in the order they appear in"
  echo "RPC_HOSTS. Copy that order into your notebook beside the split you chose."
} >> "$REPORT"

echo "==> written to $REPORT"

Pending validationThe mixed cluster, four ways — your recording sheet

Split	Layers: client / host B / slow host	Prefill tokens/s	Decode tokens/s	Link bytes per token
by memory (default)	—	—	—	—
by speed	—	—	—	—
by speed, plus a CPU host	—	—	—	—
fastest single machine, no cluster	n/a	—	—	0

your two machines: track, chip, memory and backend for each, the operating system and version of each machine · llama.cpp RPC, layer split the build number from llama-cli --version, identical on both machines · Qwen3-30B-A3B, Q4_K_M · 8,192 tokens of context · the date you ran it

Empty on purpose. The last row cannot be filled in if the model does not fit either machine alone, which is the case on the 16 GB primary path; write 'does not fit' rather than leaving it blank, because that is the answer to why you built the cluster.

Keep device ordering and heterogeneous costs visible

Begin with the working RPC setup from the preceding lab, copying its rpc.env only after reviewing addresses and model paths. On each host, use the local backend build for that platform. Run device discovery again and save the order before supplying an explicit split.

Measure a model that fits on the client alone before adding remote devices. Then add one worker at a time and record its memory contribution and effect on prompt and generation latency. A CPU worker is a separate experiment; adding capacity can introduce a much slower stage. Do not average those results with accelerator-only runs.

For the disconnection test, use an expendable request and stop only the selected lab worker. Record whether the client errors, stalls within its timeout or needs restart; do not assume automatic recovery. Restore the worker and rerun the original short probe. Keep the actual device order, per-device allocation, topology, transport evidence and raw measurements. The completed report should explain which mixture solves a capacity problem, which improves or harms latency and what the operator must do after losing a participant. Heterogeneous discovery alone is not the final acceptance condition.

Validation

You are done when all of the following are true:

• mixed-topology.md shows two different backend names among the devices, one local and one remote, and you have written which machine each belongs to;
• labbook.md has at least three lines with a lab field of part-19/lab-a-mixed-platform-cluster, differing in their tensor_split field;
• each of those lines has non-null predicted_tokens_per_s and non-null rx_bytes;
• you have each machine’s standalone decode rate from task 3 recorded beside them;
• the recording sheet is filled in, including the layer counts read from the load log rather than the proportions you asked for;
• you have written down, in four numbered answers, exactly what the client did when a host disappeared;
• the cluster was restored and served a request afterwards.

Expected outcome

A cluster that mixes backends, and a set of notes that explain its behaviour rather than merely reporting it.

Two backends in one device list. A CUDA device and a Vulkan or Metal device serving one GGUF file. Nothing else in this course does that.

The default split follows memory, and memory is often the wrong axis. On a mixed cluster the machine with the most memory is frequently not the fastest, and the default hands it the most work. Your two measurements say what that costs you.

Splitting by speed helps until it does not fit. Somewhere between the memory split and the speed split is the fastest arrangement that still allocates, and finding it is a couple of measurements rather than a theory.

One slow host taxes every token. The layer split sums the participants’ times, so a small share on a very slow device costs far more than its share suggests.

Losing a host loses the run. No failover, no degradation, no partial result. Write down the exact behaviour you saw, because it is the argument you will need when somebody proposes serving something important from a cluster like this.

Troubleshooting

Both devices report the same backend. One of the two builds is not the backend you intended. Check each host’s server banner and each machine’s llama-bench backend column, as Part 6’s install lesson describes.

The remote host appears as a CPU device when it has a GPU. The RPC server exposes a single CPU device when it finds no accelerators, which means that build has no GPU backend compiled in or its runtime failed to load. Rebuild that host. This is exactly the accidental version of task 6.

The split loads, but the layer counts are the reverse of what you asked for. Device order. Local devices come first, then the RPC hosts in the order they appear in RPC_HOSTS. Re-read mixed-topology.md.

Adding the CPU host makes loading fail rather than slow. The CPU device offers host memory, and asking it for a share that the machine also needs for the client’s own work can exhaust it. Lower its proportion.

The client hangs rather than failing when a host is stopped. Note it: a hang is a different failure from an error, and which one you get is the point of the task. Give it a couple of minutes before interrupting, and record how long you waited.

Two runs of the same split differ noticeably. Something else is using one of the machines, or a laptop is on battery. Repeat both, and if the spread persists, record the spread rather than a single number.

Cleanup

RunnableAll tracks

on every host, when the lab is over

pkill -f ggml-rpc-server

Stop the client’s llama-server too. Keep mixed-topology.md, labbook.md and rpc.env; the challenge page uses the same cluster and the same environment file. The 30B-class model is worth keeping, as Parts 20 and 22 benchmark against it.

What you learned

• Backends mix. One GGUF file, one client, and devices from different vendors and different operating systems in one run.
• The default axis is memory, and you get to choose a different one. Splitting by speed is a one-line change with a measurable effect, bounded by what each device can actually hold.
• A layer split adds up the participants. The cluster’s time per token is a sum, not a maximum, so the slowest machine sets the pace far more than its share of the layers suggests.
• A silent CPU fallback looks like a healthy host. The device list is how you catch it, and it takes one command.
• There is no fault tolerance. A host that stops takes the run with it, and you now know exactly what that looks like on your own cluster.

Record in the notebook: each machine’s track, backend and standalone decode rate; the device order; the three splits with their layer counts and measured rates; the four answers about failure behaviour; and one sentence saying which split you would use if you had to run this cluster tomorrow, and why.

Check your understanding

Question 1. A cluster of one fast 24 GB card and one slower 96 GB machine is left on the default split. What happens?

• The layers are divided evenly, because there are two devices
• The 96 GB machine gets roughly four fifths of the layers, so most of the work lands on the slower device
• The faster device gets more layers, because llama.cpp measures speed at load time
• The model fails to load, because the devices are not matched

Show the answer and why

Answer: The 96 GB machine gets roughly four fifths of the layers, so most of the work lands on the slower device

The default distributes weights and cache in proportion to each device's available memory. Nothing about speed enters into it. On a mixed cluster that regularly puts most of the model on the slowest participant, which is why this lab has you measure the alternative rather than accept the default.

Question 2. You add a CPU-only rpc-server holding a tenth of the layers. Generation slows by much more than a tenth. Why?

• The CPU host saturates the network link
• Because a layer split is sequential: the cluster's time per token is the sum of every participant's time, so a small share on a very slow device adds a large absolute amount
• Because the KV cache has to be copied to the CPU host every token
• Because llama.cpp disables the GPU backends when a CPU device is present

Show the answer and why

Answer: Because a layer split is sequential: the cluster's time per token is the sum of every participant's time, so a small share on a very slow device adds a large absolute amount

The machines take turns. A tenth of the layers on a device an order of magnitude slower contributes roughly the same time as the other nine tenths together. The network is not involved: the boundary payload is a few kilobytes either way.

Question 3. One of your RPC hosts is quietly running its CPU device instead of its GPU. Which check finds it fastest?

• Comparing the cluster's decode rate against last week's
• Listing the devices the client can see, where the host appears as CPU rather than as a CUDA, Vulkan or Metal device
• Checking the model file's checksum
• Reading the interface byte counters

Show the answer and why

Answer: Listing the devices the client can see, where the host appears as CPU rather than as a CUDA, Vulkan or Metal device

The RPC server exposes a single CPU device when it finds no accelerators, so the device list names the fault directly. A rate comparison tells you something is wrong; the device list tells you what.

Question 4. What should you conclude about running a service on an RPC cluster, after task 7?

• It is fine, because llama.cpp retries failed hosts automatically
• Every host is a hard dependency: there is no failover, so the availability of the service is the availability of all the machines multiplied together
• It is fine as long as the hosts are on the same switch
• A second client can take over when a host fails

Show the answer and why

Answer: Every host is a hard dependency: there is no failover, so the availability of the service is the availability of all the machines multiplied together

The backend has no recovery mechanism and its README describes it as a fragile proof of concept. Adding machines to a cluster like this lowers the availability of the whole rather than raising it, which is the argument for keeping anything people depend on on one machine, as Part 23 does.

Check my answersClear