Skip to main content
A server tunnel is a machine your organization runs next to its internal servers, such as a self-hosted GitLab or an internal Grafana. MCP Manager reaches those servers through it. The machine dials out to MCP Manager, so you open nothing inbound and expose nothing to the internet. Every user still signs in to the internal server with their own identity, so the server’s own audit log names the person behind each call. You connect a machine by running one command. You never run it again: the tunnel updates itself with signed releases and no downtime. See Automatic updates with no downtime. Ready to set one up? Connect an internal server through a tunnel walks the whole path. This page covers how a server tunnel works and what it means for security.
Server tunnels need the Manage tunnels capability, and they must be enabled for your workspace. Without both you don’t see the Tunnels section, and you can’t mark a server as Reached through a tunnel. Ask a workspace administrator to grant it. See Who can manage server tunnels and the capabilities reference.

What a server tunnel is for

A server tunnel solves one problem: an MCP server that lives on your internal network, used by many people, each of whom must sign in as themselves. A self-hosted GitLab is the typical case. MCP Manager’s gateway runs in the cloud and can’t reach it, and you don’t want to expose it to the internet. A server reached through a tunnel is a remote server with one extra setting: Reached through a tunnel, and the tunnel to use. It keeps everything a remote server has. You assign it to gateways, choose per-user or shared identity, and its calls run through your gateway rules and land in your logs. One tunnel can carry several servers. GitLab and Grafana on the same network are two servers bound to the same tunnel, and you assign each to gateways on its own.

How a server tunnel connects

A server tunnel connects outbound only. The machine dials out to MCP Manager over TCP 443 and joins a private, encrypted network built on WireGuard that MCP Manager coordinates. The gateway sends each request to the machine over that network. The machine then opens its own HTTPS connection to the internal server and sends the response back the same way. The gateway sends an internal server’s traffic only through its tunnel. When the tunnel can’t take a request, the request fails with a clear error. MCP Manager never falls back to calling the server’s URL directly. On the machine, one container does all of this. Inside it, a launcher that never changes runs the worker that carries your traffic, and swaps in a new worker when a new release arrives. Each person’s sign-in page opens in their own browser, which must reach the internal server, for example over your office network or VPN. See What runs inside the tunnel container.

What the tunnel machine needs

The tunnel machine is a host your organization runs, close to the servers it reaches. It needs:
  • Docker Engine on a Linux machine, running as root (the standard install). MCP Manager publishes releases for linux/amd64 and linux/arm64. Rootless Docker and Podman aren’t supported yet.
  • One server tunnel per Docker host. A server tunnel’s container is always named mcpm-server-tunnel, so Docker won’t start a second one on the same host. For failover, connect a second host.
  • To stay on. A tunnel with no machine online carries no traffic. The container runs with --restart unless-stopped, so it comes back on its own after a reboot once Docker starts.
  • A route to each internal server it will reach, over your own network.
  • Outbound TCP 443 to the hosts below. Allowlist them by hostname, not IP address, and exempt them from TLS inspection.
The coordination and relay servers are the same ones a workstation uses. If a corporate proxy blocks them, the symptoms and fixes in Network requirements apply here too.

Private certificates and outbound proxies

An internal server with your own certificate authority. The tunnel checks the HTTPS certificate of every destination except the machine itself (localhost and host.docker.internal). If your internal server’s certificate is signed by your organization’s own CA, give the tunnel that CA as PEM text, not a file path. Add this to the command, before the image name:
terminal
Without it, requests to that server fail with Upstream Certificate Rejected, and the message says what to add. HTTPS certificates covers each kind of certificate failure, and applies here too. A machine behind an outbound proxy. Add the standard HTTPS_PROXY (or ALL_PROXY) and NO_PROXY variables to the command the same way. The tunnel uses them to check in, join the private network and download releases. Docker pulls the image itself, so set Docker’s own proxy settings for that. Docker fixes these values when it creates the container. To add one to a machine that’s already running, run a new command on it, as in Limit destinations on the machine itself.

The one-time command that connects a machine

Someone with Manage tunnels selects Connect a machine on the tunnel, which makes a command, and runs it on the machine. MCP Manager fills in every value. It looks like this:
terminal
It’s one line. The origins value sits in double quotes, which bash, PowerShell and cmd all pass through as they are, so it pastes the same way into each. The container starts in the background and keeps running.

Every privilege the tunnel gets is in the command

You can read everything the tunnel is allowed to do in that one line. Docker fixes these settings when it creates the container, so nothing MCP Manager ships later can widen them. What the command leaves out matters as much. There’s no Docker socket, so the tunnel can’t start, stop or inspect any other container on the host. There’s no host folder and no privileged mode, and the container shares the host’s network only when you choose that for a Linux machine that runs Docker directly.

The token, the machine key and when you need a new command

The enrollment token in the command has three limits:
  • It works once. A second machine needs its own command.
  • It expires 24 hours after it’s generated.
  • MCP Manager shows it once and stores only a hash of it.
If MCP Manager refuses a command when the machine runs it, see The install command is refused. The first time the container starts, the machine makes its own key and keeps it in the volume. The private key never leaves the container, and only the launcher can read it. From then on, every request the machine sends to MCP Manager is signed with that key. Each new machine raises an alert, A machine joined a server tunnel, naming who issued the command. Run the command once per machine. A machine needs a new command only when it loses its container or volume, or to reach an allowed destination its command didn’t include. A machine that lost its volume (after docker volume rm or docker system prune --volumes) can no longer prove who it is, so terminate it in the Tunnels section and connect the host again with a new command. MCP Manager can replace every key it uses, including its root keys, without you running anything. See Signed releases. The first install trusts the MCP Manager web app. The command, including the image digest, comes from MCP Manager over TLS. The signature checks protect every update after the install, not the install itself. When MCP Manager has no release published for server tunnels, it can’t issue a command. It creates no token and says that no release is available yet.

More than one machine for failover

You can connect several machines to one tunnel. Each machine’s worker checks in with MCP Manager regularly, and a machine that has stopped checking in gets no traffic. MCP Manager sends each request to an online machine. When it can’t reach that machine before any part of the request was sent, it tries the next one. A request that fails after it was sent is reported and never retried, so a tool call never runs twice. Connect each machine on its own Docker host.

What the tunnel machine can see

The tunnel machine sees every credential sent through it. It’s the HTTPS client to your internal server, so each user’s token passes through it in plain text on the way to that server. Run a server tunnel only on infrastructure your organization controls, next to the servers it reaches.
Treat the tunnel machine like the servers behind it. Anyone who controls the machine can read the credentials of every user who calls a server through it. The tunnel container can’t control Docker or read your host’s files, but whoever controls the host can read the container. Use a host you’d trust with those servers’ credentials, and don’t share it with workloads you trust less.
This is the trade a server tunnel makes on purpose. A workstation is safe to run on a laptop because the only credentials it sees are its owner’s. A server tunnel carries everyone’s, so it belongs on your organization’s infrastructure, never on a personal machine.

What runs inside the tunnel container

One container runs on the tunnel machine, and three kinds of program run inside it.
  • The launcher is the container’s first process. It never changes after you run the command, and it has no network access. It checks every release before anything in it runs, starts each worker as its own user, and keeps the machine key. When a worker needs to send a request to MCP Manager, the launcher signs it, so a worker never holds the key.
  • Workers carry your traffic. A worker joins the private network, checks in with MCP Manager, sends requests on to your internal servers, and downloads the next release. Up to three run at once: the one serving traffic, a new one being checked, and an old one finishing its requests.
  • The built-in fetcher ships inside the image. It downloads the current release the first time the machine starts, and again if no stored release will run. It does nothing else.

What a worker can’t do

Workers are the part that updates, so they get the least power:
  • Read the machine key or the launcher’s records. Only the launcher can.
  • Change the code it runs. Each release is unpacked read-only after the launcher has checked it.
  • See or signal another worker. Each worker runs as its own user, with its own folder.
  • Gain privileges. A worker holds no Linux capability, and no-new-privileges stops it gaining any.
  • Take more than its share. Each worker has its own process limit, so one bad worker can’t starve the one serving your traffic.

What a server tunnel doesn’t promise

The list of places your requests may go is enforced by the worker’s code. Docker limits the kind of network the container gets, not the hosts it can reach. So if someone stole MCP Manager’s release key and took over MCP Manager’s servers, they could ship a worker that ignores the list and reaches whatever the container’s network reaches. That’s why two root keys can replace a stolen release key, so machines stop accepting it, and why the independent check can retire a worker that misbehaves. If you have strict requirements, also limit the machine’s outbound traffic at your firewall, so the network itself only reaches the servers you meant.

Automatic updates with no downtime

You run the command once. After that, the tunnel updates itself. You never run the command again to keep a machine current or secure.

What MCP Manager staff control and what your organization controls

MCP Manager staff manage updates on a staff-only page, and each change records who made it and why. Org members with Manage tunnels see each machine’s version and update status in the Tunnels section, read-only. While staff pause updates, existing machines stay on the version they run. A new machine still starts with the current release.

Signed releases: what the launcher checks before anything runs

A release is a signed package of worker code, published to Docker Hub. Every release is signed by a release key. The release key is in turn approved by MCP Manager’s root keys. There are three root keys, and any two must agree. The launcher has the root keys’ public halves built in, and it checks the whole chain on the machine before anything in a release runs. The launcher refuses a release when:
  • the signatures don’t check out all the way up to the root keys;
  • it’s older than a release the machine has already run (each release has a counter that only goes up);
  • it’s signed with an older key than the newest one the machine has accepted;
  • the download doesn’t match the fingerprint the release lists for it.
MCP Manager’s gateway, the part that talks to your machines, holds no signing key. It can tell a machine that a release exists, but it can’t make one. Every key can be replaced without you. To replace the release key, two root keys approve a new one, and machines stop accepting the old one. To replace a root key, two current root keys sign the new set, and each machine checks that the same way before it trusts it. You never run a new command for either. Updates only move forward. MCP Manager never rolls a machine back to an older release. To undo a bad release, staff publish a newer one.

How an update runs

A new release has to prove it works before the old one stops.
  1. The serving worker learns from MCP Manager that a newer release is out. It downloads the release from Docker Hub and hands it to the launcher.
  2. The launcher checks the release and starts it as a new worker beside the old one. The old worker keeps serving.
  3. The new worker joins the private network and checks in. MCP Manager tests it through its gateway, the way real traffic arrives. The new worker must reach every allowed destination with a verified TLS handshake.
  4. When the checks pass, MCP Manager switches traffic in one step. New requests go to the new worker from the next request on.
  5. The old worker stops taking new requests, finishes the ones in flight, and exits. If it doesn’t finish in time, the launcher stops it anyway, so an old worker can never hold its place.
  6. Once the new worker has proven itself, the launcher remembers it as the release to fall back to, and empties the old worker’s folder.
What “no downtime” means. No new request fails because of an update, and requests in flight finish. A very long-lived stream that’s still open when the old worker’s time runs out is closed, and the client has to reconnect. The stream doesn’t resume where it left off.

When an update goes wrong

A failed update never takes down the worker that’s already serving. If the serving worker itself crashes or goes quiet, requests to that machine fail until the launcher starts a replacement, and the tunnel’s other machines keep serving.
  • A new worker never becomes healthy. MCP Manager doesn’t switch traffic to it, the launcher drops it, and the serving worker carries on.
  • A worker keeps crashing. The launcher restarts the same release a few times. Then it falls back to the last release that proved itself on this machine. If none will run, the built-in fetcher downloads the current release. The launcher never runs a release older than the last one that proved itself.
  • A worker stops responding. Each worker has to signal the launcher regularly. A worker that goes quiet is stopped and replaced, even if its process is still running.

How MCP Manager paces updates

  • One at a time. MCP Manager updates one machine at a time across all tunnels, unless staff raise that. It releases to a first group of organizations, then to everyone. A tunnel can wait a while for its turn when many machines are queued. The Tunnels section then shows Waiting its turn. That’s normal and needs nothing from you.
  • Installs and replacements never wait. A machine’s first worker, and a replacement for a worker that stopped, aren’t updates. They don’t wait their turn, and a pause doesn’t hold them back.

The independent check: replacing a worker that stops updating

Normally the serving worker is how a machine hears about new releases. If a worker ever stopped listening, for example a worker built with a stolen release key, it could keep a machine on old code. So the launcher also runs its own check, on its own schedule, with the built-in fetcher. No release can change the fetcher. It reads a minimum release that two root keys have signed. A release key alone can’t sign it. When a worker is below that minimum, the launcher:
  1. stops signing requests for it straight away;
  2. gives it a short time to finish what it’s doing, then stops it;
  3. starts the current release beside it, so traffic moves over without a gap.
This is the safety net. Normal updates stay the fast way to replace a release.

What the Tunnels section shows about updates

Each machine in the Tunnels section shows the version of each worker and one update status: The tunnel’s page shows the target release even while staff have paused updates. You can’t start, pause or change an update from it. See When a machine doesn’t come online or doesn’t update for what to do about each state.

Limits that stop runaway software

A bug in a release can’t take over the tunnel machine, because several limits stack up. Each one still works if the ones above it fail.
  1. The kernel. The --pids-limit and --memory values in the command cap the whole container. Nothing inside can go past them.
  2. The launcher. At most three workers run at once, each with its own process limit. A worker that’s stopped is removed along with every process it started. The launcher also spaces out worker starts, and that spacing survives a restart.
  3. The memory watchdog. When the container nears its memory limit, the launcher stops the newest worker first, so the worker serving your traffic is the last to go.
  4. The worker. Each worker runs one tunnel for its whole life, so it never stacks up copies of itself.
  5. MCP Manager. The gateway caps how many workers a machine may have joined at once, and paces how fast machines join.

Allowed destinations

Allowed destinations is each tunnel’s list of the origins it may send requests to. An origin is a scheme, host and port, such as https://gitlab.example.internal. Enter only that: the editor refuses a path, a query or credentials. A default port (443 for HTTPS, 80 for HTTP) counts the same as no port. Every request through the tunnel must go to an origin on the list. That includes each step of an OAuth sign-in: discovery, client registration, the token exchange, token refresh and user info. You list the first destinations when you create the tunnel; at least one is required. To change the list afterwards, someone with Manage tunnels opens the tunnel and selects Edit under Allowed destinations. The list opens in the Connect a machine window, and saving it makes a new command that carries the new list. Adding a server through the tunnel can also add the server’s origin to the list. A tunnel allows at most 100 origins. Joined with +, the origins can be at most 32,768 bytes in all. That joined value is the one the command carries (see below). The editor shows how many of the 100 origins you’ve used, and it won’t save a list over either limit. Three rules apply:
  • Only listed origins. MCP Manager compares the canonical origin, so two spellings of one host can’t slip past the list.
  • HTTPS unless you list http://. A plain-HTTP origin works only when you add it with http:// explicitly.
  • The list only narrows. An OAuth server’s metadata can never add an origin to it. OAuth endpoints must also pass the same rule a workstation uses: they must be on the authorization server’s issuer origin, an origin its metadata names, or the MCP server’s origin. See Signing in with OAuth.
When a destination isn’t allowed, the request fails with MCPM_TUNNEL_DESTINATION_NOT_ALLOWED (HTTP 403). The message names the URL and the tunnel. MCP Manager sent nothing to that URL. To fix it, add the URL’s origin to the tunnel’s allowed destinations. If the server signs in with OAuth on another host, such as a separate identity provider, add that origin too. Each machine gets the new list at its next check-in, but it reaches the new origin only after you run a new command on it, because its command caps its destinations. See Limit destinations on the machine itself. The same error is returned when the server redirects to another origin. A server tunnel follows a redirect only within the same origin, so a request’s credentials never reach a different server, and adding the origin doesn’t change that. Point the MCP server at the address it redirects to instead. The message names both addresses. Keep the list to origins you use. During every update, the new worker must reach each allowed destination before it takes over. An origin the machine can’t reach makes the update fail with the reason, and the machine stays on its current version. That release isn’t offered to the machine again until the tunnel’s allowed destinations change or a newer release comes out.

Limit destinations on the machine itself

The command also caps destinations on the machine, so the limit holds even if someone changes the list in MCP Manager. That’s the MCPM_TUNNEL_ALLOWED_ORIGINS value in the command: a list of origins joined by +, such as https://gitlab.example.internal+https://grafana.example.internal. MCP Manager fills it in from the tunnel’s allowed destinations when it builds the command, and you can edit it before you run the command. An origin with characters a shell would misread is left out, and the window says so. If a tunnel’s list is already longer than 32,768 bytes, MCP Manager doesn’t issue a command. A machine installed with it would reach none of the destinations. The message says the list is too long and links to the tunnel’s page. There, select Edit under Allowed destinations, shorten the list and save it, and the window makes the command. The machine sends only to origins that are on both lists. Docker fixes the cap when it creates the container, so MCP Manager can’t widen it later. Keep the two lists in step: an origin on the tunnel’s list but outside the cap can’t be reached from that machine. To give a connected machine an origin you added, run a new command on it. In Connect a machine, copy a new command. On the machine, remove the old container with docker rm -f mcpm-server-tunnel, then run the new command. The machine connects again as a new machine, so terminate the old one in the Machines list. Each machine needs its own command, because a command works once. For the next machine, select re-generate command, next to the expiry time.

Terminating a machine or a tunnel

Someone with Manage tunnels can terminate a single machine or a whole tunnel in the Tunnels section. Both take effect from the next request, and requests in flight through a terminated machine fail.
  • Terminate a machine when it’s retired, lost, or may be compromised. MCP Manager revokes the machine first, so it gets no traffic from the next request on, even if it’s still running. The machine is refused at its next check-in, stops carrying traffic, and goes quiet. Other machines on the tunnel keep serving. Termination is permanent: to use that host again, connect it with a new command.
  • Terminate the tunnel with Revoke all machines, under Manage this tunnel on the tunnel’s page, to cut off a whole network at once. This terminates every machine on it and cancels the commands nobody has used. It doesn’t change whether the tunnel is on: an enabled tunnel stays enabled and takes new machines, so connect a machine with a new command to use it again. If part of the termination fails, the alert says which part, and you select Revoke all machines again. See The machines of a tunnel were only partly terminated.
Turning a tunnel off is a different action, and it terminates nothing. While a tunnel is disabled or archived, MCP Manager sends no request through it, and every server bound to it fails with Tunnel disabled. Its machines stay connected and keep checking in, and the tunnel takes no new machine and no new command. Its machines start no update while it’s off: the Tunnels section shows them as waiting. Turn it back on and requests flow through the same machines again, with nothing to reinstall. Each termination raises an alert that names who did it. To clean up the host afterwards, remove the container and its volume:
terminal
docker volume ls --filter name=mcpm-tunnel lists the volume names.

Who can manage server tunnels

Server tunnels are managed with one capability, Manage tunnels, and they must be enabled for your workspace. The capability can be granted to any role. Someone with Manage roles turns it on for the roles that need it. Manage tunnels doesn’t let its holder read anyone else’s credentials or see more of the logs than their other capabilities allow. A server goes through exactly one tunnel. To switch it, use Move to another tunnel on the server’s page.

Server tunnel or workstation

A server tunnel and a workstation both reach servers MCP Manager can’t reach directly, and both connect outbound only. They differ in who owns the machine and whose sign-in goes through it. Use a server tunnel when many people reach an internal server through one shared machine, each as themselves. Use a workstation for a server that runs on one person’s own machine.

When a request through a server tunnel fails

A request through a server tunnel can fail in four ways. The AI client gets a short error that links to its section on this page. The full message, with the URL and the tunnel, is in the request’s log entry and, except for Tunnel disabled, in an alert. A destination that isn’t on the list is covered under Allowed destinations; the other three are below.

Tunnel unavailable

Tunnel unavailable (MCPM_TUNNEL_UNAVAILABLE, HTTP 503) means the server’s tunnel has no machine that can take the request right now. The message names the tunnel. MCP Manager sent nothing to the internal server, and it never calls the server directly instead. The usual causes:
  • No machine is online. The machine is off, Docker is stopped, or the tunnel container isn’t running.
  • Its worker stopped. A release kept crashing, or the launcher had to stop the worker for memory. The launcher raises A tunnel machine reports a failure. See What each tunnel alert means.
  • The machine can’t reach MCP Manager. A firewall or proxy blocks one of the hosts it needs.
  • Its last machine was terminated.
  • A brief failure inside MCP Manager. The message then says so, and a retry succeeds.
To fix it:
  1. Open the tunnel in the Tunnels section and check whether any machine is online and when each was last seen.
  2. On the machine, check that the tunnel container is running, and read its log. The log holds the launcher’s events and the worker’s own messages:
    terminal
  3. If the log shows it can’t reach a host, have your network team allow it. See What the tunnel machine needs.
  4. If its machines were terminated, connect a machine with a new command.
To keep one machine from being a single point of failure, connect a second one. See More than one machine for failover.

Misconfigured

Misconfigured (MCPM_TUNNEL_MISCONFIGURED, HTTP 500) means a server is marked as Reached through a tunnel but MCP Manager can’t route it through one. Either it isn’t bound to exactly one active tunnel, or it isn’t a standalone remote server. Usually the server was saved without a tunnel. MCP Manager won’t guess where to send the request, and it never falls back to calling the server’s URL directly. To fix it, someone with Manage tunnels opens the server in MCP Servers and chooses its tunnel again. The next request uses the new binding. Only a standalone remote server can be reached through a tunnel; the error’s details name the server’s type when that’s the cause.

Tunnel disabled

Tunnel disabled (MCPM_TUNNEL_DISABLED, HTTP 503) means an admin disabled or archived the server’s tunnel. It’s a choice, not a fault: MCP Manager sent nothing to the network behind the tunnel, and it never calls the server directly instead. The message names the tunnel and links to its page. The tunnel’s machines aren’t terminated, so nothing needs to be reinstalled. To fix it, someone with Manage tunnels turns on the tunnel’s Enabled switch in the Tunnels list. If the tunnel was archived, switch on Show archived and unarchive it first. The next request goes through the same machines.

When a machine does not come online or does not update

These states show in the Tunnels section, on the machine. Each one says what MCP Manager knows, what it means for traffic, and what to do. Machine states are Online, Offline and Terminated. Worker states are Joining, Being checked (update), Serving traffic and Draining (being replaced).

The install command is refused

When MCP Manager refuses the command, the machine never connects. Its log says why, with a link to the page that fixes it:
terminal
To generate a new command, open the tunnel in the Tunnels section and select Connect a machine, or select re-generate command next to the expiry time if the window is still open. Before you run it, remove the refused container with docker rm -f mcpm-server-tunnel, because the new command starts a container with the same name.

A worker stays in Joining

A worker in the Joining state has started but hasn’t registered with MCP Manager yet. The machine carries no traffic through that worker until it registers. The usual causes are a firewall or proxy that blocks headscale.mcpmanager.ai, derp-a.mcpmanager.ai or derp-b.mcpmanager.ai, a release that can’t be downloaded from Docker Hub, and a join key that expired because the machine was asleep. To find the cause, read the container’s log:
terminal
See What the tunnel machine needs for the hosts to allow.

A machine shows as offline

A machine shows as Offline when it has no serving worker that checked in recently. It gets no traffic until one does. To fix it, check that the container exists and is running, and start it if it stopped:
terminal
If the container is gone and its volume is gone too, the machine lost its identity and needs a new command. If only the container is gone, connect the host again with a new command too, because a command works once. Then terminate the old machine in the Machines list.

An update shows Waiting its turn

Waiting its turn means a newer release exists and this machine hasn’t got it yet. The status gives the reason: MCP Manager updates one machine at a time across all tunnels unless staff raise that, staff paused updates, or the tunnel is off. Updates also pause by themselves for everyone when several new releases fail in a short time. The machine keeps running its current version and serves traffic as usual. Unless the tunnel is off, nothing is needed from you. If the status also says the machine hasn’t checked in recently, it can’t start anything until it’s back online. See A machine shows as offline.

An update shows Update failed

Update failed means the update didn’t finish. The status shows the reason and the time. The machine keeps serving on its current version. MCP Manager raised one alert for the tunnel and release, which you find on the Alerts page. The reason says which part failed:
  • The launcher refused the release. It failed one of the signature or version checks, or it has nothing for this machine’s operating system and CPU type. This won’t succeed on retry, so the machine isn’t offered that release again.
  • The download failed. Docker Hub couldn’t be reached, or the download was cut short. MCP Manager tries again later.
  • The new worker failed MCP Manager’s checks. The reason names the check, for example that it can’t reach a specific allowed origin. MCP Manager doesn’t offer that release to the machine again until the tunnel’s allowed destinations change or a newer release comes out, so fixing the network alone doesn’t retry it. Removing an origin the machine can’t reach offers it again. If the tunnel lists more than 100 origins, which only a list saved outside the editor can, the reason says at most 100 can be checked: remove the origins the tunnel no longer needs.
When a release failed on your machine, tell support the tunnel name, the reason and the time shown, so staff can publish a release for the machine to move to.

A machine shows No release available yet

No release available yet means MCP Manager hasn’t published a release yet, so there’s nothing to update to. A machine that has no worker yet gets none until a release is published. Machines that already run a worker are unaffected.

What each tunnel alert means

Tunnel alerts link to this page. Find yours by its title or its code. Each entry says what the alert means, what happens to traffic meanwhile, and what to do. For alerts in general, see Alerts.

Two launchers run as one tunnel machine

warning.tunnel.duplicate_agent. Two launchers check in with the same machine’s key at the same time. That happens when a virtual machine is copied, or when a second container starts with a copy of the machine’s volume. MCP Manager can’t tell their workers apart. Meanwhile: MCP Manager refuses neither copy, and traffic keeps flowing. What to do: remove the copy’s container. It most likely has the same name as the original, which the alert names: mcpm-server-tunnel. If you can’t tell which one is the copy, terminate the machine in the Machines list and connect each host again with its own new command.

A tunnel machine reports a failure

error.tunnel.launcher_failure. The launcher on the machine reports one of these, named in the alert:
  • crash_loop. A new release died several times in a row before it served steadily. The launcher won’t start it again until another release becomes stable or the launcher restarts. Meanwhile, the machine runs the last release that served steadily. Tell support the tunnel name and the alert’s text.
  • memory_kill of the worker that carries traffic. The container neared its memory limit, and the launcher had to stop the serving worker. That machine carries no traffic until the launcher starts a worker again. Requests go to the tunnel’s other machines, and with no other machine online they fail with Tunnel unavailable. If it happens again, tell support.

A tunnel machine takes no updates until it repairs its trust record

error.tunnel.launcher_failure with trust_damaged. The launcher found its record of trusted signing keys or its minimum release damaged. Meanwhile: the machine keeps running the release it has and keeps serving traffic. It refuses every new release, fixes included, until it has repaired the record. What to do: usually nothing. The launcher repairs the record by itself, from statements it reads from the release registry or gets from MCP Manager, with no reinstall. If the machine still reports this after several hours, check that it can reach Docker Hub (see What the tunnel machine needs), and look in its log for trust_damaged and then trust_rebuilt.

A tunnel machine reports a problem

warning.tunnel.launcher_warning. The launcher reports a problem it works around. The machine keeps serving traffic. The alert names the problem:
  • pulse_lost. A worker stopped answering the launcher, so the launcher stopped it.
  • memory_kill. The container neared its memory limit, and the launcher stopped a worker that wasn’t carrying traffic, such as a new release being checked.
  • journal_write_failed or journal_problem. The launcher couldn’t write or read its records in the volume. Check that the host’s disk isn’t full.
  • check_failed. The launcher’s own check of the release registries didn’t finish. It keeps the release it runs and checks again later.
  • no_process_limit, process_ceiling_below_minimum, memory_watchdog_off or swap_allowed. The container wasn’t started with the --pids-limit, --memory and --memory-swap values from the command, so one of the limits that stop runaway software is missing. Connect the host again with a new command, run as MCP Manager gives it, and terminate the old machine.
  • Any other code, such as root_set_refused, helper_failed, layout_problem, kind_mismatch or internal_error. The alert’s text gives the reason. If it repeats, tell support the tunnel name and the alert’s text.

Requests claiming to be a tunnel machine were refused

warning.tunnel.machine_refused. MCP Manager refused several requests in a row that claim to be one machine, because their signature doesn’t match the key it holds for that machine. Meanwhile: anyone who knows a machine’s id can send such requests, and they change nothing. If the machine still checks in, nothing needs doing. If the machine itself sent them, it reaches nothing, and its workers keep retrying. What to do: this happens when a machine was restored from an old backup of its volume, or when a key renewal didn’t finish. Terminate the machine and connect the host again with a new command.

A tunnel machine signed a request with a counter that went backwards

warning.tunnel.machine_refused. A launcher numbers its signed requests in order, and this request carried a number far below the highest one MCP Manager has accepted from the machine. Usually the machine was restored from a backup or snapshot of its volume, or an older copy of it runs on another host. Meanwhile: requests from that older state stay refused. Its check-ins fail, and a worker of it that restarts can’t join the tunnel again. What to do: if a copy runs on another host, remove it first. Then terminate the machine and connect the host again with a new command.

The machines of a tunnel were only partly terminated

warning.tunnel.termination_incomplete. Someone selected Revoke all machines, and only part of it finished. The alert says how many machines were terminated and what failed. Meanwhile: a machine that wasn’t terminated keeps carrying traffic, and a command that wasn’t cancelled can still connect a machine. What to do: open the tunnel and select Revoke all machines again.

A tunnel update failed on a machine

warning.tunnel.update_deferred. When MCP Manager will try the release again, the title is A tunnel update hit a problem on a machine instead. A new release didn’t start on the machine, and the alert says at which step and why. Meanwhile: the current version keeps serving. What to do: see An update shows Update failed.

Alerts that are no longer raised

error.tunnel.no_healthy_worker, warning.tunnel.replacement_incomplete and warning.tunnel.trust_state_unsaved came from the earlier tunnel agent, which is gone. MCP Manager no longer raises them, but older alerts can still show them. Nothing needs doing about an old one.

Further reading

Connect an internal server through a tunnel

The hands-on tutorial: create a tunnel, connect a machine, and let each user sign in as themselves.

Workstation MCP Servers

Servers on one person’s own machine, reached through an encrypted tunnel.

Remote MCP Servers

Servers at a URL, and the authentication methods they use.

Architecture & Trust

How the gateway path is secured, and where a server tunnel fits.

External sources

WireGuard

The encrypted-tunnel protocol that server tunnels are built on.

Docker run reference: runtime privilege and Linux capabilities

What --cap-drop, --cap-add and --security-opt do.