High-availability mode for the App Gateway Connector

• Do say: “HA implementation is ready for staging validation.”
• Do not say: “enterprise-ready HA release is complete” until the release gates pass.
• The desktop wrapper is not an HA runtime. Use the CLI connector as a managed server process.

• Use two instances for the normal production baseline.
• Use three instances when you run several BusinessProxy API replicas or need more maintenance margin.
• Put instances on different hosts or nodes if the requirement includes host failure tolerance.

• Existing streams can finish during drain; new streams go to another usable instance.
• Token rotation closes old authenticated tunnels across API replicas.
• Relay fallback remains a safety path, but steady-state fallback rate must stay below 1% before an enterprise release claim.

• The workspace has an App Gateway connector record and a current runtime token.
• Each connector host can reach BusinessProxy API over outbound HTTPS and can reach every configured internal upstream.
• Secrets are stored in the platform secret store, not in screenshots, tickets or shell history.
• Time synchronization is enabled on connector hosts so heartbeat and evidence windows are reliable.

BUSINESSPROXY_API_URL=https://api.business-proxy.com
CONNECTOR_WORKSPACE_ID=<workspace-id>
CONNECTOR_ID=<connector-id>
CONNECTOR_TOKEN=<runtime-token>
CONNECTOR_INSTANCE_ID=<unique-instance-id>
CONNECTOR_TUNNEL_CONNECTIONS=2
CONNECTOR_DRAIN_ON_EXIT=true
CONNECTOR_DRAIN_TIMEOUT=30s

• Do not reuse the same CONNECTOR_INSTANCE_ID for two live processes.
• Start with CONNECTOR_TUNNEL_CONNECTIONS=2 unless support gives a different value.
• Keep CONNECTOR_DRAIN_ON_EXIT=true so normal restarts stop accepting new work before exit.

instances * CONNECTOR_TUNNEL_CONNECTIONS >= coverage_factor * API_REPLICA_COUNT

Example:
3 instances * 2 tunnels >= 2 * 3 API replicas

set -a
. /etc/businessproxy/connector/common.env
. /etc/businessproxy/connector/instance-a.env
set +a

businessproxy-connector doctor
businessproxy-connector run

• Repeat the same command on the second host or process with instance-b.env.
• In the dashboard, wait until both instances are online and no stale or draining warning is shown.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: businessproxy-connector
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: connector
          image: businessproxy/connector:<version>
          envFrom:
            - secretRef:
                name: businessproxy-connector-runtime
          env:
            - name: CONNECTOR_INSTANCE_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: CONNECTOR_DRAIN_ON_EXIT
              value: "true"

• Add egress rules that allow HTTPS to BusinessProxy API and internal access to the private app upstreams.
• Do not expose a Kubernetes Service for inbound traffic to the connector.

[Unit]
Description=BusinessProxy Connector %i
After=network-online.target
Wants=network-online.target

[Service]
EnvironmentFile=/etc/businessproxy/connector/common.env
EnvironmentFile=/etc/businessproxy/connector/%i.env
ExecStart=/usr/local/bin/businessproxy-connector run
Restart=always
RestartSec=5s
TimeoutStopSec=75s
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

# Start two instances:
systemctl enable --now businessproxy-connector@instance-a
systemctl enable --now businessproxy-connector@instance-b

services:
  connector-a:
    image: businessproxy/connector:<version>
    restart: unless-stopped
    env_file:
      - ./common.env
    environment:
      CONNECTOR_INSTANCE_ID: connector-a
      CONNECTOR_DRAIN_ON_EXIT: "true"
  connector-b:
    image: businessproxy/connector:<version>
    restart: unless-stopped
    env_file:
      - ./common.env
    environment:
      CONNECTOR_INSTANCE_ID: connector-b
      CONNECTOR_DRAIN_ON_EXIT: "true"

• Do not stop all instances at once unless the private app outage is intentional.
• During a normal rolling restart, new requests should recover within a few seconds.
• If fallback rate rises and does not return below 1%, stop the rollout and inspect tunnel coverage.

• Do not rotate the token in the middle of initial scale-out unless the token is suspected to be exposed.
• Keep a rollback artifact for the previous connector version, but never roll back to an ad-hoc local binary.

businessproxy-connector doctor

# Expected dashboard state:
# - connector online
# - 2 or 3 online instances
# - no stale or draining instance after maintenance
# - no API replica coverage warning

npm run connector:ha-staging-readiness:check -- \
  --profile staging \
  --env-file /secure/env/businessproxy/connector-ha-staging.env \
  --evidence-dir /secure/evidence/businessproxy/connector-ha-staging/readiness

npm run connector:ha-staging-evidence:run -- \
  --profile staging \
  --env-file /secure/env/businessproxy/connector-ha-staging.env \
  --evidence-dir /secure/evidence/businessproxy/connector-ha-staging

npm run connector:ha-long-window-evidence:run -- \
  --profile staging \
  --env-file /secure/env/businessproxy/connector-ha-long-window.env

• Evidence files must not include tokens, private IPs, internal URLs, request bodies or raw metric output.
• Keep G3-G9 release gates open until the real staging evidence passes.

• Only one instance is online: check CONNECTOR_INSTANCE_ID, service restart policy, outbound HTTPS and the runtime token.
• An instance stays stale: check host time synchronization, process logs and network path to BusinessProxy API.
• An instance stays draining after maintenance: make sure the old process exited and the supervisor started the new process with the same intended instance ID.
• Fallback is high: check API replica coverage, tunnel count, connector saturation and reconnect storms.
• Only one internal app fails: run diagnostics from every connector host to that app upstream before changing HA settings.

• If the internal app itself is down, the connector cannot make it healthy.
• If all connector instances run on one failed host, host-level availability is lost.
• If the customer network blocks outbound HTTPS to BusinessProxy from every host, the connector cannot establish tunnels.