> ## Documentation Index
> Fetch the complete documentation index at: https://otoyinc.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Common issues and solutions for both Node Operators and Service Consumers.

<Tabs>
  <Tab title="Node Operators">
    ## Node Operator Errors

    ### Can't connect to Docker API

    **Error:** `Can't connect to Docker API, please ensure Docker is running and you have sufficient permissions`

    **Causes:**

    * Dispersed lacks Docker permissions
    * User not in `docker` group
    * Dispersed not started with `sudo`
    * Docker not running or installed incorrectly

    **Solutions:**

    1. Run Dispersedt with `sudo`
    2. Add your user to the docker group: `sudo usermod -aG docker $USER`
    3. Verify Docker is running: `sudo systemctl status docker`
    4. Reinstall Docker following the [registration guide](/node-operators/node-operators-register-node)

    ***

    ### Can't start NVIDIA container (nvidia-ctk not installed)

    **Error:** `Can't start NVIDIA container, please ensure nvidia-ctk is installed`

    **Cause:** NVIDIA Container Toolkit not installed or configured.

    **Solution:**

    Verify the toolkit works:

    ```bash theme={null}
    sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
    ```

    If this fails, reinstall the NVIDIA Container Toolkit:

    ```bash theme={null}
    sudo apt-get install -y nvidia-container-toolkit
    sudo nvidia-ctk runtime configure --runtime=docker
    sudo systemctl restart docker
    ```

    ***

    ### Driver/library version mismatch

    **Error:** `Can't start NVIDIA container, driver/library version mismatch`

    **Cause:** NVIDIA packages upgraded but old drivers still loaded.

    **Solutions:**

    1. Reboot your system to load new drivers
    2. Pin NVIDIA packages to prevent unattended upgrades:
       ```bash theme={null}
       sudo apt-mark hold nvidia-*
       ```

    ***

    ### Unknown error running NVIDIA container

    **Error:** `Unknown error running NVIDIA container, please check your configuration`

    **Solution:** Check the detailed error:

    ```bash theme={null}
    sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
    ```

    Common issues:

    * Missing `libnvidia-gl` package
    * Incorrect CUDA version
    * GPU not detected

    ***

    ### Node not registered

    **Error:** `Node not registered, please register it`

    **Cause:** First-time setup or registration data lost.

    **Solution:** Register with the `--register` flag:

    ```bash theme={null}
    ./dispersed --register --pubkey pk_xxx --secretkey sk_xxx
    ```

    ***

    ### Lockfile permission error

    **Error:**

    ```text theme={null}
    error: unable to open lock file: Permission denied
    terminate called after throwing an instance of 'std::runtime_error'
    what(): unable to open lock file.
    ```

    **Cause:** Dispersed previously ran with elevated permissions, now running without.

    **Solution:** Run with `sudo` or fix permissions:

    ```bash theme={null}
    sudo ./dispersed --pubkey pk_xxx --secretkey sk_xxx
    ```

    ***

    ### GPU not detected

    **Symptom:** Node appears online but receives no jobs.

    **Solutions:**

    1. Verify GPU is visible:
       ```bash theme={null}
       nvidia-smi
       ```
    2. Check CUDA installation:
       ```bash theme={null}
       nvcc --version
       ```
    3. Verify Docker GPU access:
       ```bash theme={null}
       sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
       ```
  </Tab>

  <Tab title="Service Consumers">
    ## Service Consumer Issues

    ### Job stuck in PENDING

    **Symptom:** Job created but never assigned.

    **Causes:**

    * No available Nodes matching hardware requirements
    * Requested specs too high
    * Network congestion

    **Solutions:**

    1. Lower hardware requirements (less GPU, RAM, VRAM)
    2. Wait for Node availability
    3. Check Console for network status

    ***

    ### SSH connection refused

    **Symptom:** `Connection refused` when connecting to Job.

    **Causes:**

    * Job not yet in RUNNING state
    * Firewall blocking connection
    * Wrong IP/port

    **Solutions:**

    1. Verify Job status is RUNNING (not just ASSIGNED)
    2. Check `node_urls` in Job Run response for correct hostname/port
    3. Verify your IP is in `allowed_ips` if specified

    ***

    ### SSH permission denied (publickey)

    **Symptom:** `Permission denied (publickey)` when connecting.

    **Cause:** SSH key mismatch.

    **Solutions:**

    1. Verify the public key you submitted matches your private key
    2. Check SSH key format (should be full public key including type and comment)
    3. Ensure private key has correct permissions: `chmod 600 ~/.ssh/id_ed25519`

    ***

    ### Job failed unexpectedly

    **Symptom:** Job status shows FAILED.

    **Causes:**

    * Container crashed
    * Out of memory
    * Docker image issues

    **Solutions:**

    1. Check Job logs in Console
    2. Request more RAM/VRAM
    3. Verify Docker image exists and is accessible
  </Tab>
</Tabs>

***

## API Errors

### 401 Unauthorized

**Causes:**

* Missing or invalid API key
* Expired API key
* Missing required headers

**Solutions:**

1. Verify `X-API-Key` header contains correct public key
2. Check API key hasn't expired in [Console](https://console.dispersed.com/keys)
3. Ensure all required headers present: `X-API-Key`, `X-Time`, `X-Nonce`, `X-Signature`

***

### 400 Bad Request (timestamp)

**Error:** `Timestamp out of acceptable range`

**Cause:** `X-Time` header not within 5 minutes of server time.

**Solutions:**

1. Sync your system clock: `sudo ntpdate pool.ntp.org`
2. Verify timestamp is in milliseconds (not seconds)

***

### 403 Forbidden (signature mismatch)

**Error:** `Invalid signature`

**Causes:**

* Incorrect canonical string format
* Wrong secret key
* Body not properly canonicalized

**Solutions:**

1. Verify canonical string order:
   ```text theme={null}
   {public_key}|{timestamp}|{nonce}|{method}|{path}|{query}|{body_sha256}
   ```
2. For JSON bodies, canonicalize with sorted keys and no whitespace:
   ```javascript theme={null}
   JSON.stringify(body, Object.keys(body).sort());
   ```
3. Verify you're using the correct `secret_key`

***

### 403 Forbidden (insufficient permissions)

**Error:** `Insufficient permissions`

**Cause:** API key doesn't have required permissions for the operation.

**Solutions:**

1. Check API key permissions in [Console](https://console.dispersed.com/keys)
2. Create new key with appropriate preset

***

### 404 Not Found

**Causes:**

* Resource doesn't exist
* Wrong UUID
* Resource belongs to different account

**Solutions:**

1. Verify the UUID is correct
2. Check you're querying the right endpoint

***

### 429 Too Many Requests

**Cause:** Rate limit exceeded.

**Solution:** Wait and retry with exponential backoff.

***

## Still Having Issues?

If you can't resolve your issue:

1. Check the [Console](https://console.dispersed.com) for detailed status and logs
2. Review the [API Reference](/api-reference) for correct request formats
3. Contact support through the Console help section