Troubleshooting

Common issues and solutions for both Node Operators and Service Consumers.

Node Operators

Node Operator Errors

Can’t connect to Docker API

Error: Can't connect to Docker API, please ensure Docker is running and you have sufficient permissions

Causes:

• Dispersed lacks Docker permissions
• User not in docker group
• Dispersed not started with sudo
• Docker not running or installed incorrectly

Solutions:

1. Run Dispersedt with sudo
2. Add your user to the docker group: sudo usermod -aG docker $USER
3. Verify Docker is running: sudo systemctl status docker
4. Reinstall Docker following the registration guide

---

Can’t start NVIDIA container (nvidia-ctk not installed)

Error: Can't start NVIDIA container, please ensure nvidia-ctk is installed

Cause: NVIDIA Container Toolkit not installed or configured.

Solution:

Verify the toolkit works:

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

If this fails, reinstall the NVIDIA Container Toolkit:

sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

---

Driver/library version mismatch

Error: Can't start NVIDIA container, driver/library version mismatch

Cause: NVIDIA packages upgraded but old drivers still loaded.

Solutions:

1. Reboot your system to load new drivers
2. Pin NVIDIA packages to prevent unattended upgrades:
   sudo apt-mark hold nvidia-*
   

---

Unknown error running NVIDIA container

Error: Unknown error running NVIDIA container, please check your configuration

Solution: Check the detailed error:

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Common issues:

• Missing libnvidia-gl package
• Incorrect CUDA version
• GPU not detected

---

Node not registered

Error: Node not registered, please register it

Cause: First-time setup or registration data lost.

Solution: Register with the --register flag:

./dispersed --register --pubkey pk_xxx --secretkey sk_xxx

---

Lockfile permission error

Error:

error: unable to open lock file: Permission denied
terminate called after throwing an instance of 'std::runtime_error'
what(): unable to open lock file.

Cause: Dispersed previously ran with elevated permissions, now running without.

Solution: Run with sudo or fix permissions:

sudo ./dispersed --pubkey pk_xxx --secretkey sk_xxx

---

GPU not detected

Symptom: Node appears online but receives no jobs.

Solutions:

1. Verify GPU is visible:

   nvidia-smi
   

2. Check CUDA installation:

   nvcc --version
   

3. Verify Docker GPU access:

   sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
   

Service Consumers

Service Consumer Issues

Job stuck in PENDING

Symptom: Job created but never assigned.

Causes:

• No available Nodes matching hardware requirements
• Requested specs too high
• Network congestion

Solutions:

1. Lower hardware requirements (less GPU, RAM, VRAM)
2. Wait for Node availability
3. Check Console for network status

---

SSH connection refused

Symptom: Connection refused when connecting to Job.

Causes:

• Job not yet in RUNNING state
• Firewall blocking connection
• Wrong IP/port

Solutions:

1. Verify Job status is RUNNING (not just ASSIGNED)
2. Check node_urls in Job Run response for correct hostname/port
3. Verify your IP is in allowed_ips if specified

---

SSH permission denied (publickey)

Symptom: Permission denied (publickey) when connecting.

Cause: SSH key mismatch.

Solutions:

1. Verify the public key you submitted matches your private key
2. Check SSH key format (should be full public key including type and comment)
3. Ensure private key has correct permissions: chmod 600 ~/.ssh/id_ed25519

---

Job failed unexpectedly

Symptom: Job status shows FAILED.

Causes:

• Container crashed
• Out of memory
• Docker image issues

Solutions:

1. Check Job logs in Console
2. Request more RAM/VRAM
3. Verify Docker image exists and is accessible

---

API Errors

401 Unauthorized

Causes:

• Missing or invalid API key
• Expired API key
• Missing required headers

Solutions:

1. Verify X-API-Key header contains correct public key
2. Check API key hasn’t expired in Console
3. Ensure all required headers present: X-API-Key, X-Time, X-Nonce, X-Signature

---

400 Bad Request (timestamp)

Error: Timestamp out of acceptable range

Cause: X-Time header not within 5 minutes of server time.

Solutions:

1. Sync your system clock: sudo ntpdate pool.ntp.org
2. Verify timestamp is in milliseconds (not seconds)

---

403 Forbidden (signature mismatch)

Error: Invalid signature

Causes:

• Incorrect canonical string format
• Wrong secret key
• Body not properly canonicalized

Solutions:

1. Verify canonical string order:

   {public_key}|{timestamp}|{nonce}|{method}|{path}|{query}|{body_sha256}
   

2. For JSON bodies, canonicalize with sorted keys and no whitespace:

   JSON.stringify(body, Object.keys(body).sort());
   

3. Verify you’re using the correct secret_key

---

403 Forbidden (insufficient permissions)

Error: Insufficient permissions

Cause: API key doesn’t have required permissions for the operation.

Solutions:

1. Check API key permissions in Console
2. Create new key with appropriate preset

---

404 Not Found

Causes:

• Resource doesn’t exist
• Wrong UUID
• Resource belongs to different account

Solutions:

1. Verify the UUID is correct
2. Check you’re querying the right endpoint

---

429 Too Many Requests

Cause: Rate limit exceeded.

Solution: Wait and retry with exponential backoff.

---

Still Having Issues?

If you can’t resolve your issue:

1. Check the Console for detailed status and logs
2. Review the API Reference for correct request formats
3. Contact support through the Console help section