⚠️ Homebrew support is being deprecated. Solo will stop publishing updates to Homebrew after August 31, 2026. Install via npm install -g @hiero-ledger/solo@latest instead. See System Readiness for full install instructions.

Troubleshooting

Solutions to common issues when using Solo, plus guidance on getting help. This document covers installation problems, pod readiness issues, resource constraints, and how to find additional support.
Categories:
Tags:

This guide covers common issues you may encounter when using Solo and how to resolve them.

Quick Navigation

Use this page when something is failing and you need to diagnose or recover quickly.

If you are looking for setup or day-to-day usage guidance rather than failure diagnosis, start with these pages:

Error Code Reference

Every error Solo raises carries a structured code of the form SOLO-XXXX, an ownership classification (User, Infrastructure, or Solo), and a retryable flag. When a command fails with one of these codes, look it up in the Error Codes reference for a plain-language description of why it is thrown, along with its troubleshooting steps.

Codes are grouped by category — Configuration, Deployment, Component, Validation, System, and Internal. Solo’s CLI also prints a documentation link for each error (for example https://solo.hiero.org/docs/errors/SOLO-1001) that opens its page directly.

Anatomy of a Solo error

When a command fails with a coded error, Solo prints the code and message, suggested next steps, and a documentation link:

[SOLO-2002] A deployment named 'my-deployment' already exists. Please select a different name
  → Check existing deployments: solo deployment config list
  → Choose a different name for your deployment

Learn more: https://solo.hiero.org/docs/errors/SOLO-2002
  • [SOLO-NNNN] is the error code; its leading digit identifies the category.
  • → lines are suggested remediation steps, shown when the error provides them.
  • Learn more links to that error’s online reference page.

In the terminal, this appears inside a bordered ERROR box, followed by a tip suggesting solo deployment diagnostics logs or solo deployment diagnostics report to gather more detail.

Note: Add --debug to a command to see the full error cause chain and stack traces instead of the summarized form — useful when filing a bug report. (This flag was previously named --dev. The --dev alias still works but is deprecated and will no longer be supported once Solo 0.82.0 reaches its end of support date.)

Common Issues and Solutions

Troubleshooting Installation and Upgrades

Installation and upgrade failures are common, especially when older installs or previous deployments are still present.

Symptoms

You are likely hitting an installation or upgrade problem if:

  • solo fails to start after changing versions.
  • solo one-shot single deploy fails early with validation or environment errors.
  • Commands report missing dependencies or incompatible versions.
  • A new deployment fails immediately after a previous network was not destroyed.
  • A global install fails with EEXIST: file already exists pointing at .../bin/solo. This happens when Solo is already installed under the other npm package name (@hiero-ledger/solo and @hashgraph/solo are mirrors that share the solo binary). See Resolving an EEXIST package-name conflict.

Quick Checks

  1. Confirm installation method

    If you previously installed Solo via Homebrew and are now migrating to npm, remove the Homebrew install to avoid conflicts:

    brew uninstall hiero-ledger/tools/solo

    Then install via npm:

    npm install -g @hiero-ledger/solo@latest

    If you have Solo installed under both npm package names (@hiero-ledger/solo and @hashgraph/solo), remove both before reinstalling:

    npm uninstall -g @hiero-ledger/solo || true
npm uninstall -g @hashgraph/solo || true

    If a global npm install fails with EEXIST because both package names are present, see Resolving an EEXIST package-name conflict.

  2. Verify system resources

    • Ensure your machine and Docker (or other container runtime) meet the minimum requirements described in
      System readiness.

    • If Docker Desktop or your container runtime is configured below these values, increase the allocations and retry the install or deploy.

  3. Clean up previous deployments

    If an upgrade or redeploy fails, first run a standard destroy:

    solo one-shot single destroy

Pods not reaching Ready state

If pods remain in Pending, Init, ContainerCreating, or CrashLoopBackOff, follow this sequence to identify the blocker.

  1. Check readiness and restarts

    # Show readiness and restart count for each pod
kubectl get pods -n "${SOLO_NAMESPACE}" \
  -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount

  2. Inspect pod events

    # List all pods in your namespace
kubectl get pods -n "${SOLO_NAMESPACE}"

# Describe a specific pod to see events
kubectl describe pod -n "${SOLO_NAMESPACE}" <pod-name>

  3. Map symptoms to likely causes

    SymptomLikely causeNext step
    PendingInsufficient resourcesIncrease Docker memory/CPU allocation, then retry
    PendingStorage issuesCheck disk space, free space if needed, restart Docker
    CrashLoopBackOffContainer failing to startCheck pod logs: kubectl logs -n "${SOLO_NAMESPACE}" <pod-name>
    ImagePullBackOffCan’t pull container imagesCheck internet connectivity and Docker Hub rate limits. If the image is quay.io/minio/minio, see MinIO image pull failures

CrashLoopBackOff causes and remediation

If a pod repeatedly restarts and enters CrashLoopBackOff, inspect current logs, previous logs, and events:

# Current container logs
kubectl logs -n "${SOLO_NAMESPACE}" <pod-name>

# Previous container logs (captures startup failures)
kubectl logs -n "${SOLO_NAMESPACE}" <pod-name> --previous

# Pod events and failure reasons
kubectl describe pod -n "${SOLO_NAMESPACE}" <pod-name>

Common causes include invalid runtime configuration, missing dependencies, and insufficient memory.

  • Recommended remediation sequence:

    1. If events mention OOMKilled or repeated liveness probe failures, increase Docker CPU/RAM and retry.

    2. If the issue started after a failed upgrade or deploy, run the cleanup steps in Old installation artifacts and redeploy.

    3. If only one node is affected, refresh or restart it:

      solo consensus node refresh --node-aliases node1 --deployment "${SOLO_DEPLOYMENT}"
# or
solo consensus node restart --deployment "${SOLO_DEPLOYMENT}"

    Resource allocation:

    • Ensure your machine and Docker (or other container runtime) meet the minimum requirements described in System readiness.
    • On Docker Desktop, check: Settings > Resources.

MinIO image pull failures (ImagePullBackOff on the MinIO tenant pod)

MinIO stopped publishing new community-edition images as of 2025-10-23. quay.io/minio/minio and docker.io/minio/minio now return 401 UNAUTHORIZED for every tag, including latest. Solo uses this image for the MinIO tenant (the object storage backend for record streams and backups deployed via the MinIO Operator), so any Solo deployment that enables MinIO will fail with ImagePullBackOff on the tenant pod until the fix below is applied.

Note: This affects Solo versions below v0.91.0. Starting with v0.91.0, Solo replaces the default tenant image so this workaround is no longer required. If you’re on an older version, use one of the two options below: skip MinIO entirely with environment variables (simplest), or override the tenant image with a values file if you need MinIO’s record-stream/backup storage.

Symptoms

  • kubectl get pods -n "${SOLO_NAMESPACE}" -l v1.min.io/tenant=minio shows ImagePullBackOff or ErrImagePull.

  • kubectl describe pod <minio-pool-pod> -n "${SOLO_NAMESPACE}" shows an event like:

    Failed to pull image "quay.io/minio/minio:RELEASE.2024-08-03T04-33-23Z":
... unexpected status from HEAD request ... 401 UNAUTHORIZED

  • A one-shot or network deploy run hangs or fails on the “Deploy network node” step with [SOLO-3035] Pod readiness check failed ... for labels [v1.min.io/tenant=minio].

Easier option: skip MinIO entirely with environment variables

If you don’t need record-stream/backup storage and don’t want to deal with Helm values files, the simplest fix is to not deploy MinIO at all by running the network in native block-streaming mode instead:

export ONE_SHOT_WITH_BLOCK_NODE=true
export BLOCK_STREAM_STREAM_MODE=BLOCKS

solo one-shot single deploy

This works with one-shot single deploy (no need to switch to falcon) because it’s a plain environment variable, not a CLI flag — Solo detects it and skips the MinIO/record-uploader setup automatically. Use this unless you specifically need the MinIO-backed record-stream/backup path; if you do, use the values-file override below instead.

Workaround: override the tenant image with a values file

Point the MinIO tenant at a currently-available image using Solo’s --values-file support. Silo is a MinIO fork maintained by PGSTY that keeps publishing dated, S3-API-compatible images after upstream stopped; its entrypoint transparently translates the MinIO Operator’s server/minio invocation, so it’s a drop-in replacement here. Create a plain Helm values file:

# minio-image-override.yaml
minio-server:
  tenant:
    image:
      repository: docker.io/pgsty/silo@sha256
      digest: 635197cb9f36d01bee221d34d1c7d7960f6a95c48b0b6c01d99cd13bdae51a46

Then deploy with it:

solo network deploy -f ./minio-image-override.yaml

network deploy’s --values-file/-f passes this file straight to Helm, so the plain values shape above is correct there.

If you are using one-shot instead

one-shot’s --values-file is a different shape — it is not a raw Helm values file. Its top-level keys are per-command sections (network, mirrorNode, relayNode, etc.), each holding --flag: value pairs that get forwarded to the underlying command. To apply the same override through one-shot, nest the Helm values file above inside a network section:

# falcon-values.yaml
network:
  --values-file: ./minio-image-override.yaml

solo one-shot falcon deploy --values-file ./falcon-values.yaml

one-shot single deploy does not accept --values-file at all — the flag is only registered on one-shot falcon deploy. If your existing workflow uses solo one-shot single deploy, switch it to solo one-shot falcon deploy (same underlying deploy pipeline, single is just a preset without the values-file escape hatch) to apply this workaround.

Resource constraint errors (CPU / RAM / Disk)

Resource pressure is a common cause of Pending pods, slow startup, and repeated restarts.

  1. Check Kubernetes-level CPU and memory utilization:

    kubectl top nodes
kubectl top pods -n "${SOLO_NAMESPACE}"

  2. Check host and Docker disk usage:

    # Host disk availability
df -h

# Docker disk usage (if using Docker)
docker system df

  3. Compare against the recommended local baseline:

See System readiness for the recommended memory, CPU, and disk values.

JSON-RPC Relay Out of Memory

If the relay or relay-ws pods are being killed (OOMKilled) or restarting due to memory pressure, the sections below explain why this happens and how to resolve it.

Understanding the default memory configuration

Solo ships with a default memory limit of 88Mi and an explicit V8 old-space cap of 66MB (--max-old-space-size=66) for both the relay and WebSocket services. These values are tuned for the one-shot development profile and may not be sufficient for heavy workloads.

How Node.js memory works in containers

Since Node.js 12.7.0, Node.js reads the Linux cgroup memory limit set by Kubernetes to determine the V8 old-space heap size, rather than using the host’s physical memory. Based on V8’s internal heap sizing heuristics, this tends to be roughly ~50% of the container memory limit on 64-bit systems, though the exact value depends on V8 internals and varies at both ends of the memory spectrum.

A couple of things to be aware of:

  • cgroup v2 environments: many modern Linux distributions enable cgroup v2 by default, and Kubernetes 1.25 brought cgroup v2 support to GA. Older Node.js versions may not correctly detect the container limit under cgroup v2 and could silently fall back to the host’s physical memory, allocating a much larger heap than intended. This was improved in at least Node.js 20.3.0, which upgraded libuv to 1.45.0.
  • When --max-old-space-size is explicitly set (as in Solo’s default config), it overrides the auto-sizing entirely — the cgroup-based detection only kicks in when no explicit value is provided.

This means:

  • If you increase only the pod memory limit (e.g., to 256Mi) but leave NODE_OPTIONS unchanged, old space stays at 66 MB.
  • If you remove NODE_OPTIONS, Node.js will attempt to auto-size old space based on the container limit (roughly ~128 MB for a 256Mi pod on a modern Node.js version).

Adjusting memory for heavier workloads

Create a custom values file (e.g., custom-relay-values.yaml) and pass it when deploying:

# Option 1: Explicit old-space control (recommended for precise tuning)
relay:
  resources:
    limits:
      memory: 256Mi
  config:
    NODE_OPTIONS: '--max-old-space-size=192'
ws:
  resources:
    limits:
      memory: 256Mi
  config:
    NODE_OPTIONS: '--max-old-space-size=192'

# Option 2: Let Node.js auto-detect (simpler, old space ≈ 50% of limit)
# relay:
#   resources:
#     limits:
#       memory: 256Mi
#   config:
#     NODE_OPTIONS: ""
# ws:
#   resources:
#     limits:
#       memory: 256Mi
#   config:
#     NODE_OPTIONS: ""

Then deploy or upgrade with:

solo relay node add --deployment "${SOLO_DEPLOYMENT}" --values-file custom-relay-values.yaml
# or
solo relay node upgrade --deployment "${SOLO_DEPLOYMENT}" --values-file custom-relay-values.yaml

Connection refused errors

If you cannot connect to Solo network endpoints from your machine, use this sequence to isolate the issue.

  1. Verify services and endpoints inside the cluster

    # List all services
kubectl get svc -n "${SOLO_NAMESPACE}"

# Check if endpoints are populated
kubectl get endpoints -n "${SOLO_NAMESPACE}"

  2. Use manual port forwarding (bypass automation)

    If automatic port forwarding (from solo commands or your environment) is not working, forward the required services manually. The local ports below match the Solo 0.63+ defaults — adjust to any available port if needed:

    # Consensus node (gRPC) — local port 35211 → service port 50211
kubectl port-forward svc/haproxy-node1-svc -n "${SOLO_NAMESPACE}" 35211:50211 &

# Explorer UI — local port 38080 → service port 8080
kubectl port-forward svc/hiero-explorer -n "${SOLO_NAMESPACE}" 38080:8080 &

# Mirror node ingress (REST API) — local port 38081 → service port 80
kubectl port-forward svc/mirror-1-rest -n "${SOLO_NAMESPACE}" 38081:80 &

# Mirror node gRPC
kubectl port-forward svc/mirror-1-grpc -n "${SOLO_NAMESPACE}" 5600:5600 &

# JSON-RPC relay — local port 37546 → service port 7546
kubectl port-forward svc/relay-node1-hedera-json-rpc-relay -n "${SOLO_NAMESPACE}" 37546:7546 &

    Note: For Solo 0.62 and earlier, use local ports 50211, 8080, 5551, and 7546 respectively.

  3. Confirm the expected endpoints and ports

    After forwarding, connect to the local ports shown above (for example, http://localhost:38080 for the explorer).
    For the standard exposed endpoints after a successful one-shot deployment, see How to access exposed services (mirror node, relay, explorer).

Node synchronization issues

If nodes are not forming consensus or transactions are not being processed, follow these steps.

  1. Check node state and gossip logs:

    # Download state information for a node
solo consensus state download --deployment "${SOLO_DEPLOYMENT}" --node-aliases node1

# Check logs for gossip-related issues
kubectl logs -n "${SOLO_NAMESPACE}" network-node-0 | grep -i gossip

    Look for repeated connection failures, timeouts, or gossip disconnection messages.

  2. Restart problematic nodes:

    # Refresh a specific node
solo consensus node refresh --node-aliases node1 --deployment "${SOLO_DEPLOYMENT}"

# Or restart all nodes
solo consensus node restart --deployment "${SOLO_DEPLOYMENT}"

    After restarting, submit a small test transaction and verify that it reaches consensus.

Mirror node not importing records

If the mirror node is not showing new transactions, first confirm that records are being generated and imported.

  1. Verify the pinger is running

    The --pinger flag should be enabled when deploying the mirror node. The pinger sends periodic transactions so that record files are created.

    # Check if pinger pod is running
kubectl get pods -n "${SOLO_NAMESPACE}" | grep pinger

  2. Redeploy the mirror node with pinger enabled

    If the pinger is missing or misconfigured:

    # Destroy the existing mirror node
solo mirror node destroy --deployment "${SOLO_DEPLOYMENT}" --force

# Redeploy with pinger enabled
solo mirror node add \
  --deployment "${SOLO_DEPLOYMENT}" \
  --cluster-ref kind-${SOLO_CLUSTER_NAME} \
  --enable-ingress \
  --pinger

Helm repository errors

If you see errors such as repository name already exists, you likely have a conflicting Helm repo entry.

  1. List current Helm repositories:

    helm repo list

  2. Remove the conflicting repository:

    helm repo remove <repo-name>

# Example: remove hedera-json-rpc-relay
helm repo remove hedera-json-rpc-relay

Re-run the Solo command that configures Helm after removing the conflict.

Kind cluster issues

Problems starting or accessing the Kind cluster often present as cluster creation failures or missing nodes.

  1. Cluster will not start or is in a bad state:

    # Delete and recreate the cluster
kind delete cluster -n "${SOLO_CLUSTER_NAME}"
kind create cluster -n "${SOLO_CLUSTER_NAME}"

  2. Docker context or daemon issues

    Ensure Docker is running and the correct context is active:

    # Check Docker is running
docker ps

# On macOS/Windows, ensure Docker Desktop is started.
# On Linux, ensure the Docker daemon is running:
sudo systemctl start docker

Cleanup and reset (old installation artifacts)

Previous Solo installations can cause conflicts during new deployments.
For the full teardown and full reset procedure, see the Cleanup guide.

At a high level:

  1. Run a standard destroy first:

    solo one-shot single destroy

  2. If destroy fails or Solo state is corrupted, perform a full reset, which:

    • Deletes Solo-managed Kind clusters (names starting with solo).
    • Removes the Solo home directory (~/.solo).

Windows (PowerShell) issues

These issues are specific to running Solo natively from Windows PowerShell.

Paths use backslashes. Solo stores its files under $env:USERPROFILE\.solo on Windows - the equivalent of ~/.solo on macOS and Linux. When you copy a command that uses ~/.solo/..., replace it with $env:USERPROFILE\.solo\.... For example:

Get-Content $env:USERPROFILE\.solo\logs\solo.log -Wait -Tail 50

Environment variable syntax differs. PowerShell does not use export. Set a variable for the current session, or persist it for your user:

# Current session only
$env:SOLO_LOG_LEVEL = 'debug'

# Persist for your user (all future sessions)
[System.Environment]::SetEnvironmentVariable('SOLO_LOG_LEVEL', 'debug', 'User')

Removing a variable from the current session. Use Remove-Item on the Env: drive. Setting $env:VAR = '' only blanks the value; it does not remove the variable:

Remove-Item Env:\SOLO_LOG_LEVEL

Port-forwarding fails with listen EACCES. On Windows this is usually a WinNAT reserved-port-range conflict. Solo automatically restarts the WinNAT service and retries the port-forward. If the problem persists, restart WinNAT manually from an elevated PowerShell prompt:

net stop winnat
net start winnat

Collecting diagnostic information

Before seeking help, collect the following diagnostics so issues can be reproduced and analyzed.

Solo diagnostics

  1. Capture comprehensive diagnostics for the deployment:

    solo deployment diagnostics all --deployment "${SOLO_DEPLOYMENT}"

    This creates logs and diagnostic files under ~/.solo/logs/.

Key log files

These files are often requested when reporting issues:

FileDescription
~/.solo/logs/solo.logSolo CLI command logs (human-readable, pino-pretty)
~/.solo/logs/solo.ndjsonSolo CLI command logs (machine-readable JSON, for jq)

Kubernetes diagnostics

Collect basic cluster and namespace information:

# Cluster info
kubectl cluster-info

# All resources in the Solo namespace
kubectl get all -n "${SOLO_NAMESPACE}"

# Recent events in the namespace (sorted by time)
kubectl get events -n "${SOLO_NAMESPACE}" --sort-by='.lastTimestamp'

# Node and pod resource usage
kubectl top nodes
kubectl top pods -n "${SOLO_NAMESPACE}"

Getting Help

1. Check the Logs

Always start by examining logs:

# Solo logs
cat ~/.solo/logs/solo.log | tail -100

# Pod logs
kubectl logs -n "${SOLO_NAMESPACE}" <pod-name>

2. Documentation

3. GitHub Issues

Report bugs or request features:

When opening an issue, include:

  • Solo version (solo --version)
  • Operating system and version
  • Docker/Kubernetes versions
  • Steps to reproduce the issue
  • Relevant log output
  • Any error messages

4. Community Support

Join the community for discussions and help:

Error Codes

Complete reference for all Solo error codes, including troubleshooting steps and ownership classification.

Last modified September 24, 2026: docs: add documentation minio workaround (#319) (2fadd45)