Skip to content

How to Troubleshoot Teranode (Docker Compose)

Last modified: 22-January-2025

Index

Introduction

This guide provides troubleshooting steps for common issues encountered while running Teranode with Docker Compose.

Troubleshooting

Health Checks and System Monitoring

Service Status

docker-compose ps
This command lists all services defined in your docker-compose.yml file, along with their current status (Up, Exit, etc.) and health state if health checks are configured.

Detailed Container/Pod Health

docker inspect --format='{{json .State.Health}}' container_name

Replace container_name with the name of your specific Teranode service container. For example:

docker inspect --format='{{json .State.Health}}' asset

Monitoring System Resources

  • Use docker stats to monitor CPU, memory, and I/O usage of containers: 

    docker stats

  • Consider using Prometheus and Grafana for comprehensive monitoring.

  • Look for services consuming unusually high resources.
Viewing Global Logs
docker-compose logs
docker-compose logs -f  # Follow logs in real-time
docker-compose logs --tail=100  # View only the most recent logs
Viewing Logs for Specific Microservices
docker-compose logs [service_name]
Useful Options for Log Viewing
  • Show timestamps: 
    docker-compose logs -t
  • Limit output: 
    docker-compose logs --tail=50 [service_name]
  • Since time: 
    docker-compose logs --since 2023-07-01T00:00:00 [service_name]
Checking Logs for Specific Teranode Microservices
docker-compose logs [service_name]

For Docker Compose, replace [service_name] with the appropriate service or pod name:

  • Propagation Service (service name: propagation)
  • Blockchain Service (service name: blockchain)
  • Asset Service (service name: asset)
  • Block Validation Service (service name: blockvalidation)
  • P2P Service (service name: p2p)
  • Block Assembly Service (service name: blockassembly)
  • Subtree Validation Service (service name: subtreevalidation)
  • RPC Server (service name: rpc)
  • Postgres Database (service name: postgres) [Only in Docker]
  • Aerospike Database (service name: aerospike) [Only in Docker]
  • Kafka (service name: kafka-shared) [Only in Docker]
  • Kafka Console (service name: kafka-console-shared) [Only in Docker]
  • Prometheus (service name: prometheus) [Only in Docker]
  • Grafana (service name: grafana) [Only in Docker]
Redirecting Logs to a File
docker-compose logs > teranode_logs.txt
docker-compose logs [service_name] > [service_name]_logs.txt

Remember to replace placeholders like [service_name], and label selectors with the appropriate values for your Teranode setup.

Check Services Dashboard

Check your Grafana TERANODE Service Overview dashboard:

  • Check that there's no blocks in the queue (Queued Blocks in Block Validation). We expect little or no queueing, and not creeping up. 3 blocks queued up are already a concern.

  • Check that the propagation instances are handling around the same load to make sure the load is equally distributed among all the propagation servers. See the Propagation Processed Transactions per Instance diagram.

  • Check that the cache is at a sustainable pattern rather than "exponentially" growing (see both the Tx Meta Cache in Block Validation and Tx Meta Cache Size in Block Validation diagrams).

  • Check that go routines (Goroutines graph) are not creeping up or reaching excessive levels.

Recovery Procedures

Third Party Component Failure

Teranode is highly dependent on its third party dependencies. Postgres, Kafka and Aerospike are critical for Teranode operations, and the node cannot work without them.

If a third party service fails, you must restore its functionality. Once it is back, please restart Teranode cleanly following the instructions in the How to Start and Stop Teranode in Docker guide.

Should you encounter a bug, please report it following the instructions in the Bug Reporting section.