A Developer's Guide To Running A Bitcoin Full Node And Local Block Explorer

/

The goal of this guide is set up a Bitcoin Node that is useful for tinkering with as a developer. If you complete this guide successfully, you'll have a node that is in sync with the network, contains the full history of the Bitcoin blockchain, and is fully validating all transactions.

 
Obligatory Image Evoking the idea of networked nodes

Obligatory Image Evoking the idea of networked nodes

 

You'll also be able to communicate with your node via simple JSON-RPC calls. This means you can interact with the Bitcoin Network at large without relying on any trusted third party. As a simple example of what can be done with this, we'll run an open source block-explorer, providing you with a trustless interface to view activity on the network.

Prerequisites

Because this guide is for developers, it assumes you're comfortable doing developery things, such as running Linux, using bash, and executing Python scripts. We will not utilize the Bitcoin QT GUI to set up our wallet-- in fact none of the steps in this guide will assume a GUI at all.

Where more in-depth usage of some other tech is an out of scope pre-requisite, I'll provide links to useful guides as appropriate for those portions. If you're looking to run a full node as a standard user, check out the guide on bitcoin.org.

There are a few hardware and software requirements, as enumerated below:

  • A modestly spec'd computer with at least 2 GB of RAM (4+ suggested)
  • At least 150 GB of free disk space
  • A reasonably fast network connection without stringent bandwidth requirements
  • An installation of Ubuntu 18.04
    • If you're running this on your dev machine that already has macOS or Windows installed, you can do so in VM
    • Older versions of Ubuntu probably work fine as well, but weren't tested
    • Other variants of Linux should also work fine, but you'll have to install bitcoind and handle dependencies as appropriate for your distro (more below)

I run my node with fast residential broadband on a 6 year old desktop PC. It has an Athlon FX processor, 4 GB of RAM, and slow 250 GB spinning disk. It runs like champ! The Bitcoin Core team's focus on making it possible for anyone to run a node truly has paid off.

Installing bitcoind

The Bitcoin network follows a protocol for communication and consensus. While several clients for this protocol exist, the "canonical" and most widely used version is referred to as Bitcoin Core.

To install Bitcoin Core on Ubuntu, you have a few choices:

  1. Clone the repository from GitHub and build from source
  2. Download and install official pre-compiled binaries
  3. Installing the binaries via apt from the Bitcoin Personal Package Archive (PPA)

Each method has its tradeoffs. In this guide, we'll opt for using apt and the PPA, because it's the simplest and easiest way, and makes upgrading to new versions of the software easy as well. One downside is it requires trusting the software that comes from the PPA which could, for example, be compromised by an attacker. Please evaluate these tradeoffs yourself and choose the method thats best for you.

If you're just using your node for development and tinkering and want to proceed with option #3, simply run the following commands and follow the associated prompts:

sudo apt-add-repository ppa:bitcoin/bitcoin
sudo apt-get update
sudo apt-get install bitcoind

When these commands are complete successfully, run bitcoind -version. You should see some output showing the version of the bitcoind software you're using.

Initial Block Download

You've now got the latest software needed to run a node, but you're missing 10+ years of the network's history. Remember, this is a full node! That means it stores and validates the full history of the blockchain, starting from the very first block mined by Satoshi himself. That history will need to be downloaded.

Unfortunately, this process takes a while and requires about 150 GB of disk space as of today. On my reasonably fast residential internet, I was able to download the full blockchain in about 4 days. Your mileage may vary based on your connection and geography.

Run the following command:

bitcoind -daemon -txindex=1

This runs the bitcoind process in the background with transaction indexing on. You may also want to start this process automatically on every reboot as well. You can do this by running: crontab -e and adding the following line:

@reboot bitcoind -daemon

Note that we didn't add the txindex option to the crontab. Later in this guide we'll put this option in a config file instead, which is a better way to set options for your node.

When the node software starts for the first time, it creates ~/.bitcoin/. Let's navigate there now and poke around:

cd ~/.bitcoin/
ls

You'll notice a number of files and directories have been created. We'll examine just a few of these now. First, run tail -f debug.log. You'll notice a series of log messages that look something like this:

2018-08-10 17:22:15 UpdateTip: new best=00000000000000000011b735b8fde14db39704c446dd2b1ae4b8db198c255e0d
height=536128 version=0x20000000 log2_work=89.424049 tx=333938835
date='2018-08-10 17:22:26' progress=1.000000 cache=71.6MiB(529075txo)

(When the node first begins syncing, these may be flying by pretty fast. Hit Ctrl-C to quit tailing the file and examine the messages).

These messages represent blocks being downloaded from the network. The one shown here represents the latest block mined at the time I'm writing this tutorial, but you'll be well behind this. You'll note each message shows a number of parameters, including the hash of the block (best=X), the block height (height=X), and a timestamp of when this block was mined (date=X).

To monitor the progress of the initial download, you can tail this file and look at the historical date you've caught up to. Note that, since most blocks early in the network's history were mostly empty, early blocks will download fast. As you get further along, download times will increase. My node spent a majority of the time downloading blocks from 2017 onwards.

You can also interact with the blockchain using the bitcoin-cli command line interface. For example, you can run bitcoin-cli stop to safely shutdown your node, or run bitcoin-cli -getinfo to get a summary of your node's status:

{
  "version": 160000,
  "protocolversion": 70015,
  "walletversion": 159900,
  "balance": 0.00000000,
  "blocks": 536128,
  "timeoffset": 0,
  "connections": 33,
  "proxy": "",
  "difficulty": 5949437371609.53,
  "testnet": false,
  "keypoololdest": 1531965982,
  "keypoolsize": 1000,
  "paytxfee": 0.00000000,
  "relayfee": 0.00001000,
  "warnings": ""
}

One more small piece of fun. The block data that your node is downloading is stored in binary files in the ~/.bitcoin/blocks/ directory. You won't normally interact with this data directly, but try running this quick command: 

head -c 210 ~/.bitcoin/blocks/blk00000.dat && echo '\n'`.

You'll see something like this:

��;���z{�z�,>gv�a���Q2:���K^J)�_I���+|����M��E
The Times 03/Jan/2009 Chancellor on brink of second bailout for banks��

Do you recognize that sentence? If you do, it might send chills down your spine to see it on your machine! If you don't, go Google it and learn why it should.

As mentioned, initial block download is going to take at least a few days. During this phase, the daemon may use a lot of resources. Mine pegged my processor at a consistent 20-30%, utilized over 2 gigs of RAM, and generally made my admittedly old machine slow. Once you're caught up, running the node software has a negligible effect on performance with even a modest machine.

If you've made it this far, bookmark the page and come back in a few days when your node is in sync!

Server Configuration

Welcome back! It's time to configure your node to act as a JSON-RPC server, so we can run and write code in our favorite language to interact with it.

Navigate back to the bitcoin directory and create a flat-text file called bitcoin.conf.

cd ~/.bitcoin/
touch bitcoin.conf

Henceforth, this file will be read when bitcoind starts and will be used to configure options for your node. There are a number of options you can set, and each of these can also be set at command line as well. To get a sense of the options available to you, run:

bitcoind --help

Open up the bitcoin.conf file in your favorite text editor and add the following lines:

## Configuration for the Bitcoin Core Daemon

# Accept command line and JSON-RPC commands
server=1

This is a good start, but not sufficient to enable RPC interaction with our node. To do so, we also have to create authentication credentials for the user who will log in.

It used to possible to specify a username and password in flat text in the config file, but this method is clearly insecure and has been deprecated. Instead, we have to provide the username along with the hash value of the password-- in conjunction with a salt used during hashing.

To generate this, use the following commands to download the canonical Python script provided by the Bitcoin Core team and run it with your own username and password.

wget https://raw.githubusercontent.com/bitcoin/bitcoin/master/share/rpcauth/rpcauth.py
python3 rpcauth.py myusername mysecretpassword

Be sure to replace the username and password values with something sane! The script should produce a result like this:

String to be appended to bitcoin.conf:
rpcauth=myusername:d2a45f8bd56a2014461e2f70f6783b56$636211cdbf3f943a887df72d9944be13ae61c03ef43c6d06dd5044debd4fcbd7
Your password:
mysecretpassword

As instructed, add this line to your bitcoin.conf file, which should now look something like this:

## Configuration for the Bitcoin Core Daemon

# Accept command line and JSON-RPC commands
server=1

# Index all transactions
txindex=1

# Auth Credentials For JSON-RPC server
rpcauth=myusername:d2a45f8bd56a2014461e2f70f6783b56$636211cdbf3f943a887df72d9944be13ae61c03ef43c6d06dd5044debd4fcbd7

Next, restart your node to make sure the configuration changes take effect:

bitcoin-cli stop
bitcoind -daemon

At this point, you're ready to make JSON-RPC requests to your node, as long as they're coming from the node itself. To test this out, try running the following command (note: you may have to install curl first via apt).

curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getinfo", "params": [] }' -H 'content-type: text/plain;' http://127.0.0.1:8332

Be sure to replace myusername with username you actually created. You'll be prompted for the associated password, and if all worked properly, you should see something like the following:

{
    "result": {
        "chain": "main",
        "blocks": 536148,
        "headers": 536148,
        "bestblockhash": "00000000000000000028fab7a6b5035e2745fe06ef58d817bb32f35924198f97",
        "difficulty": 5949437371609.53,
        "mediantime": 1533926692,
        "verificationprogress": 0.9999864596463776,
        "initialblockdownload": false,
        "chainwork": "000000000000000000000000000000000000000002af5dee686cd058c3ec064d",
        "size_on_disk": 203955272825,
        "pruned": false,
        "softforks": ["..."],
        "bip9_softforks": { "..." : "" },
        "warnings": ""
    },
    "error": null,
    "id": "curltest"
}

If this works, you're ready to start using your full node for building/running an app that interacts with it. To learn more about the full set of API calls available, and various libraries that exist that wrap the API, checkout this reference on bitcoin.org.

IP Whitelisting

By default, your node is configured to only accept JSON-RPC requests from localhost, i.e. the machine where it runs. This is fine if you'll be developing on the same box the node is on, and you can skip this section.

If you're developing from a different machine, like a Windows or macOS computer, you'll have to use the rpcallowip option to whitelist IP addresses which are allowed to make calls, and also ensure port 8332 is reachable. (Note that a VM, though running on the same hardware, does in fact have a unique IP).

My local development machine runs macOS, so I can use the ifconfig to determine my local ip, then whitelist it by adding something like rpcallowip=192.168.0.101 to my config file. Note that you can include this option more than once to whitelist multiple IP addresses.

The specifics of finding your development machine's IP and making port 8332 available on your node are dependent on your platform and network. These are left as an exercise to the reader. Here's an example of what your final bitcoin.conf might look like.

#### Configuration for the Bitcoin Core Daemon

# Keep my daily upload bandwith around 1 GB

maxuploadtarget=1024  # Note that if your node isn't exposed to the internet on
                      # Port 8333, you're not feeding data to other nodes and this
                      # setting won't matter.
                      # While not required, if you'd like to support the network
                      # by servering data, make sure your router is configured to
                      # forward port 8333 to the machine with your.

# Accept command line and JSON-RPC commands
server=1

# Index all transactions
txindex=1

# Auth Credentials For JSON-RPC
rpcauth=myusername:d2a45f8bd56a2014461e2f70f6783b56$636211cdbf3f943a887df72d9944be13ae61c03ef43c6d06dd5044debd4fcbd7

# Whitelist only my Mac's local IP for JSON-RPC calls
rpcallowip=192.168.0.101

Running A Block Explorer

With your full node up and running, synced to the network, and configured for use as JSON-RPC server, it's time to run your own local blockchain explorer! We'll use the open source btc-rpc-explorer, created by Dan Janosik.

This explorer is great because it doesn't use a database-- it just needs access to your node, which it interacts with via the RPC interface we just configured.

btc-rpc-explorer is written in JavaScript using Node.js, so you'll need to have the latest stable version of node and npm installed. If you don't already have these, checkout the instructions for your development platform of choice.

Once Node and npm are ready to go, clone the repo from GitHub and build it with the following commands (which assume a Unix dev environment):

git clone https://github.com/janoside/btc-rpc-explorer.git
cd btc-rpc-explorer
npm install
npm run

Next, open the app/credentials.js file in your text editor of choice and update the username and password to the values you used during the RPC configuration.

If you're dev box is not the same as your node, you'll also have to update the host parameter to the IP address of your node. This will likely be a local area network IP, unless your node is hosted outside your LAN. (Note also that your router may assign either you development box or your node a new IP in the future. Look into configuring static local IPs based on MAC addresses in your router's config panel to solve this issue).

With the credentials updated, you're ready to run the explorer. Execute the following command, then visit http://127.0.0.1:3002/ in your browser of choice.

npm start

If all has gone well, you should see something like this:

locally sourced, organic block explorers

Congratulations, you now have your very own Bitcoin full node, and a locally sourced, organic block explorer to go with it 🎉. Happy hacking!