v23 keeps using wrong link to download ledger. #63

Summary

This is mainly for people running Pi Node on macOS. It may also apply to Linux if anyone is having similar sync issues, but I only tested this on Mac. I have not used Windows in years, so I cannot help much there.

After my node updated to v23 in the Pi Node app, it would sync correctly for a while, then suddenly fall behind again.

The Pi Node app would show something like:

State: Catching up
Latest block: hours ago
Incoming connections: 0 or 1
Supporting other nodes: No / Yes

After checking the Docker container, Stellar Core logs, router settings, and Pi app files, I found that the Pi Network app kept rewriting the Stellar history archive URL back to:

https://history.temp.testnet2.minepi.com/

That URL was failing for me with SSL/certificate errors, 503 errors, or missing history archive state files.

The URL that worked for me was:

https://history.testnet2.minepi.com/

When the node used the correct URL, it could download history files, catch up, and continue syncing. When the Pi app changed it back to the temp URL within about 15 minutes of starting, the node got stuck again.

Symptoms

My node showed one or more of these:

State: Catching up
Latest block: hours ago
Missing HAS for ledger ...
Catchup failed
Could not download file

In the Stellar Core log, I saw lines like:

Missing HAS for ledger XXXXXXX: maybe stale archive validator1
Catchup failed

More importantly, I saw it trying to download from the bad temp archive:

curl -sf https://history.temp.testnet2.minepi.com//history/...

The double slash was also present:

minepi.com//history/...

Networking checks I did first

Before blaming the history archive, I checked the usual networking issues.

Check Docker port mapping

docker ps --format "table {{.Names}}\t{{.Ports}}" | grep testnet2

My container showed:

0.0.0.0:31402->31402/tcp
0.0.0.0:31403->1570/tcp
0.0.0.0:31401->8000/tcp

The important peer port is:

31402 TCP

Check whether the container is listening

The container did not have ss or netstat, so I checked /proc/net/tcp:

docker exec -it testnet2 bash -lc '
python3 - <<'"'"'PY'"'"'
import socket

wanted = {31402, 1570, 8000}

def decode_ip(hex_ip):
    return socket.inet_ntoa(bytes.fromhex(hex_ip)[::-1])

with open("/proc/net/tcp") as f:
    next(f)
    for line in f:
        parts = line.split()
        ip_hex, port_hex = parts[1].split(":")
        port = int(port_hex, 16)
        state = parts[3]
        if port in wanted:
            print(f"{decode_ip(ip_hex)}:{port}", "LISTEN" if state == "0A" else state)
PY
'

Good result:

0.0.0.0:31402 LISTEN

Router port forwarding

I only forwarded:

WAN TCP 31402 -> Mac LAN IP TCP 31402

I did not forward 31401 or 31403, because those map to local/admin services:

31401 -> 8000
31403 -> 1570

I also turned UPnP off and used a manual port forward.

CGNAT check

I compared:

curl -4 ifconfig.me

with my router’s WAN/Public IPv4 address.

They matched, so I was not behind CGNAT.

How I found the bad history URL

Inside the container:

docker exec -it testnet2 bash -lc '
grep -n "HISTORY=" /opt/stellar/core/etc/stellar-core.cfg
'

Bad result:

HISTORY="curl -sf https://history.temp.testnet2.minepi.com//{0} -o {1}"

Good result:

HISTORY="curl -sf https://history.testnet2.minepi.com/{0} -o {1}"

I also found that the config was host-mounted from macOS here:

~/Library/Application Support/Pi Network/docker_volumes/testnet_2/stellar/core/etc/stellar-core.cfg

Check the host file:

grep -n "HISTORY=" "$HOME/Library/Application Support/Pi Network/docker_volumes/testnet_2/stellar/core/etc/stellar-core.cfg"

Manual fix

This fixes the host-mounted config:

HOST_CFG="$HOME/Library/Application Support/Pi Network/docker_volumes/testnet_2/stellar/core/etc/stellar-core.cfg"

cp "$HOST_CFG" "$HOST_CFG.backup.$(date +%Y%m%d_%H%M%S)"

sed -i '' 's#https://history.temp.testnet2.minepi.com//*#https://history.testnet2.minepi.com/#g' "$HOST_CFG"

grep -n "HISTORY=" "$HOST_CFG"

Then restart the container:

docker restart testnet2

Verify inside the container:

docker exec -it testnet2 bash -lc 'grep -n "HISTORY=" /opt/stellar/core/etc/stellar-core.cfg'

You want to see:

https://history.testnet2.minepi.com/{0}

Then check the node:

docker exec -it testnet2 stellar-core http-command info

Eventually you want:

"state" : "Synced!"

or at least active catchup progress.

The annoying part

The Pi Network desktop app kept rewriting the config back to the temp URL.

For me, it seemed to happen when the app restarted the node, changed focus, or refreshed its Docker/config state.

I tried locking the file with:

chflags uchg "$HOST_CFG"

but that caused the container to fail on startup because the Pi app/container apparently needs to write to the file.

So I do not recommend locking the file.

Undo the lock if you tried it:

chflags nouchg "$HOST_CFG"

Best workaround I found

The most reliable workaround was to run a small watcher script that automatically patches the config back whenever the Pi app changes it.

Create the script:

cat > ~/pi-history-auto-fix.sh <<'SH'
#!/bin/bash

HOST_CFG="$HOME/Library/Application Support/Pi Network/docker_volumes/testnet_2/stellar/core/etc/stellar-core.cfg"
BAD="history.temp.testnet2.minepi.com"
GOOD="https://history.testnet2.minepi.com"
LAST_RESTART=0

patch_host() {
  sed -i '' 's#https://history.temp.testnet2.minepi.com//*#https://history.testnet2.minepi.com/#g' "$HOST_CFG"
}

patch_container() {
  docker exec testnet2 bash -lc '
    sed -i "s#https://history.temp.testnet2.minepi.com//*#https://history.testnet2.minepi.com/#g" /opt/stellar/core/etc/stellar-core.cfg 2>/dev/null || true
  ' 2>/dev/null || true
}

while true; do
  if [ -f "$HOST_CFG" ] && grep -q "$BAD" "$HOST_CFG"; then
    echo "$(date): bad history URL found in host config"

    patch_host
    patch_container

    NOW=$(date +%s)
    if [ $((NOW - LAST_RESTART)) -gt 30 ]; then
      echo "$(date): restarting testnet2 after patch"
      docker restart testnet2 >/dev/null 2>&1 || true
      LAST_RESTART=$NOW
    fi

    echo "$(date): patched back to $GOOD"
  fi

  docker exec testnet2 bash -lc "grep -q '$BAD' /opt/stellar/core/etc/stellar-core.cfg" 2>/dev/null
  if [ $? -eq 0 ]; then
    echo "$(date): bad history URL found inside container"

    patch_container

    NOW=$(date +%s)
    if [ $((NOW - LAST_RESTART)) -gt 30 ]; then
      echo "$(date): restarting testnet2 after container patch"
      docker restart testnet2 >/dev/null 2>&1 || true
      LAST_RESTART=$NOW
    fi

    echo "$(date): container patched back to $GOOD"
  fi

  sleep 3
done

chmod +x ~/pi-history-auto-fix.sh

Run it in a dedicated Terminal tab:

~/pi-history-auto-fix.sh

Leave it running while the Pi app/node is running.

After it patches the file, restart the container once:

docker restart testnet2

Then check:

docker exec -it testnet2 stellar-core http-command info

Monitoring commands

Check whether the bad URL came back

Host file:

grep -n "HISTORY=" "$HOME/Library/Application Support/Pi Network/docker_volumes/testnet_2/stellar/core/etc/stellar-core.cfg"

Container file:

docker exec -it testnet2 bash -lc 'grep -n "HISTORY=" /opt/stellar/core/etc/stellar-core.cfg'

Check node status

docker exec -it testnet2 stellar-core http-command info

Check recent catchup/history logs

docker exec -it testnet2 bash -lc '
grep -Ei "history.temp|history.testnet2|Missing HAS|Catchup failed|Catchup finished|Changing state|Ledger close complete" /tmp/stellar-core.log | tail -120
'

Check peers/incoming connections

docker exec -it testnet2 stellar-core http-command peers

If inbound is null, you currently have no incoming authenticated peers:

"inbound" : null

If inbound is working, you will see an inbound peer list.

TL;DR

1. Patch the host-mounted stellar-core.cfg back to:

https://history.testnet2.minepi.com/

2. Run an auto-fix watcher so the Pi app cannot leave it on:

https://history.temp.testnet2.minepi.com/

3. Restart the testnet2 container after patching.

4. Let the node catch up.

Suggested fix

The Pi Node app should stop rewriting the Testnet2 history archive URL to:

https://history.temp.testnet2.minepi.com/

or the temp archive should be fixed so it serves the same working history archive state/files as:

https://history.testnet2.minepi.com/

Right now, the app appears to overwrite the working config after the node is already running, which causes nodes that were catching up correctly to fall behind again.