Suggested speed up #1 for Tokenized

Why not restrict spynode sync to only search nodes from this list:

I am not sure what the advantage would be. It is easy enough for spynode to curate its own list of nodes to talk to and doesn’t require a third party.

from looking at the verbose console output of some sync logs I ran, I get the impression it is looking
at for example 10,000+ nodes in some case whereas according to some comments by NChain,
only about 400 nodes “do anything useful” and those 400 nodes are the ones in the blockchai api.

I may be costing me a lot of time waiting for a fresh sync to complete when sometimes I have to reset
the sync (by deleting the folder and the .json as you described).

It would be very interesting to have an option in Tokenized to use the blockchair list for spynode and
then see if the sync takes a lot less time to get to to a stable point.

During the synchronization process spynode is not scanning those nodes. It only does that when it is in sync and it is trying to make friends to monitor for double spend attempts. It does do a lot of scanning because many nodes will give out thousands of supposed peers without having verified any, but that shouldn’t slow anything down unless you are on a very restricted internet connection, and in that case it doesn’t make sense to be running spynode in the first place.

How can thousands of extra peers to consider not slow it down?
I am running on a normal 4G Wifi in Manhattan and it does finally get done in less than 2 hours roughly so is not a big problem for testing but the 10,000 fills create a large verbose log and so on.

I had another thing happen where the verbose log had 73,000 retries listed after I successfully did
build --tx --send of my ContractOffer on Sunday. The very last of the 73,000 was the one which had
the txid I needed from the ContractOffer. I took a guess on that and was right.
So want all that stuff trimmed down if possible.

I now have the luxury of being able to run Tokenized end to end processing hundreds of times, making
every possible developer error along the way and then refining my python not to make those mistakes again.

Like I said they aren’t happening during sync, so it isn’t slowing down sync. After you are synced, they are all happening in different threads, so they shouldn’t be interfering with any real processing because all they are doing is pinging an IP and doing a handshake. You don’t need to have verbose logging on if you want a more reasonable log file.

I am not sure what you mean about 73,000 retries. What is being retried 73,000 times? Nothing in spynode would be creating 73,000 different txids.

Here is the last of the approx 73,000 log items:
The last Re-Requesting in the splitlines[] array is:
b’2020/06/07 02:39:01.847755 [SpyNode] tx_tracker.go:77 Verbose - Re-Requesting tx : 1d10276e844866509edfae726c57d5ce1e8258b0b9a9f05921c0889039f8fa81’

That error means there is a problem with that tx propagating. Spynode has previously request that tx and did not receive it before the timeout so it requests it from a different node. It can happen more when the network is congested and nodes are taking too long to respond to requests, or if your internet connection is too slow.

It has little to do with the list of nodes being used as the only nodes that will get those requests have already been confirmed to be on the correct chain of headers.

Well spynode on my machine is stalling right now. I even shutdown and restarted my VM but spynode is again stalled. So OK it is network congestion. I suggest that there is no need for spynode to use the network. It should get the list of block from one of those public apis I described because that is their job to keep the list of blocks and will not be affected by network congestion.
It is 15:49 right now and you can see here that it is stalled from 15:35:

2020/06/09 15:35:18.926462 [Main] wallet.go:161 Info - Loaded wallet with 0 outputs, 0 unspent, and balance of 0.00000000
2020/06/09 15:35:18.927000 [Main] wallet.go:164 Info - Wallet address : 16GgqN2sWSTZAZobNkR6uLVUUmD8LnnSPM
2020/06/09 15:35:18.927160 [Main] node.go:114 Info - Contract address : 1QAinrbEfjv46jdpwZcqc1nB4tWH9AE8in
2020/06/09 15:35:18.927580 [SpyNode] peers.go:94 Verbose - Loaded 0 peers
2020/06/09 15:35:23.892723 [SpyNode] node.go:112 Info - Loaded blocks to height 638655
2020/06/09 15:35:23.893102 [SpyNode] node.go:116 Info - Start block height 560000
2020/06/09 15:35:23.893248 [SpyNode] transactions.go:45 Verbose - lypka No unconfirmed txs to load
2020/06/09 15:35:23.893373 [SpyNode] node.go:151 Verbose - Connecting to
2020/06/09 15:35:23.898420 [SpyNode] node.go:862 Info - Safe tx delay : 10 ms
2020/06/09 15:35:23.898684 [SpyNode] version.go:31 Verbose - ( Version : /Bitcoin SV:1.0.1/ protocol 70015, blocks 638655
2020/06/09 15:35:23.899052 [SpyNode] node.go:707 Verbose - ReadMessage : Unknown Command : protoconf
2020/06/09 15:35:23.900152 [SpyNode] headers.go:57 Info - Headers in sync at height 638655
2020/06/09 15:35:23.900216 [SpyNode] headers.go:64 Info - Blocks in sync at height 638655
2020/06/09 15:35:24.066365 [Main] node.go:380 Info - No new blocks found
2020/06/09 15:35:24.066578 [Main] node.go:384 Info - Balance : 0.00000000
2020/06/09 15:35:28.899191 [SpyNode] node.go:900 Verbose - Found 0 peers with no score

OHH I did another experiment:
I did a run where I skipped my usual sync --nostop and just did the sync without nostop and that completed quick, no problem.
So I think this pattern of output from the --nostop means that spynode sync is complete from a previous run and I am good to go to run the sync without --nostop which should stop automatically if good. So I think I need to add code to recognize that the --nostop has settled to a good state because it already
discovered enough peers in a previous run.

The big sync --nostop I ran a few days ago after clearing out spynode folder and the .json took about 4.5 hours for me.

First, I think I misunderstood what you were saying before. When you run --nostop it doesn’t stop running until you tell it to stop with CTRL-C. --nostop lets it run while in sync so that it will explore the network and find peers. Without --nostop, it will just run until it has processed all the blocks from the full node and is “in sync”. Until this message Blocks in sync at height 638655.

Second, it says Loaded 0 peers. If you have let it run --nostop for a while, then it shouldn’t be zero unless you have deleted the peers file in the spynode folder. If you have, then just let it run nostop for a while and then stop with CTRL-C. If you are running on Windows you might have to do something different to tell it to stop. If you kill it the wrong way it will not save the peers file.

Again, using the Bitcoin network directly is the best solution for us right now. We will likely add other options in the future, but we have other priorities right now.