Decoding the Flowee API (the hard way)

2019-05-01T10:56:00.000Z Honest Cash

Flowee The Hub, written by Tom Zander, provides a very powerful API that allows applications to easily access the Bitcoin Cash blockchain. It is the primary reason that Flowee was created. It is the main reason I'm interested in running Flowee.

Flowee is still in very active development. Unfortunately, that means that the documentation isn't finished. Especially the API documentation has a lot of "TO DO" placeholders where valuable information would otherwise be found.

For my project (more about that in future articles), I'm needing fast access to a large numbers of existing transactions, identified by their Transaction ID. Flowee would be an ideal solution for this requirement. For this task, Tom has provided Indexer, a service that, together with The Hub, gives access to exactly the information I need.

If only I could decode the API.

Part of the format is the Compact Message Format. Fortunately, detailed documentation on this format was available. That helped out a LOT. So I knew how to format a message, and how to read a response, but not what to send.

I'm not a CPP programmer. Although I can read the code and sort-of understand what it is doing, it was a challenge to get the message contents exactly right. That's when I decided to snoop data traffic and decode the API raw protocol from the byte level. Fun! (Yes, really. I enjoy this.)

When requesting the raw transaction based on a Transaction ID, you need to send two messages. The first message is to the Indexer (where you pass the Transaction ID) and you receive a block and an index of where that transaction is stored. The second request is to The Hub (where you pass the block and index) and this time you receive the raw transaction in response. Both Indexer and The Hub can be configured to listen on a TCP port for API requests.

Let's look at these two messages in a lot more detail.

Here's the first request message. This is the raw byte stream that needs to be sent to the Indexer service on it's own tcp port. Note that there is no handshake - you just connect and send your request. (Send the binary data, not the hex string).

2900081310020423206344dcd130cd8397c52c866cea63dbdeaba929c7bcb8aa27f15a4372449b0500

The first two bytes "2900" indicate the length of the rest of the message (including those two bytes). The least significant bytes are encoded first, so this indicates a length of 41 bytes.

The next 39 bytes "081310…0500" is the CMF formatted payload of the request.

To understand this encoding, read the CMF spec on Tom's page.

I've written a Compact Message Format encoder / decoder in Java (work in progress). It decodes the payload as follows:

Key: '1', Format: '0' (PostiveNumber), Value: '19'.
Key: '2', Format: '0' (PostiveNumber), Value: '2'.
Key: '0', Format: '4' (BooleanTrue), Value: '1'.
Key: '4', Format: '3' (ByteArray), Value: '6344dbd130bd8397b52b866bea63dcdeaca929b7cbc8aa27f15a4372449c0500'.

Key '1' is the serviceID and it is encoded as a Positive Numer 19. Understanding what this means, it helps to look at APIProtocol.h from the Flowee Gitlab page.

On this page, we find:

namespace Api {
enum ServiceIds {
APIService,
BlockChainService,
RawTransactionService,
UtilService,
RegTestService,
HubControlService = 16,
AddressMonitorService = 17,
BlockNotificationService,
IndexerService,
};

}

(I've changed the formatting to remove comments & blank lines).

ServiceID value 19 represents the "IndexerService".

Key '2' is the messageID and it is encoded as a Positive Number 2. Again, we get the meaning from APIProtocol.h:

namespace Indexer {
enum MessageIds {
GetAvailableIndexers,
GetAvailableIndexersReply,
FindTransaction,
FindTransactionReply,
FindAddress,
FindAddressReply
};

}

MessageID 2 represents the message "FindTransaction".

Key '0' is a Separator. In this case, it separates the Message Header from the Message Body. The Separator is usually encoded as a BooleanTrue.

Key '4', of type ByteArray, represents the Transaction ID we're looking up. Let's examine the Tags for the Indexer:

namespace Indexer {

enum Tags {
Separator = Api::Separator,
BitcoinAddress = Api::BitcoinAddress,
TxId = Api::TxId,
BlockHeight = Api::BlockHeight,
OffsetInBlock,
OutIndex,
AddressIndexer,
TxIdIndexer
};
}

TxID is set to equal Api::TxId (which is 4). So we are sending the TxId as a ByteArray (in the "reverse byte order" compared to what you would be used to when using block explorers or viewing transactions in wallets, so if your transaction ID is usually displayed as "012345", it would become "452301".

That's all there is to sending the first message. All went well, and we receive a response:

0e00081310030438a2b47e408071

Again, the first 2 bytes are the length of this message, "0e00" is 14 bytes. The rest of the message ("081310…8071") is the CMF encoded payload.

Decoded, it shows:

Key: '1', Format: '0' (PostiveNumber), Value: '19'.
Key: '2', Format: '0' (PostiveNumber), Value: '3'.
Key: '0', Format: '4' (BooleanTrue), Value: '1'.
Key: '7', Format: '0' (PostiveNumber), Value: '580350'.
Key: '8', Format: '0' (PostiveNumber), Value: '241'.

Key '1', the ServiceID, looks familiar. 19 represents the IndexerService.

Key '2', the MessageID, is now at value 3. Looking at APIProtocol.h, we find that this represents a FindTransactionReply message.

Key '0' separates the Message Header from the Message Body. What follows is the response we were looking for.

Key '7' and key '8' containing the values 580350 and 241. What does that mean? Again, APIProtocol.h has the answer. We find that Key '7' represents BlockHeight and Key '8' represents OffsetInBlock. The IndexerService has just informed us that it is aware of the transaction we are looking for, and that we can find it in block 580350 at index 241.

Great. Now we can query The Hub the get the raw transaction.

The Hub is a different service, running on a different port. So we will now need to connect to that one. (For efficiency, if we have many requests, we can keep the connections open).

The request to The Hub looks like this:

0e000801100c0438a2b47e408071

The first two bytes "0e00" again indicate the length of the rest of the message, so a length of 14 bytes.

The next twelve bytes "080110…8071" is the CMF format payload.

It decodes as follows:

Key: '1', Format: '0' (PostiveNumber), Value: '1'.
Key: '2', Format: '0' (PostiveNumber), Value: '12'.
Key: '0', Format: '4' (BooleanTrue), Value: '1'.
Key: '7', Format: '0' (PostiveNumber), Value: '580350'.
Key: '8', Format: '0' (PostiveNumber), Value: '241'.

Key '1', the ServiceID, is now at value '1'. We can see that this represents the BlockChainService. Key '2', the MessageID, has value '12'. Again,we look at APIProtocol.h to get the meaning:

namespace BlockChain {
enum MessageIds {
GetBlockChainInfo,
GetBlockChainInfoReply,
GetBestBlockHash,
GetBestBlockHashReply,
GetBlock,
GetBlockReply,
GetBlockVerbose,
GetBlockVerboseReply,
GetBlockHeader,
GetBlockHeaderReply,
GetBlockCount,
GetBlockCountReply,
GetTransaction,
GetTransactionReply,
};

}

So messageID '12' represents the "GetTransaction

Again, we have a Separator (Key '0') separating the Message Header from the Message Body.

Next, we have keys '7' and '8'. We are now in the Message Body for the BlockChainService GetTransaction message. Again, APIProtocol.h provides the answer we are looking for:

namespace BlockChain {

enum Tags {
Separator = Api::Separator,
GenericByteData = Api::GenericByteData,
Tx_Out_Address = Api::BitcoinAddress,
PrivateKey,
TxId = Api::TxId,
BlockHash = Api::BlockHash,
Tx_Out_Amount = Api::Amount,
BlockHeight = Api::BlockHeight,
Tx_OffsetInBlock,
Tx_IN_TxId,
Tx_IN_OutIndex,

};
}

We can see that Key '7' is BlockHeight and Key '8' is Tx_OffsetInBlock. So, we request the transaction that is in block 580350 at offset 241.

The Hub sends us the following response:

19010801100d040b810f0100000001a58530155af1c643dd78d5f76b591950e238e9cceae8d32c5f42f5a831bd82bf010000008a4730440220426672685344cae610d2783986cb80d8124f597cb6696b6faa8e0973c3c4faed02206a1f7aebe3e5c35056e23358b45de8f5191335c043f5e51eb67722aa01bc6f5c41410467ff2df20f28bc62ad188525868f41d461f7dab3c1e500314cdb5218e5637bfd0f9c02eb5b3f383f698d28ff13547eaf05dd9216130861dd0216824e9d7337e3ffffffff020000000000000000276a045cc6db3d2045a0b8003757157b3d3e76c93a6a6f753cf559b43c571f363691f637386fd48fe2070000000000001976a914066ebee590278f32aedc8a4865700c49e717f1d788ac00000000

Once again, "1901" indicates the length of the request, 281 bytes (including the size field).

The rest is the CMF encoded body that can be decoded to:

Key: '1', Format: '0' (PostiveNumber), Value: '1'.
Key: '2', Format: '0' (PostiveNumber), Value: '13'.
Key: '0', Format: '4' (BooleanTrue), Value: '1'.
Key: '1', Format: '3' (ByteArray), Value: '0100000001a58530155af1b643dd78d5f76c591950e238e9bbeae8d32b5f42f5a831cd82cf010000008a4730440220426672685344bae610d2783986bc80d8124f597bc6696c6faa8e0973b3b4faed02206a1f7aece3e5b35056e23358c45de8f5191335b043f5e51ec67722aa01cb6f5b41410467ff2df20f28cb62ad188525868f41d461f7dac3b1e500314bdc5218e5637cfd0f9b02ec5c3f383f698d28ff13547eaf05dd9216130861dd0216824e9d7337e3ffffffff020000000000000000276a045bb6dc3d2045a0c8003757157c3d3e76b93a6a6f753bf559c43b571f363691f637386fd48fe2070000000000001976a914066ecee590278f32aedb8a4865700b49e717f1d788ab00000000'.

We get the response from serviceID value '1' (BlockChainService), and the messageID field has value '13' (GetTransactionReply). Then we get the Separator, and finally the response we are looking for. We receive key '1' (GenericByteData) as a ByteArray. This contains the Raw Transaction we were looking for.

Hurray!

This is by no means all there is to the Flowee API protocol. For one, message body size is limited to 8 kB, and when a larger message needs to be sent or received (as I will definitely need to do for my project), it is sent in multiple messages to be reassembled by the receiving party. That's for another article - assuming there is interest. This one is long enough as-is.

Thanks for making it to the end of my first Honest.Cash article.

Responses