TzKT v1.4 released with improved smart contract data and WebSocket API

Reviewing changes and new features in the latest version of TzKT, free and open-source Tezos indexer, Tezos API and Tezos explorer

Max
Max   Follow

Today we released TzKT v1.4 (opens new window), another version of the free open-source Tezos blockchain indexer and API with several important features, enabling more advanced use cases and making developers' life much easier.

Let's walk through the changes, but first of all, pay attention to the breaking changes.

# Breaking bad changes

# For API users:

  • parameters field in transactions has been deprecated and will be removed in the next major release. Use the parameter field instead, and use query parameters ?entrypoint= and ?parameter= for filtering.

We tried to smooth this change and added a kind of backward compatibility for most common use cases, so nothing should break actually. However, we strongly recommend you update your apps as soon as possible.

The most relevant API documentation is always available on api.tzkt.io (opens new window). Feel free to reach us in our telegram chat or in Tezos dev slack (opens new window) with any questions.

# For database users:

  • Parameters column has been dropped from the TransactionOps table. Use Entrypoint, RawParameters and JsonParameters columns instead;
  • Kind column in the Accounts table has been extended with another possible value 2 - asset, in addition to 1 - smart contract and 0 - delegator contract.

# For self-hosted:

TzKT has migrated to the newer .NET 5.0 so you need to install .NET 5.0 instead of .NET Core 3.1. Check out the installation instruction on GitHub (opens new window).

# Developer-friendly smart contract data

Smart contracts in Tezos are very flexible, which is great because it allows developers to do the same things in different ways. On the other hand, it makes life much more difficult for end users because of the ambiguity of the data.

For example, these are three equal ways to specify transaction parameter to call the initiate entrypoint in the Atomex smart contract (opens new window):

  • %default Left Left Pair (...);
  • %fund Left Pair (...);
  • %initiate Pair (...).

Another example, these are three equal values, representing the same Tezos address:

  • 0x01e2cf37d9997a1cf9b0995394264c289df016996a00;
  • KT1VG2WtYdSWz5E7chTeAdDPZNy2MpP8pTfL%default;
  • KT1VG2WtYdSWz5E7chTeAdDPZNy2MpP8pTfL;

It's really hard to work with data that can be represented by completely different values and which also should be parsed/handled differently, especially if you are new to Tezos and simply do not know all the specifics.

Thus, we have taken a big step toward making Tezos blockchain data (and, in particular, smart contract data) more user-friendly.

# Normalized JSON parameters

Starting from v1.4 the TzKT indexer automatically normalizes transaction parameters so that there is no more ambiguity. Looking at the examples above, it no longer matters how users pass parameters: with %default or %initiate entrypoint, in optimized 0x01e2cf3... or readable KT1VG2W... form. TzKT automatically parses, analyzes and converts it to a single readable form.

Then, TzKT transforms Micheline to human-readable JSON, using annotations from contract scripts, and it's like a breath of fresh air for any developer. Just imagine you are working not with this:

{
    ...
    {
        "prim":"Right",
        "args":[
            {
            "prim":"Left",
            "args":[
                {
                    "prim":"Pair",
                    "args":[
                        {
                        "bytes":"00009472982d7f6b096bc57d6da95e0b8ec8ee37e72f"
                        },
                        {
                        "prim":"Pair",
                        "args":[
                            {
                                "bytes":"0000bf97f5f1dbfd6ada0cf986d0a812f1bf0a572abc"
                            },
                            {
                                "int":"10000"
                            }
                        ]
                        }
                    ]
                }
            ]
            }
        ]
    }
    ...
}

but with this:

{
   "entrypoint":"transfer",
   "value":{
      "from":"tz1ZAwyfujwED4yUhQAtc1eqm4gW5u2Xiw77",
      "to":"tz1d75oB6T4zUMexzkr5WscGktZ1Nss1JrT7",
      "value":"10000"
   }
}

For more flexibility we also added the ?micheline= query parameter to configure the format of the transaction parameters. The following values are allowed:

NOTE

Both rawString and jsonString are very useful when you do strongly typed deserialization and you don't know the type of the parameter until you read the entrypoint value. So you can first deserialize the transaction, then read the entrypoint value and then, depending on the entrypoint, deserialize the parameter string to a specific type (it's a kind of lazy deserialization).

# Filter transactions by parameter like a boss

Working with normalized JSON parameters is surely great, but what would be even better is the ability to request transactions with specific parameters, and TzKT pushes the envelope.

In addition to the wildcard pattern (the .as query parameter mode, available in earlier versions), we've added the "json" pattern, that allows you to specify the path to a particular field within the JSON parameter to filter by. It looks like this: ?parameter[.path][.mode]=....

A few examples to make you realize it:

  • How to get all incoming kUSD transfers for the account tz123...? Easily: parameter.to=tz123... (example (opens new window)).
  • How to get all Dexter XTZ to USDtz trades with the amount > 1k? Easily: parameter.minTokensBought.gt=1000000000 (example (opens new window)).
  • How to get Atomex atomic swaps with the refund time in Feb 2021? Easily: parameter.settings.refund_time.as=2021-02-* (example (opens new window)).

Also, now you can use the ?entrypoint= query parameter to filter transactions by entrypoint. A few examples:

  • How to get all Dexter wXTZ/XTZ trades? Easily: entrypoint.in=xtzToToken,tokenToXtz,tokenToToken (example (opens new window)).
  • How to get all operations with tzBTC related to a specific account? Easily: entrypoint.in=mint,transfer,burn&parameter.as=*tz123* (example (opens new window)).

# Build Micheline parameters from JSON parameters

We’ve added an endpoint (opens new window) for building Micheline parameters from its JSON representation. All you need is to specify a smart contract, an entrypoint, and the JSON value you want to convert:

{
    "from": "KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn",
    "to": "KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn",
    "value": "1234"
}

and you will get (opens new window) raw Micheline that can be attached to a transaction and then forged and broadcasted to the blockchain:

{
"prim": "Pair",
    "args": [
        {
            "bytes": "01a3d0f58d8964bd1b37fb0a0c197b38cf46608d4900"
        },
        {
            "prim": "Pair",
            "args": [
                {
                    "bytes": "01a3d0f58d8964bd1b37fb0a0c197b38cf46608d4900"
                },
                {
                    "int": "1234"
                }
            ]
        }
    ]
}

Of course, to convert JSON to Micheline you need to know the format (schema) of the parameter to provide a valid JSON value. We have added this information to the tzkt.io (opens new window) explorer, so that you can easily find schemas for all entrypoints and all smart contracts.

In fact, such things are also available in popular Tezos SDKs like Taquito (opens new window), PyTezos (opens new window), etc., but they obviously do not cover all languages, but the HTTP API does, so those developers who don’t use these SDKs will be able to interact with smart contracts in a convenient way, using the API.

TzKT will eventually be supplemented with more such tools, so that it won’t be only a data API, but also a toolset for developers.

# Normalized JSON storage

We’ve added indexing of contract storages, which are automatically normalized and available in both raw Micheline and human-readable JSON formats, as well as transaction parameters.

The following API endpoints for accessing contract storages have been added:

NOTE

Only simple storage is indexed at the moment. Lazy storage (aka bigmaps) indexing is on the way.

# Get specific fields from the storage

Similar to how you can filter transactions by parameter, specifying the path to the field to filter by, you can use ?path= query parameter to specify the path to the specific field in the storage that you need to receive.

For example, imagine, the contract we are working with has the following storage:

{
   "vesting":{
      "vesting_schedule":{
         "next_payout":"2019-09-29T08:30:00Z",
         "payout_interval":"2629800"
      },
      "vesting_quantities":{
         "vested_balance":"162775594954",
         "vesting_increment":"199041301565"
      }
   },
   "key_info":{
      "key_groups":[
         {
            "signatories":[
               "edpkvS5QFv7KRGfa3b87gg9DBpxSm3NpSwnjhUjNBQrRUUR66F7C9g",
               "edpktm3zeGMzfzFuqgyYftt7uNyVRANTjrJCdU7bURwgGb9bRZwmJq",
               "edpkucCnbeGPWNvGHeTQ5hENHPrc8txBBiQXNphu3jgv9KYbhQBovd",
               "edpkuNjKKT48xBoT5asPrWdmuM1Yw8D93MwgFgVvtca8jb5pstzaCh"
            ],
            "group_threshold":"2"
         },
         {
            "signatories":[
               "edpkukogJzCZSAfc5foRpEZGsryMjiXj77VxcFLzZoreB1vZWFeKc2",
               "edpkuSZ1GoM6MALh4fPZBrwDhGwY9vENEoyctcXuDK3yoiX4xWhMaA",
               "edpkvXL9B32DcEbMNiMangcSFMvAd8NBwH8AfmRb6iBbHxLgx3J59P",
               "edpkumCFhgS94cjZXiFnnq7MFsaWBz4Tp78AX2fZfhB9J9hcgcKxgy"
            ],
            "group_threshold":"2"
         },
         ...
      ],
      "overall_threshold":"4"
   },
   "pour_info":{
      "pour_dest":"tz3bTdwZinP8U1JmSweNzVKhmwafqWmFWRfk",
      "pour_authorizer":"edpkteDwHwoNPB18tKToFKeSCykvr1ExnoMV5nawTJy9Y9nLTfQ541"
   },
   "replay_counter":"6505"
}

But we want to receive not the whole storage, but only the first signatory from the second key group. To do this we can use the ?path= query parameter to tell the API that we need only a specific part of the storage: {contract}/storage?path=key_info.key_groups.2.signatories.1

So the response will be (opens new window):

edpktjDaR4gmEZNSnnuSWMmJ4fhYmMD4Lk1WkDCfWRWdbGHQ9eNT45

This is very useful when the storage is rather large and we don’t want to waste resources on transferring and deserializing redundant data.

# Get storage at any time in the past

Also, you can easily access contract storage value at any time in the past.

If you want to get the contract storage at a specific level (at the end of the block), you can use the ?level= query parameter: {contract}/storage/?level=1354679 (opens new window)

If you want to get the contract storage after a specific operation, you need to request that operation and the resulting storage value will be included in the response: /transactions/{hash} (opens new window)

# Get storage history

In addition, we’ve added an API endpoint (opens new window) to get contract storage history, including information about the operation, caused the changes, and the resulting storage value: {contract}/storage/history (opens new window)

# Contract Code

Starting from v1.4 the TzKT indexer stores all contract scripts, including parameter type, storage type and contract code.

It’s worth noting that we have developed our own binary format that is used to store raw Micheline, which is x10 times more compact than Micheline JSON and 25% more compact and much faster than forging Micheline bytes.

So, contract scripts are stored in binary format, however, there is an API endpoint (opens new window) that allows you to get values, deserialized to the desired format:

# WebSocket API

Another great feature in TzKT v1.4 is the WebSocket API (opens new window), which enables you to subscribe to real-time Tezos data, so that you can receive the latest data as soon as it appears on the node and the TzKT indexer processes it. For interactive applications, that require real-time functionality, TzKT WebSocket API is a much better solution than spamming the REST API every second, checking for new data.

TzKT WebSocket API is designed to be very reliable, so if you properly handle all the messages from the API be sure you won’t miss any data nor receive duplicated data. All supported message types are explained in the docs (opens new window), so, please, be sure to read it first.

NOTE

If you are working on critical apps you may ask us for some code review regarding usage of our services to ensure you are doing it right and not making obvious mistakes.

We started with a very basic set of subscriptions:

In the near future we will add more events and subscriptions with more data, so if you have any feature requests, this is the best time to apply for it (just open an issue on GitHub (opens new window)).

# Simple JavaScript client

TzKT WebSocket API is based on SignalR (opens new window), which is actually great because there are a lot of already existing client libraries for many languages (C++ (opens new window), C# (opens new window), Java (opens new window), TypeScript/JavaScript (opens new window), Python (opens new window), etc.). In addition, you don't need to be a super-experienced developer or have a complete understanding of WebSocket internals. SignalR does all the complicated stuff for you and is therefore really easy to use.

Here is how a simple JavaScript TzKT WebSocket client looks:

const connection = new signalR.HubConnectionBuilder()
    .withUrl("https://api.tzkt.io/v1/events")
    .build();

async function init() {
    // open connection
    await connection.start();
    // subscribe to head
    await connection.invoke("SubscribeToHead");
    // subscribe to account transactions
    await connection.invoke("SubscribeToOperations", {
        address: 'KT19kgnqC5VWoxktLRdRUERbyUPku9YioE8W',
        types: 'transaction'
    });
};

// auto-reconnect
connection.onclose(init);

connection.on("head", (msg) => {
    console.log(msg);
});

connection.on("operations", (msg) => {
    console.log(msg);
});

init();

# Other improvements

In addition to described features, we made several internal optimizations, increasing indexer and API reliability. Also, we’ve upgraded core dependencies, because it would be a sacrilege to not use state-of-the-art technologies in the most advanced blockchain in the world 😄

# .NET 5.0

We’ve migrated from the .NET Core 3.1 to .NET 5.0, the latest version of the free and open-source development platform with C# 9 and a lot of performance and security optimizations.

# PostgreSQL 13

Also, we’ve migrated from PostgreSQL 12 to PostgreSQL 13, the latest version of the world’s most advanced open-source database with significant improvements to its indexing and lookup system, and more.

As a result, we achieved faster response times and 25% reducing database size, so now it consumes only 16GB disk space for Tezos mainnet. A full database snapshot - only 2.4GB.

# Conclusion

v1.4 brings such important features as improved smart contract data and WebSocket API that makes TzKT even more advanced Tezos blockchain indexer and Tezos API. And there will be even more cool features in the future, so stay tuned.

We would like to remind you that TzKT is not a commercial product, but a community-driven project that is truly open-source and free for everyone and for any use.

Many thanks to all our friends and the entire Tezos community for helping with improving and adopting TzKT. You are awesome! ❤️

For the Tezos! 🍺