Meet TzKT v1.5, another version of the free open-source Tezos blockchain indexer and API with BigMaps support and other improvements, enabling more advanced use cases and making developers’ life even easier.

Let’s walk through the changes, but first of all, pay attention to the breaking changes.

Breaking bad changes

For API users:

• commitDate, commitHash, version, tags fields in the /software model have been deprecated and will be removed in the next major release. Use the metadata field instead;
• prim: bool values in human-readable JSON (contract storage, transaction parameters, etc.) are now converted to bool true | false instead of string "True" | "False";
• prim: big_map values in human-readable JSON are now converted to int 123 instead of string "123";
• to avoid collisions, the way to specify an index of an array item in JSON path has been changed. Before you were doing this:
  /transactions?parameter.arr.3=qwe or /storage?path=sigs.1.0.
  Now you should do this:
  /transactions?parameter.arr.[3]=qwe and /storage?path=sigs.[1].[0].

We’ve checked API logs for the last two months and didn’t find any real use-cases (queries) that could be affected by most of these changes, so we expect that actually nothing will break.

Anyway, we won’t deploy v1.5 in production right now, but in two weeks (after activation of the Florence protocol) to give API users plenty of time to carefully check the changes and update their apps if necessary. If you need more time, just let us know.

The most relevant API documentation is available on staging.api.tzkt.io. Feel free to reach us in our telegram chat or in Tezos dev slack with any questions.

For database users:

• CommitDate, CommitHash, Version, Tags columns have been dropped from the Software table. Use Metadata column instead;
• InternalOperations column in the Transactions table now contains a number of internal operations rather than a bitmask (flags).

Full-fledged BigMaps

We’ve implemented the most wanted feature - indexing of BigMaps. TzKT indexer processes and automatically converts Micheline JSON to human-readable JSON, so it’s never been so convenient to work with BigMaps in Tezos.

Let’s see how to use TzKT API to access BigMaps.

Access BigMaps

There are two ways to access BigMaps:

1. By ptr: /bigmaps/{ptr} (see docs).
2. By name: /contracts/{address}/bigmaps/{name} (see docs).

Access by ptr (pointer, or id) is slightly faster than by name, especially in nested queries like /bigmaps/543/keys, but the only disadvantage is that you have to know the ptr. Moreover, if the BigMap is inside a map or an option or so, the ptr, in theory, can change, so you will also have to track the contract storage to know the actual ptr.

On the other hand, access by name is not the fastest one, but much more user-friendly. All you need to know is the path to the BigMap in the contract storage (we talk about human-readable JSON format).

Let’s see what is a BigMap name and how to specify it, in a few examples.

Example 1

If the contract storage is (example):

{
    ...
    "ledger": 1234,
    ...
}

Then the BigMap path is ledger.
So, you can use the query: /contracts/{address}/bigmaps/ledger (example).

Example 2

Contract storage (example):

{
    ...
    "assets": {
        ...
        "ledger": 1234,
        ...
    },
    ...
}

BigMap path: assets.ledger.
Query: /contracts/{address}/bigmaps/ledger (example).

TIP

As you can see, you can use not the full path (e.g. assets.ledger), but only the last component (ledger) as a BigMap name (of course, if that name uniquely identifies the BigMap within the contract storage).

Example 3

Contract storage:

{
    ...
    "operators": {
        ...
        "metadata": 1234,
        ...
    },
    ...
    "assets": {
        ...
        "metadata": 5678,
        ...
    }
    ...
}

BigMap path: assets.metadata.
Query: /contracts/{address}/bigmaps/assets.metadata.

In this case, we can’t use just metadata as a BigMap name due to ambiguity between operators.metadata and assets.metadata, so we have to specify the path.

So, if you need to access a BigMap with a specific ptr and you need the lowest latency, it’s recommended to use access by ptr. In all other cases, you can use access by name.

BigMap keys

The most interesting part is accessing BigMap keys.

NOTE

All the API endpoints described below can be used with both BigMap access methods: by ptr and by name, so if we say /{bigmap}/keys, we mean both: /bigmaps/{ptr}/keys and /contracts/{address}/bigmaps/{name}/keys.

Get all BigMap keys

All keys of a particular BigMap can be accessed via /{bigmap}/keys API endpoint. For example, /bigmaps/511/keys.

Obviously, you almost never need ALL the keys, so you might want to use a variety of query filters to get a more precise result and not load unnecessary data in vain. Let’s briefly demonstrate what you can do with query filters in TzKT API:

How to get only removed keys?

/{bigmap}/keys?active=false (as you can see, removed keys contain the last non-null value, for more convenience)

How to get all owners of some NFT with some token_id at the Hic et nunc contract?

/{bigmap}/keys?key.nat={tokenId}

How to get current owners (with balance > 0) of that NFT?

/{bigmap}/keys?key.nat={tokenId}&value.gt=0

How to get all kUSD approvals for some address?

/{bigmap}/keys?value.approvals.{address}.null=false

How to get all kUSD holders who have a non-empty map of approvals?

/{bigmap}/keys?value.approvals.ne={}

How to get keys where some array contains some item?

/{bigmap}/keys?value.{someArray}.[*]={someItem}

TIP

You can use [*] to check if any item in the array matches the condition or [123] to check only an item at the specified index.

Also, if you request BigMap keys periodically (in a loop) to see if there were any changes, it’s strongly recommended to use the lastLevel query parameter to request only those keys that were updated since the specified block (since the last time you requested it). There is no need to request all the keys every time because it is a waste of resources.

WARNING

Remember, you should never set limit=10000 without good reason. In 99.9% of cases, if you think that setting such a large limit is a good idea and that it will solve all your problems, you’re wrong. It’s an awful idea. We allow such a large limit only for very specific use cases, so please consider using pagination and filtering by lastLevel.

Get specific BigMap key

You can get a specific key via /{bigmap}/keys/{key} API endpoint by passing either a key hash (e.g. exprueiUsweD5WNWpQ9YiBSaMwU5gDwKJuAQzddhQHvKmwZBgUAJzJ, example)
or a plain JSON value (e.g. tz1LHh1BAEf7tJDZnqqF8pSS9Uy4dLhu3bGJ, example).

In case of a complex key you can pass it as is (in a human-readable JSON format): /keys/{"nat":"152","address":"tz1UBZUkXpKGhYsP5KtzDNqLLchwF4uHrGjw"}.

Even if a key is an empty string "" you can still pass it as is: /{bigmap}/keys/""

Get BigMap keys at specific block

In addition, TzKT API allows you to get BigMap keys at any block in the past via /{bigmap}/historical_keys/{level} API endpoint. This could be helpful when you need historical values (for example, when you want to pay staking rewards and you need to know holder balances at the snapshot block to distribute rewards correctly).

However, this API endpoint is quite heavy, because it aggregates all BigMap updates, and therefore much slower than accessing current BigMap keys, so we recommend using it only when it’s really necessary and there is no way to modify your algorithm to rely on current values.

TIP

If you need to get BigMap keys at a specific block just to see how they were changed after a particular operation, it’s better (and more correct) to rather get that particular operation with included BigMap updates (diffs).

BigMap updates

You can get updates for a particular BigMap key via /{bigmap}/keys/{key}/updates API endpoint (example) to see how its value changed in time.

Also, you can get all BigMap updates via /bigmaps/updates API endpoint with various query parameters available to get a more precise result. For example:

Getting all updates of all BigMaps:
/bigmaps/updates

Getting all updates of a specific BigMap:
/bigmaps/updates?bigmap={ptr}

Getting all updates of all BigMaps of a specific contract:
/bigmaps/updates?contract={address}

Subscribe to BigMap updates

Another cool feature in TzKT v1.5 is the ability to receive BigMap updates in real-time via WebSocket, which is especially helpful for sub-indexers (in particular, we use in the DipDup, a framework for building selective Tezos indexers inspired by The Graph).

With TzKT WebSocket API you can subscribe for updates of multiple BigMaps, using flexible subscription parameters. Here is an example of a simple JS client, that subscribes for BigMap updates:

const signalR = require("@microsoft/signalr");
const connection = new signalR.HubConnectionBuilder()
    .withUrl("https://staging.api.tzkt.io/v1/events")
    .build();

connection.on("bigmaps", (msg) => { console.log(msg); });
// subscribe to all bigmaps of the 'KT123...' contract
await connection.invoke("SubscribeToBigMaps", { contract: 'KT123...' });
// and also subscribe to bigmap with ptr 123
await connection.invoke("SubscribeToBigMaps", { ptr: 123 });

await connection.start();

Check out the documentation on how to use the TzKT WebSocket API.

Storage and BigMap diffs after operation

It’s quite a common case when we need to know how a particular operation has affected the contract storage and in particular BigMaps. For that purpose, TzKT API allows to explicitly include contract storage and BigMap diffs into transaction and origination operations (other operations don't affect contract storage), so that you can see how it changed after executing the operation.

Note, that by default, when you request a list of operations, both storage and diffs fields are omitted for optimization reasons, so if you need them you have to explicitly specify it in the select query parameter, like this:
/transactions?target=KT1...&select=sender,parameter,storage,diffs.

However, if you receive operations via WebSocket API, storage and diffs are automatically included.

NOTE

Prefer this method to get historical values rather than "storage at block" or "bigmap keys at block" endpoints, because with the last ones, if there are multiple operations in the block that affected the same storage and the same BigMap keys, you will miss intermediate values and will get only the final ones (at the end of the block, not the operation).

Florence protocol

And finally, we have added Florence protocol support, which is expected to go live approximately 10 May.

For those who want to continue using the current version of TzKT, we recently released v1.4.1 with no breaking changes, but with Florence support. In this case, you don’t need to reindex data, just update the binaries.

So, if you run your own instance of TzKT, make sure to update it to either v1.4.1 or v1.5 in advance.

Other improvements

In addition to minor fixes and internal optimizations, we have significantly improved performance of filtering transactions by parameter, in case of .eq and .in modes.

In the current version a request like /transactions?parameter.to=KT1... (it's a-la getting all token transfers to a particular account) takes about 5 sec, but in the v1.5 it takes about 150 ms. More than 30 times faster 😎.

Conclusion

We have demonstrated the main features and improvements in v1.5, however, we didn’t cover all of them, so feel free to check the documentation to see all available API endpoints and query parameters.

TzKT v1.5 will go live in prod after activation of the Florence protocol (~10 May). Until then, feel free to use staging.api.tzkt.io and, please, pay attention to the breaking changes and make sure to update your apps if necessary.

As always, we want to remind you that TzKT is not a commercial product, but a community-driven project that is truly open-source and free for everyone and for any use.

Many thanks to all our friends and the entire Tezos community for helping with improving and adopting TzKT. You are awesome! ❤️

For the Tezos! 🍺