Hey, Tezos friends, long time no see! As you probably know we have been working on advanced support of Tezos tokens in TzKT for a long time, overcoming several protocol upgrades, and finally we are happy to announce TzKT v1.7, the newest version of the most popular Tezos blockchain indexer and API, perfected with generic token indexing.

However, while the main functionality is done and production-ready, we won’t deploy it to production right now, but a little later, because there are some insignificant breaking changes. We want to give people sufficient time to prepare for the update.

In this article we will review the main features of the new API, which is available for testing in the staging environment: staging.api.tzkt.io. The source code of the new version can be found on GitHub.

First of all, let’s pay attention to the breaking changes.

Breaking changes

All previously deprecated fields, query parameters, etc. have been removed. This is not really breaking changes, because most of those things were deprecated more than a year ago and all API users had plenty of time to adjust their code, but still:

• parameters, from and to query parameters have been removed from the /accounts/{address}/operations endpoint. Use parameter, timestamp.ge and timestamp.lt instead;
• parameters query parameter has been removed from the /operations/transactions endpoint. Use parameter instead;
• parameters field has been removed from the Transaction model. Use entrypoint and parameter instead;
• id, startLevel, and endLevel fields have been removed from the VotingPeriod model. Use index, firstLevel and lastLevel instead;
• commitDate, commitHash, version, and tags fields have been removed from the Software model. Use metadata.* instead;
• period field has been removed from the Proposal model. Use firstPeriod instead;
• type field in the Account model now may have another possible value: "ghost", in addition to "empty", "user", "delegate" and "contract";
• token_metadata bigmap tag is now less strict and doesn’t require the bigmap to be fully annotated, despite it’s required by TZIP-12.

Also, we have rearranged prefixes for environment variables used for configuring indexer and API services. In short, the TZKT_ prefix is no longer valid.

Before v1.7 you could pass env vars with no prefix (e.g. ConnectionStrings__DefaultConnection=...) to configure both Tzkt.Sync and Tzkt.Api services at the same time, and with prefix TZKT_ (e.g. TZKT_ConnectionStrings__DefaultConnection=...) to configure Tzkt.Sync service only.

Now you can pass env vars with no prefix to configure both Tzkt.Sync and Tzkt.Api services as before, with prefix TZKT_SYNC_ to configure Tzkt.Sync service and with prefix TZKT_API_ to configure Tzkt.Api service.

We are working on comprehensive documentation for TzKT where we will explain how that works and what cases that’s needed for.

New token API

In scope of generic token indexing, three entities have been added to the TzKT indexer: tokens, token balances and token transfers. All these entities (except for token metadata) are indexed synchronously with other data, that means that the data received from the API is always consistent and up to date with the indexer's head.

The following API endpoints for working with tokens have been added:

• /v1/tokens - returns a list of FA1.2 and FA2 tokens issued on Tezos (docs);
• /v1/tokens/count - returns a total number of all or specific tokens (docs);
• /v1/tokens/balances - returns a list of token balances (docs);
• /v1/tokens/balances/count - returns a total number of all or specific token balances (docs);
• /v1/tokens/transfers - returns a list of token transfers (docs);
• /v1/tokens/transfers/count - returns a total number of all or specific token transfers (docs);
• WebSocket subscription to receive real-time token transfers has been added as well (docs).

There are a lot of query filters, allowing you to request data more flexibly and precisely. All of them can be found in the documentation.

NOTE

In the documentation you can click on query parameters to expand the details and see available modes, like .in, .gt, etc..

Also, we have introduced new cool features: deep selecting and deep sorting, in addition to deep filtering, that you were already using with transaction parameters and bigmap keys, like /transactions?parameter.[*].txs.[*]_from=....

Deep selecting

As you know, most API endpoints have the select query parameter, which allows you to select and include into response only those fields that you really need. However, this parameter worked with root fields only, and it wasn’t possible to select nested fields. Now it’s possible!

Let’s say you are requesting token transfers, and the only information you need is a sender, recipient, amount and basic token info, such as token symbol and decimals. However, if you do just /v1/tokens/transfers (example) you receive by default a list of quite large objects:

{
  "id": 116576829,
  "level": 1840204,
  "timestamp": "2021-11-06T08:45:18Z",
  "token": {
    "id": 778919,
    "contract": {
      "alias": "hic et nunc NFTs",
      "address": "KT1RJ6PbjHpwc3M5rw5s2Nbmefwbuwbdxton"
    },
    "tokenId": "512275",
    "standard": "fa2",
    "metadata": {
      "name": "astromoon",
      "tags": [
        "nfts",
        "nft",
        ...
        "bhfyp",
        "digitalillustration"
      ],
      "symbol": "OBJKT",
      "formats": [
        {
          "uri": "ipfs://QmZarCfCdyf7A9bPKiEtZcJqrqHgiprgFv5z4cG5tPDxeQ",
          "mimeType": "image/jpeg"
        }
      ],
      "creators": [
        "tz1eVjytct5fWpp646fKTK2ULYtVPVYpFmQF"
      ],
      "decimals": 0,
      "displayUri": "ipfs://QmbUp9EdVrTz3vRnwffd8dx2gRKdAENSkE85YPQzwCNrnH",
      "artifactUri": "ipfs://QmZarCfCdyf7A9bPKiEtZcJqrqHgiprgFv5z4cG5tPDxeQ",
      "description": "",
      "thumbnailUri": "ipfs://QmNrhZHUaEqxhyLfqoq1mtHSipkWHeT31LNHb1QEbDHgnc",
      "isBooleanAmount": false,
      "shouldPreferSymbol": false
    }
  },
  "from": {
    "address": "tz1eVjytct5fWpp646fKTK2ULYtVPVYpFmQF"
  },
  "to": {
    "alias": "hic et nunc Marketplace",
    "address": "KT1HbQepzV1nVGg8QVznG7z4RcHseD5kwqBn"
  },
  "amount": "5",
  "transactionId": 116576775
}

As you can see, more than 90% of received data is useless in our case, so the deep selection is our best friend.

Let’s specify what fields we really need: select=from.address,to.address,amount,token.metadata.symbol,token.metadata.decimals (example), and now we receive the following objects:

{
  "from.address": "tz1eVjytct5fWpp646fKTK2ULYtVPVYpFmQF",
  "to.address": "KT1HbQepzV1nVGg8QVznG7z4RcHseD5kwqBn",
  "amount": "5",
  "token.metadata.symbol": "OBJKT",
  "token.metadata.decimals": 0
}

Now it looks much better and moreover significantly reduces network traffic. However, it can be even better! TzKT API allows you to rename the fields you choose to something more readable than "token.metadata.decimals".

Let’s explicitly specify field names: select=from.address as src,to.address as dst,amount,token.metadata.symbol as symbol,token.metadata.decimals as decimals (example) and now we receive from the API:

{
  "src": "tz1eVjytct5fWpp646fKTK2ULYtVPVYpFmQF",
  "dst": "KT1HbQepzV1nVGg8QVznG7z4RcHseD5kwqBn",
  "amount": "5",
  "symbol": "OBJKT",
  "decimals": 0
}

Looks much better, doesn't it?

Deep sorting

Just like the deep selecting, deep sorting allows you to sort results (ascending or descending) by nested fields (in particular, by token metadata), not just by root ones. This is quite useful, because it allows to build more advanced apps/dapps that use token metadata.

Imagine you are requesting NFTs that have a rarity field in metadata and you want to get the most rare tokens first. In this case you can sort them by this nested field sort.desc=metadata.rarity (not the best example, because rarity here is a string, but still).

You can see what fields can be used for sorting in the response description.

Deep filtering

Deep filtering isn’t a new feature, however we will still mention it, in case someone didn’t know. Deep filtering allows you to filter results by nested fields (in particular, by token metadata). This is extremely useful, because you can request data very precisely, instead of loading everything (hundreds of thousands of tokens) and filtering on the client side.

Imagine, you are requesting NFTs and want to get only those about dogs. This is how you can do that with deep filtering: /v1/tokens?metadata.tags.[*]=dog (example).

Now, when we discussed the main features of the new API, let’s take a look at real-life use cases, answering the most often questions from developers in Tezos communities.

Examples of the /v1/tokens endpoint usage

How to get all NFTs created within a specific NFT contract:
/v1/tokens?contract={address} (example);

How to get all NFTs of a specific creator:
/v1/tokens?metadata.creators.[*]={address} (example);

How to get FA1.2 tokens with the largest number of holders:
/v1/tokens?standard=fa1.2&sort.desc=holdersCount (example);

How to get recently created tokens:
/v1/tokens?sort.desc=id (example);

How to get recently active (either created or transferred) tokens:
/v1/tokens?sort.desc=lastLevel (example);

How to get the number of FA2 tokens created in Dec 2021:
/v1/tokens/count?standard=fa2&firstTime.ge=2021-12-01&firstTime.lt=2022-01-01 (example).

NOTE

The token.id is not the same as the token.tokenId. token.id is an internal ID within the indexer database and mostly used for much more performant filtering and more accurate pagination.

NOTE

Token metadata is indexed in a separate thread, that means it may appear with some delays.

Examples of the /v1/tokens/balances endpoint usage

How to get all account’s tokens:
/v1/tokens/balances?account={address}&balance.ne=0 (example);

How to get all account’s NFTs (OBJKTs):
/v1/tokens/balances?account={address}&token.metadata.symbol=OBJKT (example);

How to get recently updated account token balances:
/v1/tokens/balances?account={address}&sort.desc=lastLevel (example);

How to get top holders of a particular token:
/v1/tokens/balances?token.contract={address}&token.tokenId={tokenId}&sort.desc=balance (example), or the much faster alternative, in case you know the internal TzKT ID of the token (you can preload it and cache in your app for subsequent use): /v1/tokens/balances?token.id={id}&sort.desc=balance (example).

NOTE

All amounts in tokens, token balances and token transfers are not normalized. In order to get normalized values, you need to divide them by metadata.decimals on your side.

Examples of the /v1/tokens/transfers endpoint usage

How to get all account’s token transfers:
/v1/tokens/transfers?anyof.from.to={address} (example);

How to get account’s transfers of a specific token:
/v1/tokens/transfers?anyof.from.to={address}&token.contract={address}&token.tokenId={tokenId} (example), or the much faster alternative, in case you know the internal TzKT ID of the token (you can preload it and cache in your app for subsequent use): /v1/tokens/transfers?anyof.from.to={address}&token.id={id} (example);

How to get whale transfers (with an amount greater than some value) of a particular token:
/v1/tokens/transfers?token.id={id}&amount.gt={amount} (example);

How to get all "mints" of a particular token:
/v1/tokens/transfers?token.id={id}&from.null=true (example);

How to get all burns of a particular token:
/v1/tokens/transfers?token.id={id}&to.null=true (example);

How to get all token transfers produced by a particular transaction:
/v1/tokens/transfers?transactionId={id} (example).

NOTE

If from is null, it is a token mint, if to is null, it is a token burn, otherwise it is a token transfer.

NOTE

Token transfers contain an ID of the operation caused those transfers (it can be either transactionId, originationId, or migrationId). TzKT doesn’t join the source operation for performance reasons, but if you still need them, you can use the IDs to fetch the operations in a separate request, e.g. /v1/operations/transactions?id.in=1,2,3,4.

NOTE

You can also use operation id to see the operation on the TzKT explorer, for example tzkt.io/transactions/153129443.

Token indexing limitations

At the moment of writing almost all tokens in Tezos mainnet are well-supported by TzKT, however, the chance that someone creates a non-standard FA contract, that won’t be supported out of the box, always exists.

Token indexing has a different nature and completely differs from the indexing of blocks or transactions, etc.. Both blocks and transactions are strongly typed and, what is the most important, strictly follow well-known protocol rules. On the other hand, the only rules a particular token follows are rules designed by the developer of the smart contract. Unfortunately, it’s just impossible to make the indexer automatically understand all the rules smart contract developers can ever create, and therefore the indexer can’t automatically handle all possible tokens, but only standard-ish ones.

Token indexing in TzKT is done by tracking updates of a ledger bigmap (actually, the bigmap doesn’t have to be named "ledger", TzKT will anyway try to recognize it, we believe it’s smart enough). Therefore, there are two main limitations:

1. Only predefined ledger bigmap types are supported. If you create an FA contract with a ledger bigmap of a specific type, the indexer won’t be able to properly parse it and process token balances, so the contract will be simply ignored. Always (when possible) follow the common standards (see TZIP-12) to make sure your tokens will be indexed properly and automatically supported by Tezos wallets, explorers, etc.. No need to reinvent the wheel.

2. TzKT guarantees correctness of token balances only for tokens with "static" balances that change only by user action, producing ledger bigmap update. In case of "dynamic" balances (e.g. when token balance changes in time without on-chain events) the indexer will show the last known balance. An example of such a token is Tezos Domains, where token balance should become zero after a domain is expired. Since expiration is not an on-chain event, the indexer is not triggered to adjust the balance, so keep that in mind when checking token balances for expired domains. For active domains everything’s good though.

Actually, we do not limit you in your choice of how to design your smart contract. If you really want to create something very specific, do it! There is always an option to create your own selective indexer that will index your contract as you need. Check out DipDup, a selective indexing framework which is perfectly suited for such cases.

Token metadata

In addition to tokens, token balances, and token transfers, which are "on-chain" things, TzKT also indexes off-chain token metadata. However, TzKT doesn’t query the IPFS directly but uses an external metadata indexer, created with the DipDup framework. In case you use a self-hosted TzKT indexer and you want token metadata to be indexed as well, you will have to either spin up your own instance of the DipDup metadata indexer (it’s open-source) or use a public one.

Token metadata is indexed asynchronously, therefore it may appear in API responses with some delays. Or, it may not appear at all if a contract doesn’t follow standards regarding token metadata, because the metadata indexer is quite standard-sensitive.

Also, we want to warn you, that token metadata is completely an off-chain thing, and filled by token creators. It’s only up to them what fields and in what format to specify. While there are pretty clear standards (TZIP-12, TZIP-21), that describe the common fields and their format, people often ignore them, unfortunately. For example, it is clearly stated that "decimals should be an integer, converted to a UTF-8 string", however, some tokens have "decimals": 0, some tokens have "decimals": "0", and some have "decimals": "\u00050x000x00" (it’s packed bytes). So, if you rely on token metadata, be ready for badly formatted or untrusted or even false values.

Conclusion

Implementing token indexing was a non-trivial, but interesting challenge for us. Having an ability to access accurate token data is vital for the growing Tezos ecosystem, and we tried to fit as many needs as possible. We hope you will find it useful for your projects.

Right now we are putting the full force to continue implementing Ithaca protocol support, and will try to be ready in time. In the meantime, feel free to share any feedback regarding token indexing and TzKT in general.

As always, we are incredibly grateful to all contributors, to everyone who supports us, who gives us helpful feedback and bugreports, and who just spreads the word about TzKT, a unique Tezos blockchain indexer and API. We also highly appreciate the huge support from the Tezos Foundation.

The exponential growth of the Tezos ecosystem – the merit of each tezonian!

Cheers! 🍺