The tutorial presented here is an extended version of what Nomadic labs showed in their post. The main difference is that I tried to stay as decentralized as possible and use only public Tezos node API and PyTezos.
# Introduction
It’s a hot time for Tezos during these days, the first update of the protocol is in full swing, the voting mechanisms are being debugged. We are currently at the first phase, when protocol change proposals can be submitted on-chain, and bakers can vote for them. I recommend reading “Amending Tezos” to learn how it works in details.
“Athens A” and “Athens B”
Tezos protocol update is a tough challenge both for developers and bakers, especially given the Tezos technology stack. The former should explain the proposed changes as clearly as possible, and the latter should conduct their own research and make an informed decision.
Speaking of current proposals, Nomadic Labs did a really hard work commenting every commit they made which undoubtedly deserves a respect. I also spoke with several bakers and I was pleased to learn that they take voting very seriously, do all the checks and review all the code changes.
The purpose of this guide is to dive deeper into the technical implementation of the amendment mechanism and also to show how to use the PyTezos library in research. We will try to be as paranoid as possible and avoid using third-party information sources and tools.
Let’s get started!
# Notebook
# Run locally
You will need Python 3.6+ and Jupyter package to continue.
$ git clone https://github.com/baking-bad/pytezos.git
$ cd pytezos
$ jupyter notebook
Navigate to the examples folder and run the dyor_voting.ipynb
# Play on Google Colab
You can also play around with the code online: https://colab.research.google.com/drive/1pnAf8xnU9GTSUGRPVy2V9ABRAtZitkpt
# Current voting phase
Say, you don't read any news and all you have is public node access and documentation.
How to determine what is the current voting phase?
from pytezos.rpc import mainnet
mainnet.head.votes.current_period_kind()
result
'proposal'
Great, we are at the first stage, now we want to know when did it start and when it will ends.
We can get this information from the block metadata:
level_info = mainnet.head.metadata.get('level')
level_info
result
{'level': 349439, 'level_position': 349438, 'cycle': 85, 'cycle_position': 1278, 'voting_period': 10, 'voting_period_position': 21758, 'expected_commitment': False}
Get precise boundaries in levels and rough estimation in days
import pendulum
start_level = level_info['level'] - level_info['voting_period_position']
start_level
result
327681
end_level = start_level + 8 * 4096 - 1 # eight cycles of 4096 blocks
end_level
result
360448
start_dt = pendulum.parse(mainnet.blocks[start_level].header.get('timestamp'))
time_past = (pendulum.now() - start_dt)
print(round(time_past.total_days(), 1), 'days passed')
stdout
15.7 days passed
time_left = (end_level - level_info['level']) / (level_info['level'] - start_level) * time_past
print(round(time_left.total_days(), 1), 'days left')
stdout
7.9 days left
# Current proposals
proposals = mainnet.head.votes.proposals()
proposals
result
[['Psd1ynUBhMZAeajwcZJAeq5NrxorM6UCU4GJqxZ7Bx2e9vUWB6z', 5176], ['Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd', 13255]]
Let's examine one of proposals
proposal_id = proposals[1][0]
proposal_id
result
'Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd'
# Injection operation
It's interesting to find the author of the proposal.
In order to do that we have to search for the first proposals operation for this particular proposal_id.
from pytezos.rpc.search import SearchChain
sc = SearchChain.from_chain(mainnet.main)
Thanks to the statefullness of the Tezos blockchain we can perform a binary search inside the voting period.
The algorithm searches for the first level where number of votes changed from 0 to non-zero.
operation = sc.find_proposal_inject_operation(proposal_id)
stderr
2019-03-12 19:40:38.314 | DEBUG | pytezos.rpc.search:bisect:29 - 714 at level 338560 2019-03-12 19:40:38.905 | DEBUG | pytezos.rpc.search:bisect:29 - 23 at level 333120 2019-03-12 19:40:39.499 | DEBUG | pytezos.rpc.search:bisect:29 - 0 at level 330400 2019-03-12 19:40:40.051 | DEBUG | pytezos.rpc.search:bisect:29 - 0 at level 331760 2019-03-12 19:40:40.602 | DEBUG | pytezos.rpc.search:bisect:29 - 0 at level 332440 2019-03-12 19:40:41.198 | DEBUG | pytezos.rpc.search:bisect:29 - 11 at level 332780 2019-03-12 19:40:41.748 | DEBUG | pytezos.rpc.search:bisect:29 - 0 at level 332610 2019-03-12 19:40:42.306 | DEBUG | pytezos.rpc.search:bisect:29 - 11 at level 332695 2019-03-12 19:40:43.092 | DEBUG | pytezos.rpc.search:bisect:29 - 11 at level 332652 2019-03-12 19:40:43.898 | DEBUG | pytezos.rpc.search:bisect:29 - 11 at level 332631 2019-03-12 19:40:44.770 | DEBUG | pytezos.rpc.search:bisect:29 - 0 at level 332620 2019-03-12 19:40:45.598 | DEBUG | pytezos.rpc.search:bisect:29 - 11 at level 332625 2019-03-12 19:40:46.410 | DEBUG | pytezos.rpc.search:bisect:29 - 0 at level 332622 2019-03-12 19:40:47.229 | DEBUG | pytezos.rpc.search:bisect:29 - 0 at level 332623 2019-03-12 19:40:47.999 | DEBUG | pytezos.rpc.search:bisect:29 - 11 at level 332624
operation()
result
{'protocol': 'PsddFKi32cMJ2qPjf43Qv5GDWLDPZb3T3bF6fLKiF5HtvHNU7aP', 'chain_id': 'NetXdQprcVkpaWU', 'hash': 'onydFJLWdGhfKNBfbnSLmqDu93j9NRimkbQm9WqLWYG8eyZUyTF', 'branch': 'BL53WJx6xPn6rnTnWZmpNaWGAQqU8HTwRTDqVDmthYsxUTBewo9', 'contents': [{'kind': 'proposals', 'source': 'tz1fNdh4YftsUasbB1BWBpqDmr4sFZaPNZVL', 'period': 10, 'proposals': ['Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd', 'Psd1ynUBhMZAeajwcZJAeq5NrxorM6UCU4GJqxZ7Bx2e9vUWB6z'], 'metadata': {}}], 'signature': 'sigvUqvh7rBS8yAoE5RMieQaD5hvg9NsLeJ4kTnXdK1tXXyrHL8mX7E3KCm9q9YgYbJn3edhcUiZjdU3xNhVPEUPkSGVNbi9'}
We can perform some checks on this operation, such as signature validation:
operation.verify_signature()
The only thing we can learn about the submitter's identity is his public key:
mainnet.get_public_key(operation.source())
result
'edpktyybTTrLm2rk6bn7xtgY9t2Kgt9GnqZxEcSrunm4vKWTF9ES9o'
timestamp = mainnet.blocks[operation.get('branch')].header.get('timestamp')
inject_dt = pendulum.parse(timestamp)
f'Proposed on the {(inject_dt - start_dt).days + 1}th day of the voting period'
result
'Proposed on the 4th day of the voting period'
# Source code
Unfortunately it's practically impossible to get proposal source from the blockchain.
We could get lucky and find the submitter's node (in case it's public) which has to know this proto (according to the docs).
The other option is trying to find a node in zeronet which has participated in the voting rehearsal.
We will be back to this issue later, now let's download proposal sources.
athens_a_url = 'https://blog.nomadic-labs.com/files/Athens_proposal_A.tar'
athens_b_url = 'https://blog.nomadic-labs.com/files/Athens_proposal_B.tar'
from pytezos.rpc.protocol import Protocol
Loading sources and convert them to the internal format (as in blockchain).
This function can also import a local tar file or a directory with extracted files.
athens_a = Protocol.from_uri(athens_a_url)
stderr
1218560it [03:37, 5613.44it/s]
athens_b = Protocol.from_uri(athens_b_url)
stderr
1218560it [03:04, 6594.80it/s]
Check that the sources we have downloaded are original.
In order to do that we have to obtain binary representation of the sources according to the http://tezos.gitlab.io/mainnet/api/rpc.html#get-protocols-protocol-hash (Binary output tab).
athens_a_id = athens_a.calculate_hash()
athens_a_id
result
'Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd'
athens_b_id = athens_b.calculate_hash()
athens_b_id
result
'Psd1ynUBhMZAeajwcZJAeq5NrxorM6UCU4GJqxZ7Bx2e9vUWB6z'
assert athens_a_id == proposals[1][0]
assert athens_b_id == proposals[0][0]
# Protocol update diff
First of all we need to get sources of the current protocol:
current_proto = mainnet.protocols[operation.get('protocol')]
current_proto
result
protocols/PsddFKi32cMJ2qPjf43Qv5GDWLDPZb3T3bF6fLKiF5HtvHNU7aP
current_hash = current_proto.calculate_hash()
current_hash
result
'PsddFKi32cMJ2qPjf43Qv5GDWLDPZb3T3bF6fLKiF5HtvHNU7aP'
assert current_hash == operation.get('protocol')
Now we can generate patch file in the standard diff format.
We can optional specify number of lines before and after the change: this is useful for review.
patch = current_proto.diff(athens_a, context_size=3)
Generate github-like side-by-side patch views, powered by diff2html.js
patch.export_html(output_path='athens_a_vs_003.html')
athens_diff = athens_a.diff(athens_b)
athens_diff.export_html(output_path='athens_a_vs_athens_b.html')
Compare "Athens A" vs "Athens B"
# Bonus: get all voting operations for a proposal
As an alternative to TzScan and other indexed-blockchain solutions.
for operation in sc.find_proposal_votes_operations(proposal_id):
print(operation())
break # this can take a while
stderr
2019-03-12 19:47:43.747 | DEBUG | pytezos.rpc.search:find_state_change_intervals:14 - 13255 at level 349385 2019-03-12 19:47:44.360 | DEBUG | pytezos.rpc.search:find_state_change_intervals:14 - 13255 at level 349325 2019-03-12 19:47:45.133 | DEBUG | pytezos.rpc.search:find_state_change_intervals:14 - 13183 at level 349265 2019-03-12 19:47:45.711 | DEBUG | pytezos.rpc.search:bisect:29 - 13183 at level 349295 2019-03-12 19:47:46.308 | DEBUG | pytezos.rpc.search:bisect:29 - 13255 at level 349310 2019-03-12 19:47:46.911 | DEBUG | pytezos.rpc.search:bisect:29 - 13255 at level 349302 2019-03-12 19:47:47.482 | DEBUG | pytezos.rpc.search:bisect:29 - 13183 at level 349298 2019-03-12 19:47:48.127 | DEBUG | pytezos.rpc.search:bisect:29 - 13183 at level 349300 2019-03-12 19:47:48.715 | DEBUG | pytezos.rpc.search:bisect:29 - 13255 at level 349301
stdout
{'protocol': 'PsddFKi32cMJ2qPjf43Qv5GDWLDPZb3T3bF6fLKiF5HtvHNU7aP', 'chain_id': 'NetXdQprcVkpaWU', 'hash': 'ooRoJ2yiEqchbV4xWEGk3wjwJtm5A5GcUnLFmYNLRzUgTT52ffU', 'branch': 'BLoUbPrEGth1WPJ4uFwEVTp5eWKqUU8Jj9EHqrsndmN9xE1QxiE', 'contents': [{'kind': 'proposals', 'source': 'tz1dwu9aYb7CRNq4Y2zAjipdjFuSVKhHS8vA', 'period': 10, 'proposals': ['Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd'], 'metadata': {}}], 'signature': 'sigboCbwcDzNJQXEPyCYfoL6fd5oBeTA2ogfhydhVEN9agdZg6PzktCZ4Fvj4WA5X8RUwz3t9jxS5gGftXJcF8R5tvNrbhNd'}
Search algorithm works as follows:
- Split block interval into equal chunks
- Determine which of the intervals contain state changes
- For each interval run binary search
- If there are several changes inside single interval, run binary search again
It's obvious that the search space can be easily splitted and processed independently, i.e parallelized.
# Improvement: proposal sources on-chain
As we pointed earlier there is no convenient way to get proposal source from the blockchain. This can be implemented via smart-contract. But it's more reasonable to store compressed code diff rather than full source.
ctxless_patch = current_proto.diff(athens_a, context_size=0)
ctxless_patch.export_tar('diff.tar.bz2')
os.path.getsize('diff.tar.bz2')
result
12839
# Applying protocol diff
proto = current_proto.apply(ctxless_patch)
assert proposal_id == proto.calculate_hash()
# Summary
Using almost only public Tezos node and PyTezos we were able to do the following things:
- Get basic information about the current voting period;
- Find the moment of proposal injection;
- Load proposal sources, encode them in json and binary formats;
- Calculate proposal id (hash);
- Load current protocol sources, make code diff and visualize it using github-like markup;
- Find all voting operations for a given proposal.
Resulting diff views: “Athens A” vs “003_PsddFKi3”, “Athens A” vs “Athens B”
# Further work
# Store Tezos proposal sources on-chain
As noted, there is no straightforward way of getting the protocol sources from the blockchain during the voting period. As an improvement and a proof-of-concept I’ll try to implement this feature using a smart contract in conjunction with revised big_map_get function in PyTezos.
# Operation search
It was shown how to search for all voting operations using a modified binary search algorithm. In the following series, I will show how to speed up a search using parallelization, as well as how to extend the algorithm to a more general case.