Refactors testnetify and remove last localOsmosis Dockerfile (#2795)

* Refactor testnetify and remove Dockerfile

* Manually replace new validator public key

* Improve output formatting

* Bug fixes

* Add ACCOUNT_PUBKEY and ACCOUNT_ADDRESS input

* Add environment variables as input and config folder check

* Update docs, Makefile and rename folder

* Modify CLI arguments

* Update docker-compose.yaml defaults and print updated values

* Invert printing logic replacing verbose with quiet

Makefile

@@ -363,26 +363,32 @@ localnet-keys:
 localnet-build:
 	@DOCKER_BUILDKIT=1 docker-compose -f tests/localosmosis/docker-compose.yml build
 
-localnet-build-state-export:
-	@docker build -t local:osmosis-se --build-arg ID=$(ID) -f tests/localosmosis/mainnet_state/Dockerfile-stateExport .
-
 localnet-start:
 	@docker-compose -f tests/localosmosis/docker-compose.yml up
 
 localnet-startd:
 	@docker-compose -f tests/localosmosis/docker-compose.yml up -d
 
-localnet-start-state-export:
-	@docker-compose -f tests/localosmosis/mainnet_state/docker-compose-state-export.yml up
-
 localnet-stop:
 	@docker-compose -f tests/localosmosis/docker-compose.yml down
 
 localnet-remove: localnet-stop
 	rm -rf $(PWD)/tests/localosmosis/.osmosisd
 
+localnet-build-state-export:
+	@DOCKER_BUILDKIT=1 docker-compose -f tests/localosmosis/state_export/docker-compose.yml build
+
+localnet-start-state-export:
+	@docker-compose -f tests/localosmosis/state_export/docker-compose.yml up
+
+localnet-startd-state-export:
+	@docker-compose -f tests/localosmosis/state_export/docker-compose.yml up -d
+
+localnet-stop-state-export:
+	@docker-compose -f tests/localosmosis/docker-compose.yml down
+
 localnet-remove-state-export:
-	@docker-compose -f tests/localosmosis/mainnet_state/docker-compose-state-export.yml down
+	rm -rf $(PWD)/tests/localosmosis/state_export/.osmosisd
 
 .PHONY: all build-linux install format lint \
 	go-mod-cache draw-deps clean build build-contract-tests-hooks \

tests/localosmosis/README.md

@@ -27,9 +27,12 @@ You can now quickly test your changes to Osmosis with just a few commands:
 
 ## LocalOsmosis with Mainnet State
 
-Running LocalOsmosis with mainnet state is resource intensive and can take a bit of time. It is recommended to only use this method if you are testing a new feature that must be thoroughly tested before pushing to production.
+Running LocalOsmosis with mainnet state is resource intensive and can take a bit of time. 
+It is recommended to only use this method if you are testing a new feature that must be thoroughly tested before pushing to production.
 
-A few things to note before getting started. The below method will only work if you are using the same version as mainnet. In other words, if mainnet is on v8.0.0 and you try to do this on a v9.0.0 tag or on main, you will run into an error when initializing the genesis. (yes, it is possible to create a state exported testnet on a upcoming release, but that is out of the scope of this tutorial)
+A few things to note before getting started. The below method will only work if you are using the same version as mainnet. In other words, 
+if mainnet is on v8.0.0 and you try to do this on a v9.0.0 tag or on main, you will run into an error when initializing the genesis. 
+(yes, it is possible to create a state exported testnet on a upcoming release, but that is out of the scope of this tutorial)
 
 Additionally, this process requires 64GB of RAM. If you do not have 64GB of RAM, you will get an OOM error.
 
@@ -56,57 +59,61 @@ systemctl stop osmosisd.service
 
 ```sh
 cd $HOME
-osmosisd export 2> testnet_genesis.json
+osmosisd export 2> state_export.json
 ```
 
-After a while (~15 minutes), this will create a file called `testnet_genesis.json` which is a snapshot of the current mainnet state.
+After a while (~15 minutes), this will create a file called `state_export.json` which is a snapshot of the current mainnet state.
 
-5. Copy the `testnet_genesis.json` to the localosmosis folder within the osmosis repo
+5. Copy the `state_export.json` to the `localosmosis/state_export` folder within the osmosis repo
 
 ```sh
-cp -r $HOME/testnet_genesis.json $HOME/osmosis/tests/localosmosis
+cp $HOME/state_export.json $HOME/osmosis/tests/localosmosis/state_export/
 ```
 
-6. Ensure you have docker and docker compose installed/running:
-Docker
+6. Ensure you have docker and docker-compose installed:
+
 
 ```sh
+# Docker
 sudo apt-get remove docker docker-engine docker.io
 sudo apt-get update
 sudo apt install docker.io -y
-```
 
-Docker Compose
-
-```sh
+# Docker compose
 sudo apt install docker-compose -y
 ```
 
-7. Compile the local:osmosis-se docker image (~15 minutes, since this process modifies the testnet genesis you provided above). You may change the exported ID to whatever you want the chain-id to be. In this example, we will use the chain-id of localosmosis.
+7. Build the `local:osmosis` docker image:
 
-```sh
-cd $HOME/osmosis
-export ID=local
+```bash
 make localnet-build-state-export
 ```
 
-8. Start the local:osmosis-se docker image
+8. Run the `local:osmosis` docker image
 
 ```sh
 make localnet-start-state-export
 ```
 
-You will then go through the genesis intialization process. This will take ~15 minutes. You will then hit the first block (not block 1, but the block number after your snapshot was taken), and then you will just see a bunch of p2p error logs with some KV store logs. **This will happen for about 1 hour**, and then you will finally hit blocks at a normal pace.
+When running this command for the first time, `local:osmosis` will:
+
+- Modify the provided `state_export.json` to create a new state suitable for a testnet
+- Start the chain
+
+You will then go through the genesis initialization process. This will take ~15 minutes. 
+You will then hit the first block (not block 1, but the block number after your snapshot was taken), and then you will just see a bunch of p2p error logs with some KV store logs. 
+**This will happen for about 1 hour**, and then you will finally hit blocks at a normal pace.
 
 9. On your host machine, add this specific wallet which holds a large amount of osmo funds
 
 ```sh
-echo "bottom loan skill merry east cradle onion journey palm apology verb edit desert impose absurd oil bubble sweet glove shallow size build burst effort" | osmosisd keys add wallet --recover --keyring-backend test
+MNEMONIC="bottom loan skill merry east cradle onion journey palm apology verb edit desert impose absurd oil bubble sweet glove shallow size build burst effort"
+echo $MNEMONIC | osmosisd keys add wallet --recover --keyring-backend test
 ```
 
-You now are running a validator with a majority of the voting power with the same mainnet state as when you took the snapshot.
+You now are running a validator with a majority of the voting power with the same state as mainnet state (at the time you took the snapshot)
 
-10. On your host machine, you can now query the state export testnet like so:
+10. On your host machine, you can now query the state-exported testnet:
 
 ```sh
 osmosisd status
@@ -124,7 +131,9 @@ osmosisd tx bank send wallet osmo1nyphwl8p5yx6fxzevjwqunsfqpcxukmtk8t60m 1000000
 make localnet-remove-state-export
 ```
 
-Note: At some point, all the validators (except yours) will get jailed at the same block due to them being offline. When this happens, it make take a little bit of time to process. Once all validators are jailed, you will continue to hit blocks as you did before. If you are only running the validator for a short period of time (< 24 hours) you will not experience this.
+Note: At some point, all the validators (except yours) will get jailed at the same block due to them being offline.
+When this happens, it may take a little bit of time to process. Once all validators are jailed, you will continue to hit blocks as you did before. 
+If you are only running the validator for a short period of time (< 24 hours) you will not experience this.
 
 ## Accounts
 

tests/localosmosis/mainnet_state/Dockerfile-stateExport

-# syntax=docker/dockerfile:1
-
-
-# --------------------------------------------------------
-# Build
-# --------------------------------------------------------
-
-FROM golang:1.18.2-alpine3.15 as build
-
-RUN set -eux; apk add --no-cache ca-certificates build-base;
-RUN apk add git
-# Needed by github.com/zondax/hid
-RUN apk add linux-headers
-
-WORKDIR /osmosis
-COPY . /osmosis
-
-
-# CosmWasm: see https://github.com/CosmWasm/wasmvm/releases
-ADD https://github.com/CosmWasm/wasmvm/releases/download/v1.0.0/libwasmvm_muslc.aarch64.a /lib/libwasmvm_muslc.aarch64.a
-ADD https://github.com/CosmWasm/wasmvm/releases/download/v1.0.0/libwasmvm_muslc.x86_64.a /lib/libwasmvm_muslc.x86_64.a
-RUN sha256sum /lib/libwasmvm_muslc.aarch64.a | grep 7d2239e9f25e96d0d4daba982ce92367aacf0cbd95d2facb8442268f2b1cc1fc
-RUN sha256sum /lib/libwasmvm_muslc.x86_64.a | grep f6282df732a13dec836cda1f399dd874b1e3163504dbd9607c6af915b2740479
-
-# CosmWasm: copy the right library according to architecture. The final location will be found by the linker flag `-lwasmvm_muslc`
-RUN cp /lib/libwasmvm_muslc.$(uname -m).a /lib/libwasmvm_muslc.a
-
-RUN BUILD_TAGS=muslc LINK_STATICALLY=true make build
-
-# --------------------------------------------------------
-# Runner
-# --------------------------------------------------------
-
-FROM ubuntu
-
-COPY --from=build /osmosis/build/osmosisd /bin/osmosisd
-COPY /tests/localosmosis/mainnet_state/statesync.sh /osmosis/statesync.sh
-COPY /tests/localosmosis/mainnet_state/testnetify.py /osmosis/testnetify.py
-COPY /tests/localosmosis/testnet_genesis.json /osmosis/testnet_genesis.json
-
-ENV HOME /osmosis
-WORKDIR $HOME
-# RUN apk update
-# RUN apk add jq
-# RUN apk add moreutils
-# RUN rm -rf /var/cache/apk/*
-# RUN apk add --no-cache python3 py3-pip
-# RUN apt-get update && apt-get install -y software-properties-common gcc && \
-#     add-apt-repository -y ppa:deadsnakes/ppa
-# RUN apt-get update && apt-get install -y python3.6 python3-distutils python3-pip python3-apt
-RUN apt-get update && apt-get install -y python3
-RUN chmod +x /osmosis/statesync.sh
-RUN /osmosis/statesync.sh
-ARG ID=localosmosis
-RUN python3 /osmosis/testnetify.py --chain-id=$ID
-RUN cp testnet_genesis.json .osmosisd/config/genesis.json
-EXPOSE 26656
-EXPOSE 26657
-EXPOSE 1317
-
-ENTRYPOINT ["osmosisd"]
-CMD ["start", "--x-crisis-skip-assert-invariants"]

tests/localosmosis/mainnet_state/docker-compose-state-export.yml

-version: "3"
-
-services:
-  osmosisd:
-    image: local:osmosis-se
-    user: "root:root"
-    command:
-      - start
-      - --x-crisis-skip-assert-invariants
-      - --rpc.laddr=tcp://0.0.0.0:26657
-    ports:
-      - "26657:26657"
-      - "1317:1317"
-      - "9090:9090"
-      - "9091:9091"

tests/localosmosis/mainnet_state/statesync.sh

-#!/bin/bash
-
-# initialize osmosis
-osmosisd init --chain-id=localosmosis val
-# remove seeds
-sed -i.bak -E 's#^(seeds[[:space:]]+=[[:space:]]+).*$#\1""#' ~/.osmosisd/config/config.toml
-sed -i.bak -E 's#^(fast_sync[[:space:]]+=[[:space:]]+).*$#\1false#' ~/.osmosisd/config/config.toml

tests/localosmosis/mainnet_state/testnetify.py

-import json
-import subprocess
-import re, shutil, tempfile
-from datetime import datetime
-import argparse
-import sys
-from sys import argv
-
-class CustomHelpFormatter(argparse.HelpFormatter):
-    def _format_action_invocation(self, action):
-        if not action.option_strings or action.nargs == 0:
-            return super()._format_action_invocation(action)
-        return ', '.join(action.option_strings)
-    def _split_lines(self, text, width):
-        if text.startswith('R|'):
-            return text[2:].splitlines()
-        # this is the RawTextHelpFormatter._split_lines
-        return argparse.HelpFormatter._split_lines(self, text, width)
-
-fmt = lambda prog: CustomHelpFormatter(prog,max_help_position=30)
-parser = argparse.ArgumentParser(description="Osmosis Installer",formatter_class=fmt)
-
-choice = parser.add_argument_group('Choices')
-
-choice.add_argument(
-    '-c',
-    '--chain-id',
-    type = str,
-    default="localosmosis",
-    help='R|Chain ID for newly created testnet \nDefault: localosmosis\n ',
-    dest="chainID")
-
-if not len(sys.argv) > 1:
-    parser.set_defaults(chainID="localosmosis")
-
-args = parser.parse_args()
-
-#get values from your priv_validator_key.json to later switch with high power validator
-
-daemon_name = "osmosisd"
-
-#get bas64
-result = subprocess.run([daemon_name,"tendermint","show-validator"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
-base64 = result.stdout.strip()
-##base64 = '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"3QVAkiUIkKR3B6kkbd+QqzWDdcExoggbZV5fwH4jKDs="}'
-
-#get validator cons pubkey
-val_pubkey = base64[base64.find('key":') +6 :-2]
-##val_pubkey = "3QVAkiUIkKR3B6kkbd+QqzWDdcExoggbZV5fwH4jKDs="
-
-#osmosisd debug pubkey {base64} to get address
-debug_pubkey = subprocess.run([daemon_name,"debug", "pubkey", base64], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
-
-#address
-address = debug_pubkey.stderr[9: debug_pubkey.stderr.find("\n")]
-##based on show-valdiator
-##address = "214D831D6F49A75F9104BDC3F2E12A6CC1FC5669"
-
-#feed address into osmosisd debug addr {address} to get bech32 validator address (osmovaloper)
-bech32 = subprocess.run([daemon_name,"debug", "addr", address], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
-#osmovaloper
-bech32_val = bech32.stderr[bech32.stderr.find("Val: ") + 5: -1]
-##operator address
-##bech32_val = "osmovaloper1y9xcx8t0fxn4lygyhhpl9cf2dnqlc4nf4pymm4"
-
-#pass osmovaloper address into osmosisd debug bech32-convert -p osmovalcons
-bech32_convert = subprocess.run([daemon_name,"debug", "bech32-convert", bech32_val, "-p", "osmovalcons"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
-#osmovalcons
-final_address = bech32_convert.stderr[:bech32_convert.stderr.find("\n")]
-##osmovalcons is taken from show-validator
-##final_address = "osmovalcons1y9xcx8t0fxn4lygyhhpl9cf2dnqlc4nfpjh8h5"
-
-#own opp address
-#exchange the op_address and op_pubkey with own address and pubkey or use above mnemonic for following address
-#bottom loan skill merry east cradle onion journey palm apology verb edit desert impose absurd oil bubble sweet glove shallow size build burst effort
-#CAN MODIFY
-op_address = "osmo12smx2wdlyttvyzvzg54y2vnqwq2qjateuf7thj"
-
-#own pub key
-#op_base64_pre = subprocess.run(["osmosisd","query", "auth", "account", op_address], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
-#op_pubkey = op_base64_pre.stdout[op_base64_pre.stdout.find("key: ")+5:op_base64_pre.stdout.find("sequence")-1]
-#CAN MODIFY
-op_pubkey = "A2MR6q+pOpLtdxh0tHHe2JrEY2KOcvRogtLxHDHzJvOh"
-
-#feed address into osmosisd debug addr {address} to get bech32 validator op address (osmovaloper)
-# bech32_op = subprocess.run([daemon_name,"debug", "addr", op_address], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
-#osmovaloper
-# bech32_valoper = bech32_op.stderr[bech32_op.stderr.find("Val: ") + 5: -1]
-# osmovaloper12smx2wdlyttvyzvzg54y2vnqwq2qjatex7kgq4
-
-def sed_inplace(filename, pattern, repl):
-    '''
-    Perform the pure-Python equivalent of in-place `sed` substitution: e.g.,
-    `sed -i -e 's/'${pattern}'/'${repl}' "${filename}"`.
-    '''
-    # For efficiency, precompile the passed regular expression.
-    pattern_compiled = re.compile(pattern)
-
-    # For portability, NamedTemporaryFile() defaults to mode "w+b" (i.e., binary
-    # writing with updating)
-    with tempfile.NamedTemporaryFile(mode='w', delete=False) as tmp_file:
-        with open(filename) as src_file:
-            for line in src_file:
-                tmp_file.write(pattern_compiled.sub(repl, line))
-
-    # Overwrite the original file with the munged temporary file in a
-    # manner preserving file attributes (e.g., permissions).
-    shutil.copystat(filename, tmp_file.name)
-    shutil.move(tmp_file.name, filename)
-
-
-#validator cons pubkey:     val_pubkey
-#osmovalcons:               final_address
-#validator hex address:     address
-#osmovaloper:               bech32_valoper
-#actual account:            op_address
-#accounts pubkey:           op_pubkey
-
-
-
-#replace validator cons pubkey (this did not work well due to random forward slashes, did other way later)
-#sed_inplace("testnet_genesis.json", "b77zCh/VsRgVvfGXuW4dB+Dhg4PrMWWBC5G2K/qFgiU=", val_pubkey)
-
-#replace validator osmovalcons
-print("Replacing osmovalcons1z6skn9g6s7py0klztr7acutr3anqd52k9x5p70 with " + final_address)
-sed_inplace("testnet_genesis.json", "osmovalcons1z6skn9g6s7py0klztr7acutr3anqd52k9x5p70", final_address)
-
-#replace validator hex address
-print("Replacing 16A169951A878247DBE258FDDC71638F6606D156 with " + address)
-sed_inplace("testnet_genesis.json", "16A169951A878247DBE258FDDC71638F6606D156", address)
-
-#replace validator osmovaloper
-#print("Replacing osmovaloper1cyw4vw20el8e7ez8080md0r8psg25n0cq98a9n with " + bech32_valoper)
-#sed_inplace("testnet_genesis.json", "osmovaloper1cyw4vw20el8e7ez8080md0r8psg25n0cq98a9n", bech32_valoper)
-
-#replace actual account
-print("Replacing osmo1cyw4vw20el8e7ez8080md0r8psg25n0c6j07j5 with " + op_address)
-sed_inplace("testnet_genesis.json", "osmo1cyw4vw20el8e7ez8080md0r8psg25n0c6j07j5", op_address)
-
-#replace actual account pubkey
-print("Replacing AqlNb1FM8veQrT4/apR5B3hww8VApc0LTtZnXhq7FqG0 with " + op_pubkey)
-sed_inplace("testnet_genesis.json", "AqlNb1FM8veQrT4/apR5B3hww8VApc0LTtZnXhq7FqG0", op_pubkey)
-
-
-
-#open genesis json file with read write priv, load json
-test_gen = open("testnet_genesis.json", "r+")
-read_test_gen = json.loads(test_gen.read())
-#print(read_test_gen.keys())
-
-
-#change chain-id
-print("Current chain-id is " + read_test_gen['chain_id'])
-#CAN MODIFY
-if args.chainID:
-    new_chain_id = args.chainID
-else:
-    new_chain_id = "localosmosis"
-read_test_gen['chain_id'] = new_chain_id
-print("New chain-id is " + read_test_gen['chain_id'])
-
-
-#validator pubkey must be replaced from b77zCh/VsRgVvfGXuW4dB+Dhg4PrMWWBC5G2K/qFgiU= in two locations
-#first under read_test_gen['app_state']['staking']['validators']
-#second under read_test_gen['validators']
-#i tried to do this using sed_inplace above but the multiple slashes broke it so did this instead
-
-#first val list index
-app_state_val_list = read_test_gen['app_state']['staking']['validators']
-val_index = [i for i, elem in enumerate(app_state_val_list) if 'Sentinel' in elem['description']['moniker']][0]
-#first val list update key
-#based on val
-app_state_val_list[val_index]['consensus_pubkey']['key'] = val_pubkey
-#also update delegator shares and tokens
-current_del_share = str(app_state_val_list[val_index]['delegator_shares'])
-print("Current delegator shares is " + current_del_share)
-app_state_val_list[val_index]['delegator_shares'] = str(int(float(app_state_val_list[val_index]['delegator_shares']) + 1000000000000000)) + ".000000000000000000"
-print("New delegator shares is " + app_state_val_list[val_index]['delegator_shares'])
-print("Current delegator tokens is " + app_state_val_list[val_index]['tokens'])
-app_state_val_list[val_index]['tokens'] = str(int(app_state_val_list[val_index]['tokens']) + 1000000000000000)
-print("New delegator tokens is " + app_state_val_list[val_index]['tokens'])
-
-#second val list index
-val_list_2 = read_test_gen['validators']
-val_list_2_index = [i for i, elem in enumerate(val_list_2) if 'Sentinel' in elem['name']][0]
-#second val list update key
-#based on val
-val_list_2[val_list_2_index]['pub_key']['value'] = val_pubkey
-
-
-#distribution module fix
-dist_address = "osmo1jv65s3grqf6v6jl3dp4t6c9t9rk99cd80yhvld"
-app_state_balances_list = read_test_gen['app_state']['bank']['balances']
-dist_index = [i for i, elem in enumerate(app_state_balances_list) if dist_address in elem['address']][0]
-dist_all = app_state_balances_list[dist_index]['coins']
-osmo_index = [i for i, elem in enumerate(dist_all) if 'uosmo' in elem['denom']][0]
-current_dist_osmo_bal = dist_all[osmo_index]['amount']
-dist_offset_amt = 2
-print("Current distribution account uosmo balance is " + current_dist_osmo_bal)
-new_dist_osmo_bal = str(int(current_dist_osmo_bal) - dist_offset_amt)
-print("New distribution account uosmo balance is " + new_dist_osmo_bal)
-dist_all[osmo_index]['amount'] = new_dist_osmo_bal
-
-
-#change self delegation amount on operator address
-
-#first location
-app_state_del_list = read_test_gen['app_state']['staking']['delegations']
-del_index = [i for i, elem in enumerate(app_state_del_list) if op_address in elem['delegator_address']][0]
-#first val list update share (add 1 BN)
-current_share = app_state_del_list[del_index]['shares']
-print("Current self delegation is " + str(current_share))
-new_share = str(int(float(current_share)) + 1000000000000000)+".000000000000000000"
-print("New self delegation is " + new_share)
-app_state_del_list[del_index]['shares'] = new_share
-
-#second location
-app_state_dist_list = read_test_gen['app_state']['distribution']['delegator_starting_infos']
-dist_index = [i for i, elem in enumerate(app_state_dist_list) if op_address in elem['delegator_address']][0]
-#second val list update stake (add 1 BN)
-current_stake = app_state_dist_list[dist_index]['starting_info']['stake']
-print("Current stake is " + str(current_stake))
-new_stake = str(int(float(current_stake)) + 1000000000000000)+".000000000000000000"
-print("New stake is " + new_stake)
-app_state_dist_list[dist_index]['starting_info']['stake'] = new_stake
-
-
-#get index of val power
-val_power_list = read_test_gen['validators']
-val_power_index = [i for i, elem in enumerate(val_power_list) if 'Sentinel' in elem['name']][0]
-#change val power (add 1 BN)
-current_power = int(val_power_list[val_power_index]['power'])
-print("Current validator power is " + str(current_power))
-new_power = str(current_power + 1000000000)
-print("New validator power is " + new_power)
-val_power_list[val_power_index]['power'] = new_power
-#get index of val power in app state (osmovaloper) (bech32_valoper)
-last_val_power_list = read_test_gen['app_state']['staking']['last_validator_powers']
-#last_val_power_index = [i for i, elem in enumerate(last_val_power_list) if bech32_valoper in elem['address']][0]
-last_val_power_index = [i for i, elem in enumerate(last_val_power_list) if 'osmovaloper1cyw4vw20el8e7ez8080md0r8psg25n0cq98a9n' in elem['address']][0]
-val_power = int(read_test_gen['app_state']['staking']['last_validator_powers'][last_val_power_index]['power'])
-print("Current validator power in second location is " + str(val_power))
-new_val_power = str(val_power + 1000000000)
-print("New validator power in second location is " + new_val_power)
-read_test_gen['app_state']['staking']['last_validator_powers'][last_val_power_index]['power'] = new_val_power
-
-
-#update last_total_power (last total bonded across all validators, add 1BN)
-last_total_power = int(read_test_gen['app_state']['staking']['last_total_power'])
-print("Current last total power is " + str(last_total_power))
-new_last_total_power = str(last_total_power + 1000000000)
-print("New last total power is " + new_last_total_power)
-read_test_gen['app_state']['staking']['last_total_power'] = new_last_total_power
-
-
-#update operator address amount (add 1 BN)
-#find wallet index in bank balance
-bank_balance_list = read_test_gen['app_state']['bank']['balances']
-op_amount_index = [i for i, elem in enumerate(bank_balance_list) if op_address in elem['address']][0]
-#get uosmo index from wallet
-op_wallet = read_test_gen['app_state']['bank']['balances'][op_amount_index]['coins']
-op_uosmo_index = [i for i, elem in enumerate(op_wallet) if 'uosmo' in elem['denom']][0]
-#update uosmo amount
-op_uosmo = int(op_wallet[op_uosmo_index]['amount'])
-print("Current operator address uosmo balance is " + str(op_uosmo))
-new_op_uosmo = str(op_uosmo + 1000000000000000)
-print("New operator address uosmo balance is " + new_op_uosmo)
-op_wallet[op_uosmo_index]['amount'] = new_op_uosmo
-
-
-#update total OSMO supply (add 2 BN)
-#supply list (ibc, ion, osmo)
-supply = read_test_gen['app_state']['bank']['supply']
-
-#get index of osmo supply
-osmo_index = [i for i, elem in enumerate(supply) if 'uosmo' in elem['denom']][0]
-
-#get osmo supply value
-osmo_supply = supply[osmo_index]['amount']
-print("Current OSMO supply is " + osmo_supply)
-
-#update osmo supply to new total osmo value (add 2 Billion OSMO)
-#subtract by however much module account is subtracted by
-osmo_supply_new = int(osmo_supply) + 2000000000000000 - dist_offset_amt
-print("New OSMO supply is " + str(osmo_supply_new))
-supply[osmo_index]['amount'] = str(osmo_supply_new)
-
-
-#update bonded_tokens_pool module balance osmo1fl48vsnmsdzcv85q5d2q4z5ajdha8yu3aq6l09 (add 1BN)
-#get list of bank balances
-bank_bal_list = read_test_gen['app_state']['bank']['balances']
-#get index of module account
-module_acct_index = [i for i, elem in enumerate(bank_bal_list) if 'osmo1fl48vsnmsdzcv85q5d2q4z5ajdha8yu3aq6l09' in elem['address']][0]
-#get current value
-module_denom_list = bank_bal_list[module_acct_index]['coins']
-osmo_bal_index = [i for i, elem in enumerate(module_denom_list) if 'uosmo' in elem['denom']][0]
-osmo_bal = bank_bal_list[module_acct_index]['coins'][osmo_bal_index]['amount']
-print("Current bonded tokens pool module account balance is " + osmo_bal)
-#increase by 1BN
-new_osmo_bal = int(osmo_bal) + 1000000000000000
-print("New bonded tokens pool module account balance is " + str(new_osmo_bal))
-bank_bal_list[module_acct_index]['coins'][osmo_bal_index]['amount'] = str(new_osmo_bal)
-
-
-#edit epoch params
-#change epoch duration to 21600s
-epochs_list = read_test_gen['app_state']['epochs']['epochs'][0]
-duration_current = epochs_list['duration']
-print("Current epoch duration is " + duration_current)
-#21600s for 6 hour epoch
-#CAN MODIFY
-new_duration = '21600s'
-print("New epoch duration is " + new_duration)
-epochs_list['duration'] = new_duration
-
-#change current_epoch_start_time
-start_time_current = epochs_list['current_epoch_start_time']
-print("Current epoch start time is " + start_time_current)
-#today = date.today()
-now = datetime.now()
-#date_format = now.strftime("%Y-%m-%d")
-date_format = now.strftime("%Y-%m-%d"+"T"+"%H:%M:")
-start_time_current_list = list(start_time_current)
-start_time_current_list[:17] = date_format
-start_time_new = ''.join(start_time_current_list)
-epochs_list['current_epoch_start_time'] = start_time_new
-print("New epoch start time is " + start_time_new)
-
-
-#edit gov params
-#change VotingPeriod
-current_voting_period = read_test_gen['app_state']['gov']['voting_params']['voting_period']
-print("Current voting period is " + current_voting_period)
-#180s for 3 minute voting period
-#CAN MODIFY
-new_voting_period = "180s"
-print("New voting period is " + new_voting_period)
-read_test_gen['app_state']['gov']['voting_params']['voting_period'] = new_voting_period
-
-
-print("Please wait while file writes over itself, this may take 60 seconds or more")
-#go back to begining of file, write over with new values
-test_gen.seek(0)
-json.dump(read_test_gen, test_gen)
-
-#delete remainder in case new data is shorter than old
-test_gen.truncate()

tests/localosmosis/state_export/docker-compose.yml

+version: "3"
+
+services:
+
+  osmosisd:
+    image: local:osmosis
+    build:
+      context: ../../../
+      dockerfile: Dockerfile
+      args:
+        RUNNER_IMAGE: alpine:3.16
+        GO_VERSION: 1.18
+    volumes:
+      - ./scripts/start.sh:/osmosis/start.sh
+      - ./scripts/testnetify.py:/osmosis/testnetify.py
+      - ./state_export.json:/osmosis/state_export.json
+      - ./.osmosisd/:/osmosis/.osmosisd/
+    entrypoint:
+      - /osmosis/start.sh
+    environment:
+      - MONIKER=val
+      - CHAIN_ID=localosmosis
+    ports:
+      - 26657:26657
+      - 1317:1317
+      - 9090:9090
+      - 9091:9091

tests/localosmosis/state_export/scripts/start.sh

+#!/bin/sh
+set -e 
+set -o pipefail
+
+OSMOSIS_HOME=$HOME/.osmosisd
+CONFIG_FOLDER=$OSMOSIS_HOME/config
+
+DEFAULT_MNEMONIC="bottom loan skill merry east cradle onion journey palm apology verb edit desert impose absurd oil bubble sweet glove shallow size build burst effort"
+DEFAULT_CHAIN_ID="localosmosis"
+DEFAULT_MONIKER="val"
+
+# Override default values with environment variables
+MNEMONIC=${MNEMONIC:-$DEFAULT_MNEMONIC}
+CHAIN_ID=${CHAIN_ID:-$DEFAULT_CHAIN_ID}
+MONIKER=${MONIKER:-$DEFAULT_MONIKER}
+
+install_prerequisites () {
+    apk add -q --no-cache \
+        dasel \
+        python3 \
+        py3-pip
+}
+
+edit_config () {
+
+    # Remove seeds
+    dasel put string -f $CONFIG_FOLDER/config.toml '.p2p.seeds' ''
+
+    # Disable fast_sync
+    dasel put bool -f $CONFIG_FOLDER/config.toml '.fast_sync' 'false'
+
+    # Expose the rpc
+    dasel put string -f $CONFIG_FOLDER/config.toml '.rpc.laddr' "tcp://0.0.0.0:26657"
+}
+
+if [[ ! -d $CONFIG_FOLDER ]]
+then
+
+    install_prerequisites
+
+    echo "Chain ID: $CHAIN_ID"
+    echo "Moniker:  $MONIKER"
+
+    echo $MNEMONIC | osmosisd init -o --chain-id=$CHAIN_ID --home $OSMOSIS_HOME --recover $MONIKER 2> /dev/null
+    echo $MNEMONIC | osmosisd keys add my-key --recover --keyring-backend test > /dev/null 2>&1
+
+    ACCOUNT_PUBKEY=$(osmosisd keys show --keyring-backend test my-key --pubkey | dasel -r json '.key' --plain)
+    ACCOUNT_ADDRESS=$(osmosisd keys show -a --keyring-backend test my-key --bech acc)
+
+    VALIDATOR_PUBKEY_JSON=$(osmosisd tendermint show-validator --home $OSMOSIS_HOME)
+    VALIDATOR_PUBKEY=$(echo $VALIDATOR_PUBKEY_JSON | dasel -r json '.key' --plain)
+    VALIDATOR_HEX_ADDRESS=$(osmosisd debug pubkey $VALIDATOR_PUBKEY_JSON 2>&1 --home $OSMOSIS_HOME | grep Address | cut -d " " -f 2)
+    VALIDATOR_ACCOUNT_ADDRESS=$(osmosisd debug addr $VALIDATOR_HEX_ADDRESS 2>&1  --home $OSMOSIS_HOME | grep Acc | cut -d " " -f 3)
+    VALIDATOR_OPERATOR_ADDRESS=$(osmosisd debug addr $VALIDATOR_HEX_ADDRESS 2>&1  --home $OSMOSIS_HOME | grep Val | cut -d " " -f 3)
+    VALIDATOR_CONSENSUS_ADDRESS=$(osmosisd debug bech32-convert $VALIDATOR_OPERATOR_ADDRESS -p osmovalcons  --home $OSMOSIS_HOME 2>&1)
+
+    python3 -u testnetify.py \
+    -i /osmosis/state_export.json \
+    -o $CONFIG_FOLDER/genesis.json \
+    -c $CHAIN_ID \
+    --validator-hex-address $VALIDATOR_HEX_ADDRESS \
+    --validator-operator-address $VALIDATOR_OPERATOR_ADDRESS \
+    --validator-consensus-address $VALIDATOR_CONSENSUS_ADDRESS \
+    --validator-pubkey $VALIDATOR_PUBKEY \
+    --account-pubkey $ACCOUNT_PUBKEY \
+    --account-address $ACCOUNT_ADDRESS \
+    --prune-ibc
+
+    edit_config
+fi
+
+osmosisd start --home $OSMOSIS_HOME --x-crisis-skip-assert-invariants

tests/localosmosis/state_export/scripts/testnetify.py

+import json
+import argparse
+from datetime import datetime
+from dataclasses import dataclass
+
+# Classes 
+@dataclass
+class Validator:
+    moniker: str
+    pubkey: str
+    hex_address: str
+    operator_address: str
+    consensus_address: str
+
+@dataclass
+class Account:
+    pubkey: str
+    address: str
+
+# Contants
+BONDED_TOKENS_POOL_MODULE_ADDRESS = "osmo1fl48vsnmsdzcv85q5d2q4z5ajdha8yu3aq6l09"
+DISTRIBUTION_MODULE_ADDRESS = "osmo1jv65s3grqf6v6jl3dp4t6c9t9rk99cd80yhvld"
+DISTRIBUTION_MODULE_OFFSET = 2
+
+config = {
+    "governance_voting_period": "180s",
+    "epoch_duration": '21600s',
+}
+
+def replace(d, old_value, new_value):
+    """
+    Replace all the occurences of `old_value` with `new_value`
+    in `d` dictionary
+    """
+    for k in d.keys():
+        if isinstance(d[k], dict):
+            replace(d[k], old_value, new_value)
+        elif isinstance(d[k], list):
+            for i in range(len(d[k])):
+                if isinstance(d[k][i], dict) or isinstance(d[k][i], list):
+                    replace(d[k][i], old_value, new_value)
+                else:
+                    if d[k][i] == old_value:
+                        d[k][i] = new_value
+        else:
+            if d[k] == old_value:
+                d[k] = new_value
+
+def replace_validator(genesis, old_validator, new_validator):
+    
+    replace(genesis, old_validator.hex_address, new_validator.hex_address)
+    replace(genesis, old_validator.consensus_address, new_validator.consensus_address)
+
+    # replace(genesis, old_validator.pubkey, new_validator.pubkey)
+    for validator in genesis["validators"]:
+        if validator['name'] == old_validator.moniker:
+            validator['pub_key']['value'] = new_validator.pubkey
+        
+    for validator in genesis['app_state']['staking']['validators']:
+        if validator['description']['moniker'] == old_validator.moniker:
+            validator['consensus_pubkey']['key'] = new_validator.pubkey
+
+    # This creates problems
+    # replace(genesis, old_validator.operator_address, new_validator.operator_address)
+    
+    # replacing operator_address in lockup > synthetic_locks
+    # for synthetic_lock in genesis['app_state']['lockup']['synthetic_locks']:ß
+    #     if synthetic_lock['synth_denom'].endswith(old_validator.operator_address):
+    #         synthetic_lock['synth_denom'] = synthetic_lock['synth_denom'].replace(
+    #             old_validator.operator_address, new_validator.operator_address)
+
+    # Replacing operator_address in incentives > gauges
+    # for gauge in genesis['app_state']['incentives']['gauges']:
+    #     if gauge['distribute_to']['denom'].endswith(old_validator.operator_address):
+    #         gauge['distribute_to']['denom'] = gauge['distribute_to']['denom'].replace(
+    #             old_validator.operator_address, new_validator.operator_address)
+
+def replace_account(genesis, old_account, new_account):
+
+    replace(genesis, old_account.address, new_account.address)
+    replace(genesis, old_account.pubkey, new_account.pubkey)
+
+def create_parser():
+
+    parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    description='Create a testnet from a state export')
+
+    parser.add_argument(
+        '-c',
+        '--chain-id',
+        type = str,
+        default="localosmosis",
+        help='Chain ID for the testnet \nDefault: localosmosis\n'
+    )
+
+    parser.add_argument(
+        '-i',
+        '--input',
+        type = str,
+        default="state_export.json",
+        dest='input_genesis',
+        help='Path to input genesis'
+    )
+
+    parser.add_argument(
+        '-o',
+        '--output',
+        type = str,
+        default="testnet_genesis.json",
+        dest='output_genesis',
+        help='Path to output genesis'
+    )
+    
+    parser.add_argument(
+        '--validator-hex-address',
+        type = str,
+        help='Validator hex address to replace'
+    )
+
+    parser.add_argument(
+        '--validator-operator-address',
+        type = str,
+        help='Validator operator address to replace'
+    )
+
+    parser.add_argument(
+        '--validator-consensus-address',
+        type = str,
+        help='Validator consensus address to replace'
+    )
+
+    parser.add_argument(
+        '--validator-pubkey',
+        type = str,
+        help='Validator pubkey to replace'
+    )
+
+    parser.add_argument(
+        '--account-pubkey',
+        type = str,
+        help='Account pubkey to replace'
+    )
+
+    parser.add_argument(
+        '--account-address',
+        type = str,
+        help='Account address to replace'
+    )
+
+    parser.add_argument(
+        '-q',
+        '--quiet',
+        action='store_false',
+        help='Less verbose output'
+    )
+
+    parser.add_argument(
+        '--prune-ibc', 
+        action='store_true',
+        help='Prune the IBC module'
+    )
+
+    parser.add_argument(
+        '--pretty-output', 
+        action='store_true',
+        help='Properly indent output genesis (increases time and file size)'
+    )
+
+    return parser
+
+def main():
+
+    parser = create_parser()
+    args = parser.parse_args()
+
+    new_validator = Validator(
+        moniker = "val",
+        pubkey = args.validator_pubkey,
+        hex_address = args.validator_hex_address,
+        operator_address = args.validator_operator_address,
+        consensus_address = args.validator_consensus_address
+    )
+
+    old_validator = Validator(
+        moniker = "Sentinel dVPN",
+        pubkey = "b77zCh/VsRgVvfGXuW4dB+Dhg4PrMWWBC5G2K/qFgiU=",
+        hex_address = "16A169951A878247DBE258FDDC71638F6606D156",
+        operator_address = "osmovaloper1cyw4vw20el8e7ez8080md0r8psg25n0cq98a9n",
+        consensus_address = "osmovalcons1z6skn9g6s7py0klztr7acutr3anqd52k9x5p70"
+    )
+
+    new_account = Account(
+        pubkey = args.account_pubkey,
+        address = args.account_address
+    )
+
+    old_account = Account(
+        pubkey = "AqlNb1FM8veQrT4/apR5B3hww8VApc0LTtZnXhq7FqG0",
+        address = "osmo1cyw4vw20el8e7ez8080md0r8psg25n0c6j07j5"
+    )
+
+    print("📝 Opening {}... (it may take a while)".format(args.input_genesis))
+    with open(args.input_genesis, 'r') as f:
+        genesis = json.load(f)
+    
+    # Replace chain-id
+    if args.verbose:
+        print("🔗 Replace chain-id {} with {}".format(genesis['chain_id'], args.chain_id))
+    genesis['chain_id'] = args.chain_id
+
+    # Update gov module
+    if args.verbose:
+        print("🗳️ Update gov module")
+        print("\tModify governance_voting_period from {} to {}".format(
+            genesis['app_state']['gov']['voting_params']['voting_period'],
+            config["governance_voting_period"]))
+    genesis['app_state']['gov']['voting_params']['voting_period'] = config["governance_voting_period"]
+
+    # Update epochs module
+    if args.verbose:
+        print("⌛ Update epochs module")
+        print("\tModify epoch_duration from {} to {}".format(
+            genesis['app_state']['epochs']['epochs'][0]['duration'],
+            config["epoch_duration"]))
+        print("\tReset current_epoch_start_time")
+    genesis['app_state']['epochs']['epochs'][0]['duration'] = config["epoch_duration"]
+    genesis['app_state']['epochs']['epochs'][0]['current_epoch_start_time'] = datetime.now().isoformat() + 'Z'
+    
+    # Prune IBC
+    if args.prune_ibc:
+        if args.verbose:
+            print("🕸 Pruning IBC module")
+
+        genesis['app_state']["ibc"]["channel_genesis"]["ack_sequences"] = []
+        genesis['app_state']["ibc"]["channel_genesis"]["acknowledgements"] = []
+        genesis['app_state']["ibc"]["channel_genesis"]["channels"] = []
+        genesis['app_state']["ibc"]["channel_genesis"]["commitments"] = []
+        genesis['app_state']["ibc"]["channel_genesis"]["receipts"] = []
+        genesis['app_state']["ibc"]["channel_genesis"]["recv_sequences"] = []
+        genesis['app_state']["ibc"]["channel_genesis"]["send_sequences"] = []
+
+        genesis['app_state']["ibc"]["client_genesis"]["clients"] = []
+        genesis['app_state']["ibc"]["client_genesis"]["clients_consensus"] = []
+        genesis['app_state']["ibc"]["client_genesis"]["clients_metadata"] = []
+
+    # Impersonate validator
+    if args.verbose:
+        print("🚀 Replace validator")
+
+        # print("\t{:50} -> {}".format(old_validator.moniker, new_validator.moniker))
+        print("\t{:20} {}".format("Pubkey", new_validator.pubkey))
+        print("\t{:20} {}".format("Consensus address", new_validator.consensus_address))
+        print("\t{:20} {}".format("Operator address", new_validator.operator_address))
+        print("\t{:20} {}".format("Hex address", new_validator.hex_address))
+
+    replace_validator(genesis, old_validator, new_validator)
+
+    # Impersonate account
+    if args.verbose:
+        print("🧪 Replace account")
+        print("\t{:20} {}".format("Pubkey", new_account.pubkey))
+        print("\t{:20} {}".format("Address", new_account.address))
+    
+    replace_account(genesis, old_account, new_account)
+        
+    # Update staking module
+    if args.verbose:
+        print("🥩 Update staking module")
+
+    # Replace validator pub key in genesis['app_state']['staking']['validators']
+    for validator in genesis['app_state']['staking']['validators']:
+        if validator['description']['moniker'] == old_validator.moniker:
+            
+            # Update delegator shares
+            validator['delegator_shares'] = str(int(float(validator['delegator_shares']) + 1000000000000000)) + ".000000000000000000"
+            print("\tUpdate delegator shares to {}".format(validator['delegator_shares']))
+
+            # Update tokens
+            validator['tokens'] = str(int(validator['tokens']) + 1000000000000000)
+            print("\tUpdate tokens to {}".format(validator['tokens']))
+            break
+    
+    # Update self delegation on operator address
+    for delegation in genesis['app_state']['staking']['delegations']:
+        if delegation['delegator_address'] == new_account.address:
+
+            # delegation['validator_address'] = new_validator.operator_address
+            delegation['shares'] = str(int(float(delegation['shares'])) + 1000000000000000) + ".000000000000000000"
+            print("\tUpdate {} delegation shares to {} to {}".format(new_account.address, delegation['validator_address'], delegation['shares']))
+            break
+
+    # Update genesis['app_state']['distribution']['delegator_starting_infos'] on operator address
+    for delegator_starting_info in genesis['app_state']['distribution']['delegator_starting_infos']:
+        if delegator_starting_info['delegator_address'] == new_account.address:
+            delegator_starting_info['starting_info']['stake'] = str(int(float(delegator_starting_info['starting_info']['stake']) + 1000000000000000))+".000000000000000000"
+            print("\tUpdate {} stake to {}".format(delegator_starting_info['delegator_address'], delegator_starting_info['starting_info']['stake']))
+            break
+
+    if args.verbose:
+        print("🔋 Update validator power")
+
+    # Update power in genesis["validators"]
+    for validator in genesis["validators"]:
+        if validator['name'] == old_validator.moniker:
+            validator['power'] = str(int(validator['power']) + 1000000000)
+            print("\tUpdate {} validator power to {}".format(validator['address'], validator['power']))
+            break 
+    
+    for validator_power in genesis['app_state']['staking']['last_validator_powers']:
+        if validator_power['address'] == old_validator.operator_address:
+            validator_power['power'] = str(int(validator_power['power']) + 1000000000)
+            if args.verbose:
+                print("\tUpdate {} last_validator_power to {}".format(old_validator.operator_address, validator_power['power']))
+            break
+    
+    # Update total power
+    genesis['app_state']['staking']['last_total_power'] = str(int(genesis['app_state']['staking']['last_total_power']) + 1000000000)
+    if args.verbose:
+        print("\tUpdate last_total_power to {}".format(genesis['app_state']['staking']['last_total_power']))
+
+    # Update bank module
+    if args.verbose:
+        print("💵 Update bank module")
+
+    for balance in genesis['app_state']['bank']['balances']:
+        if balance['address'] == new_account.address:
+            for coin in balance['coins']:
+                if coin['denom'] == "uosmo":
+                    coin["amount"] = str(int(coin["amount"]) + 1000000000000000)
+                    print("\tUpdate {} uosmo balance to {}".format(new_account.address, coin["amount"]))
+                    break
+            break
+    
+    # Add 1 BN uosmo to bonded_tokens_pool module address
+    for balance in genesis['app_state']['bank']['balances']:
+        if balance['address'] == BONDED_TOKENS_POOL_MODULE_ADDRESS:
+            # Find uosmo
+            for coin in balance['coins']:
+                if coin['denom'] == "uosmo":
+                    coin["amount"] = str(int(coin["amount"]) + 1000000000000000)
+                    print("\tUpdate {} (bonded_tokens_pool_module) uosmo balance to {}".format(BONDED_TOKENS_POOL_MODULE_ADDRESS, coin["amount"]))
+                    break
+            break
+    
+    # Distribution module fix
+    for balance in genesis['app_state']['bank']['balances']:
+        if balance['address'] == DISTRIBUTION_MODULE_ADDRESS:
+            # Find uosmo
+            for coin in balance['coins']:
+                if coin['denom'] == "uosmo":
+                    coin["amount"] = str(int(coin["amount"]) - DISTRIBUTION_MODULE_OFFSET)
+                    print("\tUpdate {} (distribution_module) uosmo balance to {}".format(DISTRIBUTION_MODULE_ADDRESS, coin["amount"]))
+                    break
+            break
+
+    # Update bank balance 
+    for supply in genesis['app_state']['bank']['supply']:
+        if supply["denom"] == "uosmo":
+            print("\tUpdate total uosmo supply from {} to {}".format(supply["amount"], str(int(supply["amount"]) + 2000000000000000 - DISTRIBUTION_MODULE_OFFSET)))
+            supply["amount"] = str(int(supply["amount"]) + 2000000000000000 - DISTRIBUTION_MODULE_OFFSET)
+            break
+    
+    print("📝 Writing {}... (it may take a while)".format(args.output_genesis))
+    with open(args.output_genesis, 'w') as f:
+        if args.pretty_output:
+            f.write(json.dumps(genesis, indent=2))
+        else:
+            f.write(json.dumps(genesis))
+
+if __name__ == '__main__':
+    main()
\ No newline at end of file