Recently we added the wasm binaries to the `rerun-if-changed` list. The problem with that is that
they have a later mtime than the `invoked.timestamp` file and this file's mtime is used to determine
if the `build.rs` script needs to be re-run. The solution to this is that we copy the mtime of this
`invoked.timestamp` file and add it to the wasm binaries. Then cargo/rustc doesn't constantly wants
to rerun the `build.rs` script.
* Sync: Improve major sync detection
When we still have a full import queue, we should still report that we are in major sync mode,
otherwise validators may will already start producing blocks.
* Use median
* Review comments
If `--validator` is passed we also listen on this address and as `--dev` is a shortcut for multiple
CLI args, including `--validator`, we should make it consistent.
* trie: Optimize `keys` function
Instead of iterating the entire state and collecting all keys that match the given prefix, we can
directly use the optimized prefix iterator.
* Add a test
* generate_storage_alias: Rewrite as proc macro attribute
This rewrites the `generate_storage_alias!` declarative macro as proc-macro attribute. While doing
this the name is changed to `storage_alias`. The prefix can now also be the name of a pallet. This
makes storage aliases work in migrations for all kind of chains and not just for the ones that use
predefined prefixes.
* Fix compilation and FMT
* Moare fixes
* 🤦
* ......
* Rework the syntax and support instancing
* FMT
* Prefix variants with `Storage`
* Make it compile
* Fix where clause on rust stable
When passing reserved nodes only with a peer id it was failing with the `DuplicateBootnode` error.
Besides that there are some clean ups. We for example added the bootnodes twice to the `known_addresses`.
* pallet-asset: Fix transfer of a large amount of an asset
Before this pr transferring a large amount of an asset would check that transferring the asset would
not overflow the supply of the asset. However, it doesn't make sense to check for asset supply
overflow when we just transfer from one account to another account and don't increase the supply in
any way. It also required to extend the `can_deposit` method of `fungible` and `fungibles` with a
`mint` parameter. If this parameter is set to `true`, it means we want to mint the amount of an
asset before transferring it into an account. For `can_withdraw` we don't need to add an extra
parameter, because withdrawing should never be able to underflow the supply. If that would happen,
it would mean that somewhere the supply wasn't increased while increasing the balance of an account.
* Update frame/assets/src/functions.rs
* Update frame/assets/src/functions.rs
* Update frame/assets/src/functions.rs
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* FMT
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
This tries to fix the CI if there is no polkadot companion. Currently we don't update Polkadot
master in Cumulus, which means we may use some old commit that isn't compiling with the latest
Substrate master anymore. This can happen if there was a pr that had a companion in Polkadot, but no
companion was required for Cumulus. Then Cumulus will still point to some old Polkadot commit that
isn't compiling anymore with the latest Substrate commit. So, we need to tell the script to use the
latest master of Polkadot. If there is a companion for Polkadot, it would simply override the extra
dependency patch later on.
With the latest optimizations of the `FinalityNotification` generation, the aux data pruning started
to print a warning. The problem here was that we printed a warning and stopped the adding of blocks
to prune when we hit the `heigh_limit`. This is now wrong, as we could for example have two 512 long
forks and then we start finalizing one of them. The second fork head would be part of the stale
heads at some point (in the current implementation when we finalize second fork head number + 1),
but then we would actually need to go back into the past than `heigh_limit` (which was actually
last_finalized - 1). We now go back until we reach the canonical chain.
Also fixed some wrong comment that was added by be about the content of the `finalized` blocks in
the `FinalityNotification`.
* Finality notification: Optimize calculation of stale heads
While looking into some problem on Versi where a collator seemed to be stuck. I found out that it
was not stuck but there was a huge gap between last finalized and best block. This lead to a lot
leaves and it was basically trapped inside some loop of reading block headers from the db to find
the stale heads. While looking into this I found out that `leaves` already supports the feature to
give us the stale heads relative easily. However, the semantics change a little bit. Instead of
returning all stale heads of blocks that are not reachable anymore after finalizing a block, we
currently only return heads with a number lower than the finalized block. This should be no problem,
because these other leaves that are stale will be returned later when a block gets finalized which
number is bigger than the block number of these leaves.
While doing that, I also changed `tree_route` of the `FinalityNotification` to include the
`old_finalized`. Based on the comment I assumed that this was already part of it. However, if
wanted, I can revert this change.
* FMT
* Update client/service/src/client/client.rs
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Do not include the last finalized block
* Rename function
* FMT
* Fix tests
* Update figure
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Prepare for rust stable 1.59
Besides preparing the UI tests this also adds a new script update-rust-stable.sh script for
simplifying the update of a rust stable version. This script will run all UI tests for the new
rust stable version and updating the expected output.
* Ensure we run the UI tests in CI
* use staging ci image
* More test updates
* Unignore test (#11097)
* empty commit for pipeline rerun
* empty commit for pipeline rerun
* Try to make clippy happy
* More clippy fixes
* FMT
* ci image production
Co-authored-by: alvicsam <alvicsam@gmail.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
These docs didn't mention how the removal works internally. This is important for the user to know
that calling such a method multiple times in the same block leads always to the same result.
This pr changes all default execution strategies to `Wasm`. This is basically a deprecation of the
native runtime. While the native runtime isn't removed and can still be used, it will not be used
anymore by default.
This will also improve the usage for people who want to run random commits, which most often forget
to run with `--execution wasm`. Otherwise they often run into storage root mismatches because of
using the native runtime.
* transactional: Fix some nitpicks
This fixes some nitpicks related to the transactional storage stuff from me. As everything was
merged too fast, here are some nitpicks from me. First, the entire functionality is moved into its
own file to have a clear separation. Secondly I changed the `set_transactional_level` to not take
`Layer` by reference. Besides that I have added some docs etc.
* Add some comment
* Move tests
* 🤦
* import-blocks: Do not read `stdin` to memory
This fixes a bug with `import-blocks` reading the entire `stdin` before starting to import the
blocks. However, for huge files that uses quite a lot of memory. We can just read from `stdin` step
by step as we do it with a file. This ensures that we don't read the entire input at once into memory.
* FMT
* Fix warning
* pallet-macro: Ensure that building with `missing_docs` works
Before this pr it was failing when compiling with `missing_docs`, because the macro was generating
functions etc without the appropriate docs. In the case of this pr it is mainly about hiding these
functions in the docs as they are internal api anyway.
* Fix UI test
Ensure that the sp api macros don't use functions without providing the full path to the function.
This hygiene ensures that we don't actually try to call a method of an imported trait for example.
* sp-core: `full_crypto` doesn't imply `std`
This pr changes the feature set of `secp256k1` to not use `global-context` when only the
`full_crypto` is enabled. It will be slower when the `std` feature is not enabled as the context
always needs to be recreated, but that is fine.
* Update client/cli/src/arg_enums.rs
Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Davide Galassi <davxy@datawok.net>
Because Rust currently doesn't support specialization, it prevents users from implementing
`SlotWorker` for their own types. This pr solves this by removing the generic implementation of
`SlotWorker` for `SimpleSlotWorker` and providing some wrapper type for that.
* sc-cli: Fix bugs after switching to clap3
Before switching to clap3 we support cli options like `--reserved-nodes A B` and after you needed to
pass `--reserved-nodes` cli option multiple times `--reserved-nodes A --reserved-nodes B`. This is
fixed by setting `multiple_occurrences(true)` option. This also done for all the other `Vec` cli
options in `sc-cli`. Besides that `--sync` wasn't supporting case insensitive parsing of the value.
This is now also supported. For both regressions a test is added. Besides that the pr removes all
the `rename_all = PascalCase` attributes, because they are not needed. All other `ArgEnum`s were
checked and all are already using `ignore_case(true)`.
* Bring back `PascalCase`, because otherwise it falls back to `kebab-case`...
* state-machine: Move all functionality from trie backend to the essence
This is required for some future changes of me and it also makes more sense to have all the
functionality inside the essence. Besides that it changes the child root cache to directly
cache the hash.
* Update primitives/state-machine/src/trie_backend_essence.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* FMT
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* wasm-builder: Support latest nightly
With latest nightly, aka rust version 1.60+ namespaced features are added. This changes the handling
of optional dependencies. We currently have features that enable optional dependencies when `std` is
enabled. This was before no problem, but now the wasm-builder detects them as enabled. To support
the transition period until 1.60 is released as stable, this pr adds an heuristic to not enable these
optional crates in the wasm build when they are enabled in the `std` feature. This heuristic fails
when someones enables these optional dependencies from the outside as well as via the `std` feature,
however we hope that no one is doing this at the moment. When namespaced features are enabled, these
dependencies needs to be enabled using `dep:dependency-name` to solve this properly.
https://doc.rust-lang.org/cargo/reference/unstable.html#namespaced-features
* Remove accidentally added features
* Remove `u32_trait`
This trait only existed because there wasn't any const generic support at time of creation. However,
we now have support for it :)
* FMT
* pallet-scheduler: Fix migrations V2 to V3
V2 already supported origins, so we need to move them over instead of setting it to `Root`. Besides
that it also removes the custom `Releases` enum and moves it over to `StorageVersion`.
* Fixes
* Fixes
* 🤦
This is required for projects like Cumulus that have dependencies on `westend-runtime`, but this
runtime is only added when the `runtime-benchmarks` feature is enabled. By having all features
"enabled" in `cargo metadata` we ensure that all crates can be found.
* Remove some last `AccountId32::default()` leftovers
As we removed `Default` of account id, we have overseen some last bits. This pr removes these last
bits.
* Fix
When building a wasm binary from a different repo inside a local workspace, we did not used the
correct `Cargo.toml` to find the correct patches and features. The solution to this is to just walk
up from the target directory until we find the workspace we are currently compiling. If this
heuristic isn't working, we print a warning and let the user set an env variable
`WASM_BUILD_WORKSPACE_HINT` to tell the `wasm-builder` where the actual workspace is.
* runtime-interface: Implement `register_only` functions
The runtime interface supports versioning of functions. Currently, if you add a new function it will
be used by the runtime automatically. This results in requiring all nodes of a network to upgrade
before the runtime is upgraded, otherwise they will fail to instantiate the new runtime because of
missing host functions. This pr introduces `register_only` functions. This can be used when a new
runtime interface function should be introduced, but the actual usage can be deferred. This means
that nodes will have the host function for this, but the runtime will still use the old version of
the function when being compiled for wasm. However, when a runtime is enacted that uses the new host
function, the "old nodes" will already have the host function and will continue to work.
* Update primitives/runtime-interface/src/lib.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* Update primitives/runtime-interface/proc-macro/src/utils.rs
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
* FMT
Co-authored-by: cheme <emericchevalier.pro@gmail.com>
`runtime_print!` is printed by default using `debug`, aka not being visible. With `log::error!` it
will be printed directly to the user. Production networks like Polkadot disable logging, but for
them we run special nodes that have logging enabled.
The keystore would print "Invalid password" when a key was stored using an incorrect public key.
This pr improves the error message to communicate better to the user on what is wrong.
* Remove transaction-pool `test-helpers` feature
`test-helpers` feature is a bad idea in general, because once the feature is enabled somewhere in
the workspace, it is enabled anywhere. While removing the feature, the tests were also rewritten to
get rid off other "only test" related code.
Contributes towards: https://github.com/paritytech/substrate/issues/9727
* Apply suggestions from code review
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Fix benches
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Improve SS58 related errors
This improves the SS58 error, especially when it comes to parsing public keys with unknown SS58
address formats.
* Make CI happy
* More fixes
* More
* 🤦
* fml...
* chain-spec: Make chain spec writing deterministic
This switches the chain spec to use `BTreeMap` internally instead of `HashMap`. This ensures that
reading and writing the same chain spec always results in the same file.
* fmt
* SlotDuration: Always fetch the slot duration from the runtime
The slot duration should always be fetched from the runtime instead of being cached in the db. The
slot duration is only fetched on startup of the node, so the performance isn't that important. This
is especially helpful for the case when the slot duration of a chain should be changed through a
runtime upgrade (there be dragons, so take care).
* Fix docs
* Remove logging
* Fix warning
* rpc-author: Downgrade the logging level of some logs
Currently these messages are printed with `warn` levels, but they don't bring the node operator
anything. The problem with these log lines is that they are triggered by external users when they
send invalid transactions. This pr "resolves" this by downgrading the log severity to debug, aka not
shown by default.
* FMT
* Remove cargo unleash CI job
The CI job is currently breaking, because we released some crates. We also don't need the job
currently because releases are not done. So, the job currently just annoys people and eats CPU time.
This brings us to the conclusion that this job is currently not required and can be readded at a
later point when we are sure that we need it.
* Hopefully fix UI test
* Another try
* Remove `reserved_keyword` ui
* Introduce `SecretUri`
* `inspect-key`: Adds support for `expect-public`
`expect-public` can be used to check that a given secret uri corresponds to the given public key.
This is mainly useful when the secret uri is protected by a password and a new derived account
should be generated. With `--expect-public` the user can pass the public key/account-id of the
"base" secret uri aka the one without any derivation to ensure the correct password was inserted.
* Fixes
* 🤦
* Apply suggestions from code review
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
* Review feedback
* FMT
* Bump the versions
Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
This adds support to the wasm-override feature to compare the spec_name. If the spec_name doesn't
match, a warning will be printed and the override will be ignored.
There is no reason for this function, tokio already blocks automatically until all tasks are ended.
Another reason to remove this feature is `mpsc_background_tasks` unbounded channel. Recently this
channel was reporting too many unprocessed elements. We assume that this was a result of a lot of
very shot lived tasks that somehow flooded this channel.
* Offchain worker: Enable http2 and improve logging
Apparently some webpages now return http2 by default and that silently breaks the offchain http
extension. The solution to this is to enable the `http2` feature of hyper. Besides that, this pr
improves the logging to make it easier to debug such errors.
* FMT
* Adds http2 test
* authority-discovery: Support multiple authority ids per peer id
An peer id can be mapped to multiple authority ids, because an authority id is a session key that
could be changed every session. Before this pr the internal authority discovery cache assumed that
each authority id can only be mapped to one peer id. However, this isn't true since we changed the
default implementation of the authority discovery to combine the current and next session authorities.
* Review feedback
* Update client/authority-discovery/src/worker/addr_cache.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Early return on no peer ids
* Update client/authority-discovery/src/worker/addr_cache.rs
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Update types in comment
* FMT
* Add warning
* Update client/authority-discovery/src/worker/addr_cache.rs
Co-authored-by: Andronik Ordian <write@reusable.software>
* Feedback
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
* Cli: Introduce `--detailed-log-output` flag
If this CLI flag is given, detailed log output will be enabled. This includes the log level, log
target ad the thread name. Before this was only enabled when a log level higher than `info` should
be logged.
* Update client/tracing/src/logging/mod.rs
Co-authored-by: David <dvdplm@gmail.com>
Co-authored-by: David <dvdplm@gmail.com>
* Offchain-worker: Make it possible to disable http support
If a chain doesn't require http support in its offchain workers, this pr enables them to disable the
http support.
* Switch to bitflags
* Use Capabilities
* Update client/offchain/src/lib.rs
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Fix test
* Update client/offchain/src/lib.rs
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* pallet-utility: Fix possible mismatch between native/wasm
The `batched_calls_limit` constant value includes the `size_of` of the runtime `Call`. As we compile
the runtime for native/wasm, we need to align the call size to ensure that it is the same on
wasm/native. This also solves the problem of different metadata outputs for the same runtime.
* Review feedback
* pallet-scheduler: Introduce `OriginPrivilegeCmp`
When a scheduled task should be canceled, the origin that tries to
cancel the task is compared to the origin the task should be executed
with. Before this pr this check only allowed that both origins are
equal. However, this is problematic as this means that for example a
council origin it needs to be have the same amount of yes votes to
cancel the scheduled task. While a council origin with more yes votes
should be able to cancel this task. This happened recently on Kusama and
lead to a failed cancelation of a scheduled task. With this pr the two
origins are compared and the cancelling origin needs to have greater or
equal privileges as the origin that scheduled the task. What a greater,
equal or less privilege is, can be configured in the runtime.
For simplicity, a `EqualPrivilegeOnly` implementation is provided that
only checks if two origins are equal. So, this mimics the old behaviour.
* FMT
* fix import
* Small optimizations
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
* pallet-multisig: Improve opaque call handling
Before the opaque call was just a type redefinition of `Vec<u8>`. With metadata v14 that was
breaking external tools, as they stopped looking at the type name. To improve the situation the
`WrapperKeepOpaque` type is introduced that communicates to the outside the correct type info.
* Cleanup
* Fix benchmarks
* FMT
It is possible that `Instant::now()` is returning an earlier clock time when being called a second
time. To guard against this, we should use `saturating_duration_since`.
The babe `authoring_blocks` test was flaky because it could happen that
we imported the first block multiple times. This lead to import the
genesis epoch multiples times. `EpochChanges` was assuming that
importing a genesis epoch while there was already an imported epoch
means that there is a "gap". However, this wasn't true as we just
imported 2 genesis epochs. The bug is solved by checking that the
already imported epochs are not all genesis epochs.
* sp-debug-derive: Add `force-debug` feature
Sometimes users want to have proper debug information even inside the
wasm runtime, when they use `RuntimeDebug`. To not fork Substrate to
change a type to a normal `Debug` derive, this pr introduces the
`force-debug` feature. This feature can be enabled by the user in their
runtime and then the types will use the normal Debug derive instead of
printing `wasm:stripped`.
* Fixes
Bastian Köcher