Commit Graph

861 Commits

Author SHA1 Message Date
Pierre Krieger 22ec13cf65 Rewrite the network TestNet (#3016)
* Add a memory-only option for the network

* Tests cleanup

* Make grandpa/aura/babe compile

* Aura and Babe tests now passing

* More work on tests rewrite

* Attempt to fix grandpa

* Make more grandpa tests pass

* More grandpa tests work

* Work on sync tests

* More work

* light_peer_imports_header_from_announce passes

* can_sync_small_non_best_forks passes

* syncing_node_not_major_syncing_when_disconnected passes

* blocks_are_not_announced_by_light_nodes passing

* All sync tests passing 🎉

* Some TestNet cleanup

* Work on grandpa tests

* More grandpa work

* GrandPa work

* Add check about block_import

* Remove the temporarily added Sync

* Fix network tests warnings

* voter_persists_its_votes passes

* Fix imports in network tests

* Fix service tests

* Call on_block_imported 🤷

* Add shortcut

* Finish using shortcut
2019-07-05 11:19:03 +01:00
Pierre Krieger abf33fe479 Use bip39 from crates.io (#3029) 2019-07-05 10:11:58 +02:00
Pierre Krieger 8bca52128f Adjust the RPC service for the WASM-browser use case (#3013)
* Use SpawnTaskHandle to pass to the RPC

* Create the RPC server in lib.rs

* Create the RPC servers in a separate function

* Keep a local version of the RPC handlers

* Make rpc-servers compile for WASM

* Add RpcSesssion

* Clean up

* Address review

* Address pull request review
2019-07-05 09:55:50 +02:00
André Silva 13164304b3 grandpa: make observer relay round messages (#3021)
* grandpa: make observer relay round messages

* grandpa: observer also notes next round

* grandpa: no need to note commit round
2019-07-05 09:55:10 +02:00
Pierre Krieger a199a989b8 on_start now returns the precise elements to request (#3003)
* on_start now returns the precise elements to request

* Fix test
2019-07-05 09:40:26 +02:00
André Silva d5bc7325b9 grandpa: round catchup messages (#2801)
* grandpa: initial structure for catch up messages

* grandpa: answer catch up requests

* grandpa: inject catch up messages into global stream

* grandpa: keep track of pending catch up request

* grandpa: block catchup until all referenced blocks are imported

* grandpa: unify catch up and commit streams

* grandpa: simplify communication stream/sink types

* grandpa: note gossip validator on catch up message import

* grandpa: fix cost on catch up message validation

* grandpa: check signatures on catch up messages

* grandpa: clean up catch up request handling state

* grandpa: adjust costs on invalid catch up requests

* grandpa: release lock before pushing catch up message

* grandpa: validate catch up request against peer view

* grandpa: catch up docs

* grandpa: fix tests

* grandpa: until_imported: add tests for catch up messages

* grandpa: add tests for catch up message gossip validation

* grandpa: integrate HistoricalVotes changes

* grandpa: add test for neighbor packet triggering catch up

* grandpa: add test for full voter catch up

* grandpa: depend on finality-grandpa 0.8 from crates

* granda: use finality-grandpa test helpers

* grandpa: add PSM cost for answering catch up requests

* grandpa: code style fixes

Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>

* grandpa: more trailing commas

* grandpa: lower cost of invalid catch up requests near set change

* grandpa: process catch up sending on import of neighbor message

* grandpa: add comments on HistoricalVotes

* grandpa: use finality-grandpa v0.8.1 from crates.io

* grandpa: fix test compilation
2019-07-04 20:40:16 +01:00
Bastian Köcher 7a2165cff2 Make wasm-builder less spammy (#3020)
* Don't clutter the output that much

* Support building from `crates` or from `path`

* Upgrade the versions

* Update `Cargo.lock`
2019-07-04 17:57:21 +02:00
cheme fb8bb00158 Update trie dependencies. (#3017)
* Update trie dependencies.

* bump rutime impl version.
2019-07-04 15:56:13 +02:00
David Craven 51e345c901 Generic keystore (#3008)
* Add KeyTypeId.

* Implement clone for sr25519::Pair.

* Extend Pair with to_raw_vec.

* Implement TypedKey for Signature and Pair.

* Add trait Public.

* Make keystore generic.

* Fixup clone.

* Fix tests.

* Update service.

* Fix imports.

* Fix build.

* Fix babe build.

* Fix subkey build.

* Make authority setup generic.

* Update node-template.

* Fix build.

* Remove unsafe code.

* Fix tests.
2019-07-04 13:14:55 +02:00
Bastian Köcher fe08221479 Build WASM binaries as part of cargo build (#2868)
* Introduce `wasm-builder` and `wasm-builder-runner` to retire `build.sh`

Make use of `wasm-builder` in `test-runtime`.

* Add build script and remove the wasm project

* Port `node-runtime` to new wasm-builder

* Make `substrate-executor` tests work with `wasm-builder`

* Move `node-template` to `wasm-builder`

* Remove `build.sh` :)

* Remove the last include_bytes

* Adds the missing build.rs files

* Remove `build.sh` from CI

* Debug CI

* Make it work in CI

* CI attempt 3

* Make `substrate-runtime-test` compile on stable

* Ahhh, some missed `include_bytes!`

* AHH

* Add suggestions

* Improve search for `Cargo.lock` and don't panic if it is not found

* Searching from manifest path was no good idea

* Make the `wasm-builder` source better configurable

* Expose the bloaty wasm binary as well

* Make sure to rerun WASM recompilation on changes in dependencies

* Introduce new `WASM_BUILD_TYPE` env and make sure to call `build.rs` on
changes to env variables

* Remove `build.sh` from READMEs

* Rename the projects

* Fixes CI

* Update lock file

* Fixes merge-conflict

* Apply suggestions from code review

Co-Authored-By: TriplEight <denis.pisarev@parity.io>

* Try to make windows happy

* Replace all back slashes in paths with slashes

* Apply suggestions from code review

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

* Use cargo from `CARGO` env variable

* Fix compilation

* Use `rustup` for running the nightly build

* Make individual projects skipable

* Fix compilation

* Fixes compilation

* Build all WASM projects in one workspace

* Replace more back slashes!

* Remove `inlcude_bytes!`

* Adds some documentation

* Apply suggestions from code review

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Apply suggestions from code review

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* More review comments

* Update `Cargo.lock`

* Set license

* Apply suggestions from code review

Co-Authored-By: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

* More review comments + adds `TRIGGER_WASM_BUILD` env

* Fix doc tests

* Increase version + update README

* Switch crates.io version of `wasm-builder`

* Update README

* Switch to released version of `wasm-builder-runner`
2019-07-04 11:34:06 +02:00
Pierre Krieger eca9c36b75 Can now disable the keystore (#3004)
* Can now disable the keystore

* Fix service test

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix cli
2019-07-04 10:33:43 +02:00
Bastian Köcher 4fe9e8732f Downgrade memory-db in test-runtime to fix WASM compilation (#3010)
* Downgrade `memory-db` in `test-runtime` to fix WASM compilation

* Remove unneeded `memory-db` dependency

* Update `Cargo.lock`
2019-07-03 21:26:36 +02:00
Pierre Krieger fa535da069 Implement Executor for Service and SpawnHandle (#3007)
* Implement Executor for Service and SpawnHandle

* Update lib.rs

* Fix the race condition
2019-07-03 15:36:49 +02:00
DemiMarie-parity 81d8a5d01d BABE Randomness using PreRuntime digests (#2929)
* Initial work on exposing pre-runtime digests

This provides the primitive API, as well as exposing it from BABE.

* Initial work on using pre-digests in runtimes

This includes both code to expose them from `srml_system`, as well as
using it in (currently dead) code in `srml_babe`.

* Bump `{spec,impl}_version`

* Add `u64_backend` feature to curve25519-dalek

Otherwise, it errors out at compile-time.

* Bump `Cargo.lock`

* Do not depend on the schnorrkel crate in the runtime

The schnorrkel crate does not work on `#![no_std]`, but the runtime only
needs constants from it.  This adds our own definitions of those
constants, and checks them for correctness at compile-time.

* Actually implement storage of VRF outputs

* Trivial formatting change

* Provide a `hash_randomness` function in BABE

for processing VRF outputs.

* Implement a basic randomness generating function

It just XORs the VRF outputs together.

* Actually implement on-chain randomness

Blake2b is used for hashing.

* Update dependencies

* Run `cargo update` where needed

* Re-add a newline at EOF

* Remove broken and unsafe code

XOR is not a hash function, and must not be used as such.  The
implementation was also needlessly unsafe.

* Run `cargo update` where needed

* Remove spurious dependency

* Document security guarantees of BABE randomness

* Add a `RandomnessBeacon` trait

* Document `RandomnessBeacon::random`

* Fix silly compile error (unexpected type arguments)

* Fix BABE randomness

* Implement `FindAuthor` for `babe::Module`

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>
Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>

* Respond to suggestions from code review and fix bugs

* Store an authority index, not the authority itself.
* Avoid unnecessary decoding.
* Implement relative slots and BABE randomness fully and correctly.

* Remove spurious dependency

* Fix error reported by rust-analyzer

* Update Cargo.lock files

* `wrapping_add` → `checked_add`

The epoch index will not overflow.  Panic if it does.

* Move randomness documentation to trait

* Fix compile error in test suite

* Explain 2^64 limit

Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>
2019-07-03 14:49:07 +02:00
Tomasz Drwięga 2217c1e9a1 Persistent Local Storage for offchain workers. (#2894)
* WiP.

* Implement offchain storage APIs.

* Change compare_and_set to return bool.

* Add offchain http test.

* Fix tests.

* Bump spec version.

* Fix warnings and test.

* Fix compilation.

* Remove unused code.

* Introduce Local (fork-aware) and Persistent storage.

* Fix borked merge.

* Prevent warning on depreacated client.backend

* Fix long lines.

* Clean up dependencies.

* Update core/primitives/src/offchain.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Update core/primitives/src/offchain.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>
2019-07-02 20:11:06 +02:00
Gavin Wood 9c6ebfeccd Initial mechanics for 80:20 fee split (#2912)
* Initial mechanics for 80:20 fee split

Also:
- Introduce extra functions for Imbalance manipulation;
- Store treasury pot in an account, letting total issuance account for
  it.

* Fix some tests

* Fix some tests

* Minor cleanups

* Update parity-codec version (#2855)

* Update parity-codec version

* Update grandpa, rhododendron and trie-bench

* Use primitive-types from crates.io

* Bump impl version

* Fix trie-bench version

* Fix lock files

* Fix versions

* Update codec to 4.1

* merge fix

* Revert merge

* More reversions

* Remove accidental code

* Update locks

* Bump runtime

* Update locks

* Tweaks and label TODO

* Update srml/treasury/src/lib.rs

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* Update issue number

* Update core/sr-primitives/src/traits.rs

Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>

* Fix wasm build

* Fix subkey build
2019-07-02 11:41:03 +02:00
Bastian Köcher bfe5347724 Adds a SpawnTaskHandle to core service (#2992)
* Adds a `SpawnTaskHandle` to core service

* Fixes compilation

* Remove `into()`
2019-07-02 10:39:41 +02:00
Weiliang Li 24595c0920 Update without_std.rs (#2990)
fix typo
2019-07-02 08:41:55 +02:00
kaichao 3e2c77aa21 typo fix in core (#2987) 2019-07-01 19:33:38 +02:00
Roman Borschel bd7566ef5e Fix peer discovery delays. (#2982)
* Fix DiscoveryBehaviour::poll.

The previous implementation regularly returned `NotReady` from `poll`
despite the inner Kademlia behaviour having events ready, thus letting
the `poll`ing be largely driven by the task wakeups from the Delay for
the next random Kademlia query, inducing major delays in consuming
the ready Kademlia events and thus slowing progress.

The discovery test now passes quickly, as expected.

* Ensure the Delay is polled right after reset.

For task wakeup.
2019-07-01 12:46:23 +02:00
Niklas Adolfsson 5f1538b834 cargo update -p smallvec (#2983) 2019-07-01 10:04:43 +02:00
Svyatoslav Nikolsky 83e014c214 change meaning of none in blockchain cache (#2973) 2019-06-30 17:54:51 +02:00
Bastian Köcher 18be937cb5 Adds if_std! macro (#2979)
This macro compiles and executes the given code only when `std` feature
is enabled. This can be useful for debugging without needing to worry
that your code does not compile on `no_std`.
2019-06-29 20:12:46 +02:00
Pierre Krieger 7b1e8dc3df Update sysinfo to 0.9 (#2978) 2019-06-29 17:58:12 +02:00
Bastian Köcher 62b7c05def Implement a proper generic resolution in decl_storage! (#2913)
* Add failing test case

* move storage maps to blake2_128 (#2268)

* remove default hash, introduce twox_128 and blake2

* use blake2_128 & create ext_blake2_128

* refactor code

* add benchmark

* factorize generator

* fix

* parameterizable hasher

* some fix

* fix

* fix

* fix

* metadata

* fix

* remove debug print

* map -> blake2_256

* fix test

* fix test

* Apply suggestions from code review

Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>

* impl twox 128 concat (#2353)

* impl twox_128_concat

* comment addressed

* fix

* impl twox_128->64_concat

* fix test

* Fix compilation and cleanup some docs

* Lol

* Remove traits from storage types that are not generic

* Get instance test almost working as wanted

* Make `srml-support-test` compile again :)

* Fixes test of srml-support

* Fix compilation

* Break some lines

* Remove incorrect macro match arm

* Integrates review feedback

* Update documentation

* Fix compilation
2019-06-27 13:40:22 +02:00
Bastian Köcher 23ea5d1795 Remove Mutex from core-service (#2961)
* Remove `Mutex` from `core-service`

* Fixes sync test
2019-06-27 10:07:34 +02:00
Tomasz Drwięga 3f142d0561 Enable hosts filtering. (#2959) 2019-06-26 18:10:31 +02:00
Pierre Krieger 1b73b6532a Remove tokio dependencies (#2935)
* Remove dependencies on tokio

* Make service not depend on tokio

* Fix service tests

* Manually poll the import queue if failed to start

* Spawn all tasks at the end

* Remove executor from TelemetryOnConnect

* Remove TaskExecutor from offchain workers

* Remove TaskExecutor from AuthoritySetup

* Remove TaskExecutor from service

* Remove tokio dependency from RPC

* Remove finality-grandpa from WASM checks

* Fix offchain tests

* Line widths

* Fix RPC tests

* Fix service tests

* Fix bad futures polling

* Address some concerns

* Better error handling

* Is it the connectivity test that's not passing? I don't know, let's try

* Revert "Is it the connectivity test that's not passing? I don't know, let's try"

This reverts commit 28bbe51f0e2e4885fe1f901e11078604604cb212.

* Fix test
2019-06-26 17:21:17 +02:00
cheme f69c48c7b8 Additional state check. (#2915)
* check resulting state root on import.

* get root from last method.

* Fix test runtime (was changing state after root calculation).

* Do reset new authorities (just take before root calculation).

* bump impl runtime version.

* Update core/sr-api-macros/tests/trybuild.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>
2019-06-26 16:28:12 +02:00
Stanislav Tkach 443ad90b47 Update codec version to the 4.1 version (#2948)
* Update codec version to the 4.1 version

* Bump impl_version

* Update lock files

* Update codec to 4.1.1 version

* Bump impl version
2019-06-26 16:26:24 +02:00
Fedor Sakharov e735853ca3 Add DHT key-value behaviour (#2937)
* Add DHT key-value behaviour

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Apply suggestions from code review

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

* Return which key failed to be inserted
2019-06-26 13:30:39 +02:00
Robert Habermeier 0ddf4a2a00 Generic authorship and uncle inclusion module (#2941)
* generalized uncle processing

* add some uncle tests

* set author and do event handling

* OnePerAuthorPerHeight no longer O(n^2) and test

* bump impl_version of node

* Documentation and style fixes

Co-Authored-By: DemiMarie-parity <48690212+DemiMarie-parity@users.noreply.github.com>

* fix #2949: index-based FindAuthor wrapper for srml-session

* use for_each_tuple
2019-06-26 12:24:58 +02:00
Pierre Krieger db71e0cc48 Expose the telemetry external transport in the config (#2939)
* Expose the telemetry external transport in the config

* Fix tests
2019-06-26 09:05:49 +02:00
thiolliere 8b89c7ae62 Update parity-scale codec minor version (#2951)
* update parity-scale codec minor version

* update to 3.5.3

* update locks

* increase impl version

* Update to 3.5.4

* Update WASM lockfiles
2019-06-26 08:48:12 +02:00
Pierre Krieger 846941550d Update to libp2p 0.10 (#2945)
* Update to libp2p 0.10

* Bump zeroize in substrate-network
2019-06-25 15:56:25 +02:00
thiolliere cf7dbe8151 Staking module: approximate fraction into perbill to avoid overflow (#2889)
* approximate fraction into perbill

* test

* fix comment

* line width

* bump impl version

* rename test for better naming

* test overflow

* Apply suggestions from code review

Co-Authored-By: Niklas Adolfsson <niklasadolfsson1@gmail.com>
2019-06-24 13:23:54 +02:00
Pierre Krieger e919d03331 Add ProtocolBehaviour (#2922)
* Add ProtocolBehaviour

* Fix tests

* Line widths

* Address concerns

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove TODO
2019-06-24 13:22:03 +02:00
DemiMarie-parity 48aa32bece Relative slots (#2820)
* Initial work on relative slots for BABE

* More work

* Update core/consensus/babe/src/lib.rs

`Aura` → `Babe`

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

* More work on relative slots

* Add missing field in test-runtime

* Bump `impl_version` and `authoring_version`

* Fix compile errors and warnings

* Upgrade dependencies

* Update dependencies more

* Revert some updates to dependencies

Somehow, those broke the build

* Fix compilation errors

* `Duration` → `u128` in calculations

* `slot_duration` is in milleseconds, not seconds

* Median algorithm: ignore blocks with slot_num < sl

* Fix silly compile error

* Store a duration, rather than an instant

It is more useful

* Fix compilation errors

* `INVERSE_NANO` → `NANOS_PER_SEC`

Also: `1000_000_000` → `1_000_000_000`

Suggested-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

* Un-bump `authoring_version`

* Disable median algorithm when `median_required_blocks` is 0

Otherwise it would panic.

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Simplify panic

* Fix build error

* Create `SignedDuration` struct

for signed `Duration` values.

Suggested-by: Bastian Köcher

* Refactor median algorithm into separate function

* Add issues for FIXMEs and respond to code review

* Fix minor warnings
2019-06-22 12:21:29 -04:00
Pierre Krieger 437a6bc6b1 Move the network status reporting to the service (#2916)
* Move the network status reporting to the service

* Fix tests

* Fix build
2019-06-21 17:13:11 -04:00
Pierre Krieger 01fcdc2b1a Add documentation at the root of substrate-network (#2902)
* Add documentation at the root of substrate-network

* Apply suggestions from code review

Co-Authored-By: Toralf Wittner <tw@dtex.org>

* Grmlblbl

* Adjustments

* Link to Yamux specs
2019-06-21 14:59:57 -04:00
Pierre Krieger f45cfb0d16 Use a background task instead of a thread for the import queue (#2909)
* Use a background task instead of a thread for the import queue

* Update Cargo.locks

* Comment
2019-06-21 09:46:26 -04:00
Pierre Krieger 32a14ba2b8 The network is no longer an Option (#2920) 2019-06-20 16:03:50 +01:00
Pierre Krieger c4877bc05b Remove on_connect from TelemetryConfig (#2888) 2019-06-19 15:41:35 +02:00
Pierre Krieger 80a4cd2b0d Add support for noise (#2613) 2019-06-19 15:36:58 +02:00
Svyatoslav Nikolsky 5030097799 Do not return empty entries from state_queryStorage (#2906)
* do not return empty entries from state_queryStorage

* revert back None -> null change

* Revert "revert back None -> null change"

This reverts commit 318eb043e03b7aabbc3f176e02c15fd4595d16db.
2019-06-19 15:31:14 +02:00
Bastian Köcher 37acb90847 Remove UserError and switch to error::Error (#2899)
* Remove `UserError` and switch to `error::Error`

* More cleanup

* Update core/executor/src/error.rs

Co-Authored-By: Sergei Pepyakin <s.pepyakin@gmail.com>

* Fix typo

* Update core/executor/src/allocator.rs

Co-Authored-By: Stanislav Tkach <stanislav.tkach@gmail.com>
2019-06-19 15:08:59 +02:00
Pierre Krieger 7efc504d59 Import queue API revamp (#2856)
* ImportQueue methods are now mut

* Link methods are now mut

* Remove Arc from BasicSyncQueue

* Fix tests

* Remove BasicSyncQueue

* Change the import queue API

* Add buffered_link

* Remove obsolete tests

* Comments and style improvement pass

* Fix grandpa and comment cleanup

* Update core/consensus/common/src/import_queue.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>
2019-06-18 21:50:48 -04:00
Pierre Krieger de6d541c74 Fix the failing networking tests (#2904) 2019-06-18 21:03:38 -04:00
Fredrik Harrysson 925a90f57f Add documentation to the main sync code (#2879)
* Added some documentation and question on the main sync code

* Remove strange formatting change

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Improve comment per suggestion

* Remove comment about re-using PeerInfo, no real reason and no real consensus if it's better or worse.

* Fix redundancy in comment

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Explaining more things and fixing review comments.

* Better explanation of MAJOR_SYNC_BLOCKS
* Explains the ancestor search strategy a bit
* Corrects explanation of what a stale block is
* Adds a bit of context for `request_ancestry` function

* Revert code change to not confuse the PR
2019-06-18 21:38:13 +01:00
Pierre Krieger 4a44a07fd0 Move libp2p tests to custom_proto (#2884) 2019-06-18 10:36:42 +02:00