Request based PoV distribution (#2640)

* Indentation fix.

* Prepare request-response for PoV fetching.

* Drop old PoV distribution.

* WIP: Fetch PoV directly from backing.

* Backing compiles.

* Runtime access and connection management for PoV distribution.

* Get rid of seemingly dead code.

* Implement PoV fetching.

Backing does not yet use it.

* Don't send `ConnectToValidators` for empty list.

* Even better - no need to check over and over again.

* PoV fetching implemented.

+ Typechecks
+ Should work

Missing:

- Guide
- Tests
- Do fallback fetching in case fetching from seconding validator fails.

* Check PoV hash upon reception.

* Implement retry of PoV fetching in backing.

* Avoid pointless validation spawning.

* Add jaeger span to pov requesting.

* Add back tracing.

* Review remarks.

* Whitespace.

* Whitespace again.

* Cleanup + fix tests.

* Log to log target in overseer.

* Fix more tests.

* Don't fail if group cannot be found.

* Simple test for PoV fetcher.

* Handle missing group membership better.

* Add test for retry functionality.

* Fix flaky test.

* Spaces again.

* Guide updates.

* Spaces.

polkadot/node/network/availability-distribution/src/metrics.rs

@@ -24,7 +24,7 @@ pub const SUCCEEDED: &'static str = "succeeded";
 /// Label for fail counters.
 pub const FAILED: &'static str = "failed";
 
-/// Label for chunks that could not be served, because they were not available.
+/// Label for chunks/PoVs that could not be served, because they were not available.
 pub const NOT_FOUND: &'static str = "not-found";
 
 /// Availability Distribution metrics.
@@ -47,6 +47,12 @@ struct MetricsInner {
 	/// to a chunk request. This includes `NoSuchChunk` responses.
 	served_chunks: CounterVec<U64>,
 
+	/// Number of PoVs served.
+	///
+	/// Note: Right now, `Succeeded` gets incremented whenever we were able to successfully respond
+	/// to a PoV request. This includes `NoSuchPoV` responses.
+	served_povs: CounterVec<U64>,
+
 	/// Number of times our first set of validators did not provide the needed chunk and we had to
 	/// query further validators.
 	retries: Counter<U64>,
@@ -66,12 +72,19 @@ impl Metrics {
 	}
 
 	/// Increment counter on served chunks.
-	pub fn on_served(&self, label: &'static str) {
+	pub fn on_served_chunk(&self, label: &'static str) {
 		if let Some(metrics) = &self.0 {
 			metrics.served_chunks.with_label_values(&[label]).inc()
 		}
 	}
 
+	/// Increment counter on served PoVs.
+	pub fn on_served_pov(&self, label: &'static str) {
+		if let Some(metrics) = &self.0 {
+			metrics.served_povs.with_label_values(&[label]).inc()
+		}
+	}
+
 	/// Increment retry counter.
 	pub fn on_retry(&self) {
 		if let Some(metrics) = &self.0 {
@@ -103,6 +116,16 @@ impl metrics::Metrics for Metrics {
 				)?,
 				registry,
 			)?,
+			served_povs: prometheus::register(
+				CounterVec::new(
+					Opts::new(
+						"parachain_served_povs_total",
+						"Total number of povs served by this backer.",
+					),
+					&["success"]
+				)?,
+				registry,
+			)?,
 			retries: prometheus::register(
 				Counter::new(
 					"parachain_fetch_retries_total",