Add changelog generation (#761)

* WIP Release notes generation and templates

* WIP Add new sections to the template

* WIP renaming and wip

* Fix runtime template

* Add doc, NO_CACHE and tweaking of the templates

* Renaming cl into cumulus to make room for the polkadot and substrate

* Fetch data from Substrate and Polkadot

* WIP convert bash script to ruby

* Convert to Ruby

* Fix host function delection

* Extract priority to a macro

* Fix misc changes

* Draft release workflow

* Fix runtime dir

* Add ENV to ignore runtimes

* Install tooling separately

* WIP troubleshooting - remove sudo

* Minor formatting fixes

* Fix workflow

* Add missing dep

* Linting

* Fix changelog script

* Add missing tera install

* Use absolute paths

* Fix path + cleanup

* Fix changelog generation

* Add missing pre-release ENV

* Fix rust version ENV

* Fix release notes path

* Fix output

* Fix runtime_dir for cumulus

* Fix ENV substitutions

* Fix styling

* Debugging

* Styling

* Fix call to fetch the runtime version

* Cleanup and doc

* Delete sample .env

* Update scripts/changelog/templates/change.md.tera

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

* Change XCM emoji marker for a ✉️

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

cumulus/scripts/changelog/bin/changelog

@@ -0,0 +1,116 @@
+#!/usr/bin/env ruby
+
+# frozen_string_literal: true
+
+# call for instance as:
+# ./bin/changelog statemine-v5.0.0
+#
+# You may set the ENV NO_CACHE to force fetching from Github
+# You should also ensure you set the ENV: GITHUB_TOKEN
+
+require_relative '../lib/changelog'
+require 'logger'
+
+logger = Logger.new($stdout)
+logger.level = Logger::DEBUG
+logger.debug('Starting')
+
+owner = 'paritytech'
+repo = 'cumulus'
+ref1 = ARGV[0]
+ref2 = ARGV[1] || 'HEAD'
+output = ARGV[2] || 'release-notes.md'
+
+ENV['REF1'] = ref1
+ENV['REF2'] = ref2
+
+gh_cumulus = SubRef.new(format('%<owner>s/%<repo>s', { owner: owner, repo: repo }))
+
+polkadot_ref1 = gh_cumulus.get_dependency_reference(ref1, 'polkadot-client')
+polkadot_ref2 = gh_cumulus.get_dependency_reference(ref2, 'polkadot-client')
+
+substrate_ref1 = gh_cumulus.get_dependency_reference(ref1, 'sp-io')
+substrate_ref2 = gh_cumulus.get_dependency_reference(ref2, 'sp-io')
+
+logger.debug("Polkadot from:   #{polkadot_ref1}")
+logger.debug("Polkadot to:     #{polkadot_ref2}")
+
+logger.debug("Substrate from:  #{substrate_ref1}")
+logger.debug("Substrate to:    #{substrate_ref2}")
+
+cumulus_data = 'cumulus.json'
+substrate_data = 'substrate.json'
+polkadot_data = 'polkadot.json'
+
+logger.debug("Using CUMULUS:   #{cumulus_data}")
+logger.debug("Using SUBSTRATE: #{substrate_data}")
+logger.debug("Using POLKADOT:  #{polkadot_data}")
+
+logger.warn('NO_CACHE set') if ENV['NO_CACHE']
+
+# This is acting as cache so we don't spend time querying while testing
+if ENV['NO_CACHE'] || !File.file?(cumulus_data)
+  logger.debug(format('Fetching data for Cumulus into %s', cumulus_data))
+  cmd = format('changelogerator %<owner>s/%<repo>s -f %<from>s -t %<to>s > %<output>s',
+               { owner: owner, repo: repo, from: ref1, to: ref2, output: cumulus_data })
+  system(cmd)
+else
+  logger.debug("Re-using:#{cumulus_data}")
+end
+
+if ENV['NO_CACHE'] || !File.file?(polkadot_data)
+  logger.debug(format('Fetching data for Polkadot into %s', polkadot_data))
+  cmd = format('changelogerator %<owner>s/%<repo>s -f %<from>s -t %<to>s > %<output>s',
+               { owner: owner, repo: 'polkadot', from: polkadot_ref1, to: polkadot_ref2, output: polkadot_data })
+  system(cmd)
+else
+  logger.debug("Re-using:#{polkadot_data}")
+end
+
+if ENV['NO_CACHE'] || !File.file?(substrate_data)
+  logger.debug(format('Fetching data for Substrate into %s', substrate_data))
+  cmd = format('changelogerator %<owner>s/%<repo>s -f %<from>s -t %<to>s > %<output>s',
+               { owner: owner, repo: 'substrate', from: substrate_ref1, to: substrate_ref2, output: substrate_data })
+  system(cmd)
+else
+  logger.debug("Re-using:#{substrate_data}")
+end
+
+SHELL_DIGEST = ENV['SHELL_DIGEST'] || 'digests/shell-srtool-digest.json'
+WESTMINT_DIGEST = ENV['WESTMINT_DIGEST'] || 'digests/westmint-srtool-digest.json'
+STATEMINE_DIGEST = ENV['STATEMINE_DIGEST'] || 'digests/statemine-srtool-digest.json'
+STATEMINT_DIGEST = ENV['STATEMINT_DIGEST'] || 'digests/rococo-srtool-digest.json'
+ROCOCO_DIGEST = ENV['ROCOCO_DIGEST'] || 'digests/rococo-srtool-digest.json'
+
+# Here we compose all the pieces together into one
+# single big json file.
+cmd = format('jq \
+    --slurpfile cumulus %s \
+    --slurpfile substrate %s \
+    --slurpfile polkadot %s \
+    --slurpfile srtool_shell %s \
+    --slurpfile srtool_westmint %s \
+    --slurpfile srtool_statemine %s \
+    --slurpfile srtool_statemint %s \
+    --slurpfile srtool_rococo %s \
+    -n \'{
+            cumulus: $cumulus[0],
+            substrate: $substrate[0],
+            polkadot: $polkadot[0],
+        srtool: [
+        { name: "rococo", data: $srtool_rococo[0] },
+        { name: "shell", data: $srtool_shell[0] },
+        { name: "westmint", data: $srtool_westmint[0] },
+        { name: "statemint", data: $srtool_statemint[0] },
+        { name: "statemine", data: $srtool_statemine[0] }
+    ] }\' > context.json', cumulus_data, substrate_data, polkadot_data,
+             SHELL_DIGEST,
+             WESTMINT_DIGEST,
+             STATEMINE_DIGEST,
+             STATEMINT_DIGEST,
+             ROCOCO_DIGEST)
+system(cmd)
+
+cmd = format('tera --env --env-key env --include-path templates \
+        --template templates/template.md.tera context.json > %s', output)
+system(cmd)