Sync the lists with polkadot-js/phishing every day, and publish them with checks (#4)

* feat: sync the lists with polkadot-js/phishing daily, and check them before publishing

The lists had not moved since January 2026, the site that serves them was
copied by hand, and the rebrand had turned five real phishing domains into
pezkuwichain.* names, unblocking them.

- scripts/syncUpstream.mjs and sync-upstream.yml take upstream's lists every
  day, lay pezkuwi/overlay.json over them, and commit to master. A day out of
  bounds (too many sites unblocked or added, scam addresses dropped, one of our
  domains blocked, one of our entries lost) writes nothing and opens an issue.
- pezkuwi/overlay.json holds our own entries with their reasons: the five
  pezkuwichain.* names stay blocked on purpose, and our domains are protected.
- scripts/lists.mjs holds the rules; scripts/lists.test.mjs tests them and the
  sync end to end; scripts/checkLists.mjs checks the committed files, including
  that they are what sortAll writes. lists.yml runs all three.
- pezkuwi/host/ is what phishing.pezkuwichain.io runs: it pulls master hourly
  and refuses a rewritten history, a blocked own domain, a shrink over 2%, or
  shards that disagree, with a sandbox test of each refusal.
- sortAll strips a trailing dot. Upstream's 817 "scam.xyz." entries were filed
  under an empty top-level domain in all/all.json, where checkIfDenied never
  looks; 816 repeated a listed name, and prenads.xyz was blocked nowhere else.
- sortAll uses Node's own directory functions instead of @pezkuwi/dev, so the
  sync and the checks run without installing the build toolchain.
- The auto-approve, auto-merge and crosscheck workflows, and the IPFS job,
  depended on tokens and secrets this repository does not have; removed. The
  remaining workflows are pinned to commits, with no default permissions and
  a timeout on every job.

* chore(lists): first sync with polkadot-js/phishing

Written by scripts/syncUpstream.mjs from upstream bf5e7c6:
658 sites added; nodelinkapp.xyz, org.in and terra-claim.com unblocked by
upstream; the five polkadot-js domains the rebrand had renamed blocked again;
816 duplicate "scam.xyz." entries folded into their names and the stray
all/all.json gone. 54,051 blocked sites, 279 scam addresses.

* fix(lists): commit the .build shard, and refuse a shard git ignores

.gitignore held "build/", which also matched all/build/all.json, the shard of
the .build top-level domain. Every sort wrote it and no commit held it, so the
eight .build scam sites were never published; upstream has the same gap. The
directory is excluded from the ignore rule, the shard is committed, and
checkLists.mjs now fails on any shard git would ignore (with the old rule
restored it does).

Also clears the repository's lint on lists.test.mjs: imports sorted, and
node:test's describe and it calls marked void.

* test(lists): await describe and it, as the repository's lint requires

no-floating-promises requires the promises node:test's describe and it return
to be handled, and no-void forbids handling them with void. Awaited, the 25
cases run as before.

pezkuwi/host/phishing-publish.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=Publish pezkuwichain/phishing master to phishing.pezkuwichain.io
+Documentation=https://github.com/pezkuwichain/phishing/tree/master/pezkuwi
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/phishing-publish
+ProtectSystem=strict
+ReadWritePaths=/var/lib/phishing-publish /var/www/phishing /run/lock
+PrivateTmp=yes
+ProtectHome=yes
+NoNewPrivileges=yes

pezkuwi/host/phishing-publish.sh

@@ -0,0 +1,142 @@
+#!/usr/bin/env bash
+#
+# /usr/local/sbin/phishing-publish: keeps phishing.pezkuwichain.io on the lists in
+# pezkuwichain/phishing master.
+#
+# This is the reviewed copy. It is installed by hand, as root, with its systemd units:
+#
+#   install -m 0755 pezkuwi/host/phishing-publish.sh /usr/local/sbin/phishing-publish
+#   install -m 0644 pezkuwi/host/phishing-publish.service pezkuwi/host/phishing-publish.timer /etc/systemd/system/
+#   systemctl daemon-reload && systemctl enable --now phishing-publish.timer
+#
+# The host pulls and nothing pushes to it, so no key to this host sits anywhere else. The
+# extension and the apps fetch these lists to decide what to block, so the host does not
+# publish whatever master holds. It checks first:
+#
+#   - master descends from the commit published last: no rewritten history;
+#   - every file parses; all.json holds a deny list; each all/<tld>/all.json holds only that
+#     top-level domain, and together they hold exactly the deny list;
+#   - the deny list does not shrink by more than MAX_SHRINK_PERCENT against what is served now;
+#   - no entry blocks one of our own domains (PROTECTED).
+#
+# Then it replaces the served lists in one rsync (--delay-updates) and leaves index.html alone.
+#
+#   exit 0   published, or nothing new
+#   exit 1   refused; the served lists stay exactly as they were
+set -euo pipefail
+
+REPO_URL=https://github.com/pezkuwichain/phishing.git
+STATE_DIR=/var/lib/phishing-publish
+WEB_ROOT=/var/www/phishing
+LOCK_FILE=/run/lock/phishing-publish.lock
+MAX_SHRINK_PERCENT=2
+MIN_DENY=10000
+# Kept here, not read from the repository: this check must hold even if master does not.
+PROTECTED="dks.news pex.mom pex.network pezkiwi.app pezkuwi.com pezkuwichain.io"
+
+export LC_ALL=C GIT_TERMINAL_PROMPT=0
+umask 022
+
+exec 9>"$LOCK_FILE"
+if ! flock -n 9; then
+  echo "another publish is running"
+  exit 0
+fi
+
+MIRROR="$STATE_DIR/repo.git"
+install -d -m 700 "$STATE_DIR"
+if [ ! -d "$MIRROR" ]; then
+  git init -q --bare "$MIRROR"
+fi
+git --git-dir="$MIRROR" fetch -q --no-tags "$REPO_URL" +refs/heads/master:refs/remotes/origin/master
+
+new="$(git --git-dir="$MIRROR" rev-parse --verify refs/remotes/origin/master)"
+old="$(cat "$STATE_DIR/published" 2>/dev/null || true)"
+if [ "$new" = "$old" ]; then
+  exit 0
+fi
+if [ -n "$old" ] && ! git --git-dir="$MIRROR" merge-base --is-ancestor "$old" "$new"; then
+  echo "REFUSED: master ${new:0:12} does not descend from the published ${old:0:12}; its history was rewritten"
+  exit 1
+fi
+
+stage="$(mktemp -d "$STATE_DIR/stage.XXXXXX")"
+trap 'rm -rf -- "$stage"' EXIT
+git --git-dir="$MIRROR" archive "$new" all.json address.json all | tar -x -C "$stage"
+if [ -n "$(find "$stage" ! -type f ! -type d)" ]; then
+  echo "REFUSED: the lists at ${new:0:12} hold something other than files and directories"
+  exit 1
+fi
+
+# shellcheck disable=SC2086 # PROTECTED is a list of names, split on purpose
+python3 - "$stage" "$WEB_ROOT" "$MAX_SHRINK_PERCENT" "$MIN_DENY" $PROTECTED <<'PY'
+import json
+import os
+import sys
+
+stage, web, max_shrink, min_deny, protected = sys.argv[1], sys.argv[2], int(sys.argv[3]), int(sys.argv[4]), sys.argv[5:]
+
+
+def refuse(reason):
+    print(f"REFUSED: {reason}")
+    sys.exit(1)
+
+
+def load(path):
+    try:
+        with open(path, encoding="utf-8") as f:
+            return json.load(f)
+    except (OSError, ValueError) as e:
+        refuse(f"{path} does not parse: {e}")
+
+
+lists = load(os.path.join(stage, "all.json"))
+deny = lists.get("deny") if isinstance(lists, dict) else None
+if not isinstance(deny, list) or not all(isinstance(u, str) for u in deny):
+    refuse("all.json holds no deny list of names")
+if len(deny) < min_deny:
+    refuse(f"all.json blocks {len(deny)} sites, fewer than {min_deny}")
+
+address = load(os.path.join(stage, "address.json"))
+if not isinstance(address, dict) or not all(isinstance(v, list) for v in address.values()):
+    refuse("address.json is not a map of sites to addresses")
+
+# An entry ending in a dot ("scam.xyz.") has an empty top-level domain, so no shard below can hold
+# it and the shards-match check refuses it: the package would never find it in all/<tld>/all.json.
+sharded = []
+for top in sorted(os.listdir(os.path.join(stage, "all"))):
+    if not os.path.isdir(os.path.join(stage, "all", top)):
+        refuse(f"all/{top} is not a top-level domain directory")
+    shard = load(os.path.join(stage, "all", top, "all.json"))
+    if not isinstance(shard, list) or any(not isinstance(u, str) or u.rsplit(".", 1)[-1] != top for u in shard):
+        refuse(f"all/{top}/all.json holds entries of another top-level domain")
+    sharded += shard
+if sorted(sharded) != sorted(deny):
+    refuse("all/ and all.json disagree")
+
+served = os.path.join(web, "all.json")
+if os.path.exists(served):
+    was = len(load(served).get("deny", []))
+    if len(deny) * 100 < was * (100 - max_shrink):
+        refuse(f"the deny list would shrink from {was} to {len(deny)} sites, more than {max_shrink}%")
+
+
+def parts(host):
+    return host.lower().rstrip(".").split(".")[::-1]
+
+
+# The package's rule: an entry blocks a host when its labels match the host's from the right.
+for domain in protected:
+    d = parts(domain)
+    for entry in deny:
+        e = parts(entry)
+        if len(e) <= len(d) and d[:len(e)] == e:
+            refuse(f"the entry {entry} would block our domain {domain}")
+
+print(f"checked: {len(deny)} blocked sites, {sum(len(v) for v in address.values())} scam addresses")
+PY
+
+rsync -a --checksum --delete --delay-updates --chown=root:root --exclude index.html "$stage/" "$WEB_ROOT/"
+printf '%s\n' "$new" >"$STATE_DIR/published.new"
+mv -f "$STATE_DIR/published.new" "$STATE_DIR/published"
+echo "published ${new:0:12}"

pezkuwi/host/phishing-publish.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Publish the phishing lists every hour
+
+[Timer]
+OnCalendar=hourly
+RandomizedDelaySec=5min
+Persistent=true
+
+[Install]
+WantedBy=timers.target

pezkuwi/host/test-publish.sh

@@ -0,0 +1,124 @@
+#!/usr/bin/env bash
+# pezkuwi/host/test-publish.sh [lists directory]
+#
+# phishing-publish.sh against a scratch repository and web root: it publishes good lists, leaves
+# index.html alone, and refuses, serving exactly what it served before, when master's history
+# was rewritten, a protected domain would be blocked, the deny list shrinks too far, an entry
+# ends in a dot, or all/ holds something other than top-level domain directories.
+#
+# The lists directory (default: the repository root) must hold publishable lists: all.json,
+# address.json, all/. Nothing here touches a network or a real web root.
+set -uo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+LISTS="$(cd "${1:-$HERE/../..}" && pwd)"
+SB="$(mktemp -d "${TMPDIR:-/tmp}/phishing-publish-test.XXXXXX")"
+trap 'rm -rf "$SB"' EXIT
+PASS=0
+FAIL=0
+
+check() {
+  if [ "$2" = "$3" ]; then printf '  PASS  %-58s %s\n' "$1" "$3"; PASS=$((PASS + 1))
+  else printf '  FAIL  %-58s expected [%s] got [%s]\n' "$1" "$2" "$3"; FAIL=$((FAIL + 1)); fi
+}
+g() { git -C "$SB/work" -c user.email=t@example.invalid -c user.name=t "$@"; }
+served() { (cd "$SB/www" && find . -type f -print0 | LC_ALL=C sort -z | xargs -0 sha256sum) | sha256sum | cut -c1-16; }
+publish() { "$SB/publish.sh" >"$SB/out" 2>&1; echo $?; }
+# edit <message> <python applied to the lists in the work tree>
+edit() {
+  (cd "$SB/work" && python3 -c "$2")
+  g add -A
+  g commit -qm "$1"
+  g push -q origin HEAD:master
+}
+
+sed -e "s#^REPO_URL=.*#REPO_URL=$SB/remote.git#" \
+    -e "s#^STATE_DIR=.*#STATE_DIR=$SB/state#" \
+    -e "s#^WEB_ROOT=.*#WEB_ROOT=$SB/www#" \
+    -e "s#^LOCK_FILE=.*#LOCK_FILE=$SB/publish.lock#" \
+    -e "s#--chown=root:root ##" \
+    "$HERE/phishing-publish.sh" >"$SB/publish.sh"
+chmod +x "$SB/publish.sh"
+if grep -nE '^(REPO_URL=https|STATE_DIR=/var|WEB_ROOT=/var|LOCK_FILE=/run)' "$SB/publish.sh"; then
+  echo "HARNESS: a production path survived the rewrite; refusing to run" >&2
+  exit 2
+fi
+
+git init -q --bare -b master "$SB/remote.git"
+git init -q -b master "$SB/work"
+g remote add origin "$SB/remote.git"
+cp -r "$LISTS/all.json" "$LISTS/address.json" "$LISTS/all" "$SB/work/"
+g add -A && g commit -qm lists && g push -q origin HEAD:master
+# Every later case pushes to this remote. If the first push did not land, each would fail on git
+# and the "served lists did not change" checks would pass for that reason alone.
+if [ "$(git --git-dir="$SB/remote.git" rev-parse --verify -q refs/heads/master)" != "$(g rev-parse HEAD)" ]; then
+  echo "HARNESS: the scratch remote did not receive master; no case ran" >&2
+  exit 2
+fi
+
+# The web root as the host had it: older lists, a stray file the sync no longer writes, the page.
+mkdir -p "$SB/www/all/com"
+python3 - "$LISTS/all.json" "$SB/www" <<'PY'
+import json, sys
+deny = json.load(open(sys.argv[1]))["deny"]
+older = deny[: len(deny) - len(deny) // 100]
+json.dump({"allow": [], "deny": older, "denySub": []}, open(f"{sys.argv[2]}/all.json", "w"))
+PY
+printf '{}\n' >"$SB/www/address.json"
+printf '[]\n' >"$SB/www/all/all.json"
+printf '<html>page</html>\n' >"$SB/www/index.html"
+INDEX="$(sha256sum "$SB/www/index.html")"
+
+echo "== publishes good lists"
+check "exit" 0 "$(publish)"
+check "all.json is master's" "$(sha256sum <"$LISTS/all.json")" "$(sha256sum <"$SB/www/all.json")"
+check "a shard is master's" "$(sha256sum <"$LISTS/all/com/all.json")" "$(sha256sum <"$SB/www/all/com/all.json")"
+check "the stray all/all.json is gone" no "$([ -e "$SB/www/all/all.json" ] && echo yes || echo no)"
+check "index.html untouched" "$INDEX" "$(sha256sum "$SB/www/index.html")"
+check "the published commit is recorded" "$(git -C "$SB/work" rev-parse HEAD)" "$(cat "$SB/state/published")"
+check "nothing new: exit" 0 "$(publish)"
+check "nothing new: silent" "" "$(cat "$SB/out")"
+
+refused() { # refused <label> <python> <text the refusal must give>
+  local before rc
+  before="$(served)"
+  edit "$1" "$2"
+  rc="$(publish)"
+  check "$1: refused" "1 REFUSED" "$rc $(grep -o '^REFUSED' "$SB/out")"
+  # For the reason too: checks overlap, and a refusal from the wrong one would hide that the right
+  # one no longer works.
+  check "$1: for its reason" yes "$(grep -qF -- "$3" "$SB/out" && echo yes || echo no)"
+  check "$1: the served lists did not change" "$before" "$(served)"
+  g reset -q --hard HEAD~1
+  g push -q -f origin HEAD:master
+  rm -rf "$SB/state/repo.git"
+}
+
+L='import json, os; a = json.load(open("all.json"))'
+W='; json.dump(a, open("all.json", "w"))'
+refused "blocks a protected domain" "$L; a['deny'].append('pezkuwichain.io'); os.makedirs('all/io', exist_ok=True); s = json.load(open('all/io/all.json')) if os.path.exists('all/io/all.json') else []; json.dump(s + ['pezkuwichain.io'], open('all/io/all.json', 'w'))$W" "would block our domain pezkuwichain.io"
+refused "shrinks more than the limit" "$L; keep = set(a['deny'][: len(a['deny']) // 2]); a['deny'] = [u for u in a['deny'] if u in keep]$W
+for t in os.listdir('all'):
+    json.dump([u for u in json.load(open(f'all/{t}/all.json')) if u in keep], open(f'all/{t}/all.json', 'w'))" "the deny list would shrink"
+refused "an entry ends in a dot" "$L; a['deny'].append('scam-test.xyz.')$W" "all/ and all.json disagree"
+refused "a file in all/" "open('all/all.json', 'w').write('[]')" "is not a top-level domain directory"
+refused "shards disagree with all.json" "import json; s = json.load(open('all/com/all.json')); json.dump(s[1:], open('all/com/all.json', 'w'))" "all/ and all.json disagree"
+
+echo "== rewritten history"
+edit "one more" "import json; a = json.load(open('address.json')); a['next.example'] = []; json.dump(a, open('address.json', 'w'))"
+g commit -q --allow-empty -m "published next"
+g push -q origin HEAD:master
+[ "$(g rev-list --count HEAD)" = 3 ] || { echo "HARNESS: expected three commits before rewriting history" >&2; exit 2; }
+check "a new commit publishes" 0 "$(publish)"
+published="$(served)"
+g reset -q --hard HEAD~2
+# A different list on the rewritten branch, so publishing it would change what is served.
+(cd "$SB/work" && python3 -c 'import json; a = json.load(open("address.json")); a["rewritten.example"] = []; json.dump(a, open("address.json", "w"))')
+g commit -qam "rewritten"
+g push -q -f origin HEAD:master
+check "a non-descendant master: refused" "1 REFUSED" "$(publish) $(grep -o '^REFUSED' "$SB/out")"
+check "a non-descendant master: served lists unchanged" "$published" "$(served)"
+
+echo
+echo "RESULT: $PASS passed, $FAIL failed"
+[ "$FAIL" -eq 0 ]

pezkuwi/overlay.json

@@ -0,0 +1,38 @@
+{
+	"protect": [
+		"dks.news",
+		"pex.mom",
+		"pex.network",
+		"pezkiwi.app",
+		"pezkuwi.com",
+		"pezkuwichain.io"
+	],
+	"deny": [
+		{
+			"date": "2026-09-13",
+			"reason": "Imitates the PezkuwiChain name and is not ours; blocked before anyone can use it (Serok, 2026-09-13).",
+			"url": "pezkuwichain.com"
+		},
+		{
+			"date": "2026-09-13",
+			"reason": "Imitates the PezkuwiChain name and is not ours; blocked before anyone can use it (Serok, 2026-09-13).",
+			"url": "pezkuwichain.online"
+		},
+		{
+			"date": "2026-09-13",
+			"reason": "Imitates the PezkuwiChain name and is not ours; blocked before anyone can use it (Serok, 2026-09-13).",
+			"url": "pezkuwichain.org"
+		},
+		{
+			"date": "2026-09-13",
+			"reason": "Imitates the PezkuwiChain name and is not ours; blocked before anyone can use it (Serok, 2026-09-13).",
+			"url": "pezkuwichain.site"
+		},
+		{
+			"date": "2026-09-13",
+			"reason": "Imitates the PezkuwiChain name and is not ours; blocked before anyone can use it (Serok, 2026-09-13).",
+			"url": "pezkuwichainweb.org"
+		}
+	],
+	"undeny": []
+}

pezkuwi/upstream.json

@@ -0,0 +1,3 @@
+{
+	"commit": "bf5e7c642701e548f5d6c722a3f81fc086c11abe"
+}