update backend to make regeneration easier...

Add the complete option to complete and finalize the indexes after
a larger -m run...  I've been rebuilding the indexes a lot as I
tweek the format so this has become necessary...

NOTES.txt

@@ -1,6 +1,11 @@
 Fetch all of the archive:
 scp -r freefall:/local/mail/archive/*/freebsd-snapshots .
 
+Expand all the "validly" formed messages to a file:
+zcat $(zgrep -l 'PGP SIGNED MESSAGE' *) > ../complete.txt
+
+We only lose 2012, and we'll loose everything before 2015 later.
+
 XXX - don't use the following, doesn't decode body properly
 Explode the mbox to files:
 cat 201810* freebsd-snapshots | split -p '^From ' - snap.
@@ -9,12 +14,11 @@ The stdin trick is needed so the numbering/lettering continues properly.
 Correct way to split the bodies off.
 python splitbody.py arch/complete.txt arch/snap.
 
-Expand all the "validly" formed messages to a file:
-zcat $(zgrep -l 'PGP SIGNED MESSAGE' *) > ../complete.txt
-
-We only lose 2012, so no big deal.
+Bulk import, -m is used to not check if they exist till the end:
+for i in arch/snap.????; do sh addinfo.sh -m $i; done
 
-Bulk import:
-for i in snap.??; do sh addinfo.sh -m < $i; done
+Clean things up and skip checking for ones before given date
+sh addinfo.sh -c 20180900
 
--m is used to not check if they exist till the end.
+Copy them up to freefall:
+scp snapshot.* freefall:public_html/FreeBSD-snap

README.md

@@ -9,6 +9,27 @@ file.
 The only file needed for this is the snapaid.sh script.  The other files
 are used for generating the index.
 
+NOTE: The xz vs non-xz versions of some of the images are not able to be
+differentiated.  Currently shorting rules should always put the xz version
+before the non-xz version.
+
+NOTE: Only snapshots that have SHA512 hashes are included.  This excludes
+most snapshots from 2015 and before.  The tool could be updated to include
+SHA256, but not a priority currently.
+
 NOTE: Not all of the snapshots are in the database.  Some snapshot names,
 like 11.0-RC1, don't contain all the info others do, and are not
 included.  In the future, hopefully this will be fixed.
+
+backend
+-------
+
+The backend is just a simple text file the indexes all the published
+snapshots.  It is built from the emails to the freebsd-snapshot
+list.  After verification of the email's signature, the SHA512 entry
+lines are extracted, the file name is parsed, and added to the complete
+index.  The message-id of the email is in the index so that the frontend
+can d/l the original email, verify the GPG signature locally.  The
+complete index is used for verifying a snapshot that has already been
+downloaded.  Another index is also maintained which only contains the
+currently available to d/l snapshots.

addinfo.sh

@@ -2,14 +2,28 @@
 
 set -e
 
-args=`getopt m $*`
+usage() {
+	echo 'Usage: $0 [ -m ] <file>'
+	echo 'Usage: $0 -c <date>'
+	echo ''
+	echo 'date is specified as YYYYMMDD'
+
+	if [ x"$1" != x"" ]; then
+		exit $1
+	fi
+}
+
+args=`getopt cm $*`
 if [ $? -ne 0 ]; then
-	echo 'Usage: $0 [ -m ]'
-	exit 2
+	usage 2
 fi
 set -- $args
 while :; do
 	case "$1" in
+	-c)
+		complete=1
+		shift
+		;;
 	-m)
 		more=1
 		shift
@@ -19,9 +33,31 @@ while :; do
 		;;
 	esac
 done
+if [ x"$complete" = x"1" -a x"$more" = x"1" ]; then
+	echo '-m and -c cannot be specified at the same time.'
+	usage 2
+elif [ x"$complete" = x"1" -a $# -ne 1 ]; then
+	echo 'must only specify a date with -c'
+	usage 2
+elif [ x"$complete" != x"1" -a $# -ne 1 ]; then
+	echo 'must specify exactly one file'
+	usage 2
+fi
 
 mkdir "$0.running"
 
+if [ x"$complete" = x"1" ]; then
+	sort -u snapshot.complete.idx | xz > snapshot.complete.idx.xz
+	awk '$5 >= "'"$1"'" { 
+			if (!system("wget --method=HEAD " $9))
+				print
+		}
+		' snapshot.idx | sort -u | xz > snapshot.idx.xz
+	rm snapshot.idx snapshot.complete.idx
+	rmdir "$0.running"
+	exit 0
+fi
+
 # minimize file
 tmpfname="tmp.snapinf.asc"
 awk '
@@ -40,7 +76,7 @@ awk '
 
 	$0 == "-----END PGP SIGNATURE-----" {
 		output = 0
-	}' > "$tmpfname"
+	}' "$1" > "$tmpfname"
 
 if ! gpg --verify "$tmpfname"; then
 	echo 'failed verify'