From 942af17f577e5017340cf7091353ce5dab1a9f47 Mon Sep 17 00:00:00 2001 From: John-Mark Gurney Date: Wed, 7 Dec 2022 12:24:52 -0800 Subject: [PATCH] update blog post now that snapaid.sh makes things easier... add ability to update blog posts, missing is proper redirect from the old version. --- .../2018/07/making-freebsd-magnet-links.html | 3 + .../2022/12/making-freebsd-magnet-links.html | 172 ++++++++++++++++++ layout/blog.j2 | 3 +- 3 files changed, 177 insertions(+), 1 deletion(-) create mode 100644 content/2022/12/making-freebsd-magnet-links.html diff --git a/content/2018/07/making-freebsd-magnet-links.html b/content/2018/07/making-freebsd-magnet-links.html index 115426d..9089f81 100644 --- a/content/2018/07/making-freebsd-magnet-links.html +++ b/content/2018/07/making-freebsd-magnet-links.html @@ -9,6 +9,9 @@ tags: - magnet --- +
️ This post is old, the most up to date one is +here.
+ For the last few years, I've been producing torrents and publishing magnet links, but there is some special work that I do to make these. The first few releases, I inserted a bogus tracker into the torrent, diff --git a/content/2022/12/making-freebsd-magnet-links.html b/content/2022/12/making-freebsd-magnet-links.html new file mode 100644 index 0000000..3904ffa --- /dev/null +++ b/content/2022/12/making-freebsd-magnet-links.html @@ -0,0 +1,172 @@ +--- +title: Making FreeBSD magnet links +description: > + Making FreeBSD magnet links +created: !!timestamp '2018-07-03' +time: 4:49 PM +updated: !!timestamp '2022-12-07' +updated_time: 12:20 PM +tags: + - FreeBSD + - magnet +--- + +Note: This was originally posted +here, +but things have been improved since then and so it has been updated. + +For the last few years, I've been producing torrents and publishing +magnet links, but there is some special work that I do to make these. +The first few releases, I inserted a bogus tracker into the torrent, +because despite there being plenty of tools out there for producing +trackerless (DHT) torrents, they were all GUI and I never found any that +were command line based. The other was there was/is no tool for +extracting the info hash and building the magnet link. There may be +tools now, but I couldn't find any when I started 3 years ago. + +The following steps are based upon the recent release of FreeBSD 11.2‑R, +adjust as necessary. + +
    +
  1. Fetch the FreeBSD release and release announcement into a directory + (I create a per release directory). + remove the CHECKSUM.SHA256, CHECKSUM.SHA512 and index.html* files. +
    
    +   $ REL=12.4-RELEASE
    +   $ RELABV=12.4R
    +   $ curl https://www.funkthat.com/~jmg/FreeBSD-snap/snapshot.idx.xz | xzcat | awk ' $2 == "'"$REL"'" { print $9 }' | xargs -L 1 -P 3  wget --no-verbose -c --limit-rate=7000k
    +   $ wget https://www.freebsd.org/releases/$RELABV/announce.asc
    +
    +   
    + This step depends upon + snapaid running. + If it is no longer running for some reason, addinfo.sh could + be run manually against the release email (not the downloaded + announce.asc file as the message id is needed) to generate + snapshot.idx.xz locally, and use that instead. +
  2. Verify the GPG key that signed the above files. This is usually Glen + Barber's key, but not always. I have met and verified his fingerprint + in person, If you have verified someone's key who has signed Glen's + key, that is another good way. +
  3. Verify the release announcement: +
    
    +   $ gpg --verify announce.asc
    +   Warning: using insecure memory!
    +   gpg: Signature made Mon Dec  5 17:34:59 2022 PST using RSA key ID 478FE293
    +   gpg: Good signature from "Glen Barber " [unknown]
    +   gpg:                 aka "Glen Barber " [unknown]
    +   gpg:                 aka "Glen Barber " [unknown]
    +   gpg:                 aka "Glen Barber " [unknown]
    +   gpg: WARNING: This key is not certified with a trusted signature!
    +   gpg:          There is no indication that the signature belongs to the owner.
    +   Primary key fingerprint: 78B3 42BA 26C7 B2AC 681E  A7BE 524F 0C37 A0B9 46A3
    +        Subkey fingerprint: 8D12 403C 2E6C AB08 6CF6  4DA3 0314 58A5 478F E293
    +   
    +
  4. In the past I have used BitTornado for other things, so I ended up + using it as the basis to make the tool for creating trackerless torrent + files. The modifications were simple. It appears that the original + BitTornado CVS tree is off-line (anyways, it was served insecurely), + but it looks like + effigies/BitTornado is + similar enough that it could be modified and used. I copied + btmakemetafile.py to btmaketrackerless.py and applied the following + patch: +
    
    +   $ diff -u btmakemetafile.py btmaketrackerless.py 
    +   --- btmakemetafile.py   2004-05-24 12:54:52.000000000 -0700
    +   +++ btmaketrackerless.py        2016-10-10 17:13:32.742081000 -0700
    +   @@ -23,9 +23,9 @@
    +    def prog(amount):
    +        print '%.1f%% complete\r' % (amount * 100),
    +
    +   -if len(argv) < 3:
    +   +if len(argv) < 2:
    +        a,b = split(argv[0])
    +   -    print 'Usage: ' + b + ' <trackerurl> <file> [file...] [params...]'
    +   +    print 'Usage: ' + b + ' <file> [file...] [params...]'
    +        print
    +        print formatDefinitions(defaults, 80)
    +        print_announcelist_details()
    +   @@ -33,9 +33,9 @@
    +        exit(2)
    +
    +    try:
    +   -    config, args = parseargs(argv[1:], defaults, 2, None)
    +   -    for file in args[1:]:
    +   -        make_meta_file(file, args[0], config, progress = prog)
    +   +    config, args = parseargs(argv[1:], defaults, 1, None)
    +   +    for file in args[0:]:
    +   +        make_meta_file(file, None, config, progress = prog)
    +    except ValueError, e:
    +        print 'error: ' + str(e)
    +        print 'run with no args for parameter explanations'
    +
    +   
    + If you notice, the only thing that is done is to drop the first argument, + and instead of passing it into make_meta_file, a None is passed + instead. This will simply not add trackers to the torrent file. +
  5. I then run the following script to verify the downloaded files, and + generate the torrent files: +
    
    +   $ cat cmp.sh
    +   #!/bin/sh -
    +   # REL=12.4-RELEASE
    +   # RELABV=12.4R
    +   # xzcat ~jmg/public_html/FreeBSD-snap/snapshot.idx.xz | awk ' $2 == "'"$REL"'" { print $9 }' | xargs -L 1 -P 3  wget --no-verbose -c --limit-rate=7000k
    +   # XXX - following command 404, manually saved from relase announcement
    +   # wget https://www.freebsd.org/releases/$RELABV/announce.asc
    +   # wget -c https://download.freebsd.org/ftp/releases/CI-IMAGES/$REL/amd64/Latest/FreeBSD-$REL-amd64-BASIC-CI.raw.xz
    +   # wget -c https://download.freebsd.org/ftp/releases/VM-IMAGES/$REL/riscv64/Latest/FreeBSD-$REL-riscv-riscv64.{qcow2,raw,vhd,vmdk}.xz
    +
    +   grep -h '^   SHA512' announce.asc | sed -e 's/SHA512 (\(.*\)) = \(.*\)/\2 \1/' | sort -k 2 > sha512.from.asc
    +
    +   while read hash fname; do
    +           if [ -e "$fname" ]; then
    +                   if [ -e "$fname".torrent ]; then
    +                           echo skipping "$fname"...
    +                           continue
    +                   fi
    +
    +                   sigfile=$(grep -l -- "$fname" *.asc | head -n 1)
    +                   echo checking "$fname", sig in: "$sigfile"
    +                   #res=`sha512 -q "$fname"`
    +                   #res=`shasum -a 512 "$fname" | awk '{ print $1 }'`
    +                   res=$(openssl sha512  < "$fname" | awk '{ print $2 }')
    +                   echo "File is: $res"
    +                   if [ x"$res" != x"$hash" ]; then
    +                           echo missmatch!  "$fname"
    +                           exit 1
    +                   fi
    +                   btmaketrackerless.py "$fname" &
    +           else
    +                   echo missing "$fname"
    +                   exit 1
    +           fi
    +   done < sha512.from.asc
    +
    +   
    +
  6. Once all the torrents have been generated, I then make the magnet + links: +
    
    +   $ cat btmakemagnet.sh
    +   #!/bin/sh -
    +
    +   # metainfo file.: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso.torrent
    +   # info hash.....: 06091dabce1296d11d1758ffd071e7109a92934f
    +   # file name.....: FreeBSD-10.3-RELEASE-sparc64-bootonly.iso
    +   # file size.....: 203161600 (775 * 262144 + 0)
    +   # announce url..: udp://tracker.openbittorrent.com:80
    +   # btshowmetainfo 20030621 - decode BitTorrent metainfo files
    +
    +   for i in *.torrent; do
    +           btshowmetainfo.py "$i" | awk '
    +   $0 ~ "^info hash" { info = $3 }
    +   $0 ~ "^file name" { name = $3 }
    +   END {
    +           print "magnet:?xt=urn:btih:" info "&dn=" name
    +   }'
    +   done
    +   
    +
  7. I then create the magnet links file, and update the + Torrents wiki page. +
diff --git a/layout/blog.j2 b/layout/blog.j2 index ac7b172..38a0798 100644 --- a/layout/blog.j2 +++ b/layout/blog.j2 @@ -9,7 +9,8 @@

{{ resource.meta.title }}

-

Posted: {% if resource.meta.listable %}{{ resource.meta.created.strftime('%B %e, %Y') }} at {{ resource.meta.time }}{% else %}not yet{% endif %}

+{% if resource.meta.updated %}

Updated: {{ resource.meta.updated.strftime('%B %e, %Y') }} at {{ resource.meta.updated_time }}

{% endif %} +

{% if resource.meta.updated %}Originally {% endif %}Posted: {% if resource.meta.listable %}{{ resource.meta.created.strftime('%B %e, %Y') }} at {{ resource.meta.time }}{% else %}not yet{% endif %}

{% if resource.meta.tags %}