Creating Jigsaw Download Images for DVD and CD Sets

This page describe how Jigdo (Jigsaw Download) works. Jigdo was originally created for Debian use, but can be used by anybody. Jigdo is used to create various CD or DVD images of the same file set without having to store all the image files on the server. The server setup should be done by a server admin; the information here is for informational purposes only.

İçindekiler

1 What Jigdo does
2 How Jigdo works
3 Using Jigdo
4 How much does this save?
- 4.1 The Original CD Sets
- 4.2 The Jigsaw Download Images
5 Creating the Images

What Jigdo does

A Linux distribution is mostly a set of files put together on a CD or DVD. For a variety of reasons, multiple versions of CDs and DVDs must be offered containing mostly the same content, with little differences. For example, one CD may contain only Open source packages and another contain both Open Source and proprietary packages.

Offering all image versions for download requires much more disk space on a server. Debian was heavily impacted because it attempted to target a large number of platforms. SUSE Linux attempts to be available only on the most common hardware, but that is still a significant number.

On the distribution, many packages are hardware independant ("noarch") like doc, sources or scripts.

Whatever CD/DVD images you have, distribution must also have an "inst-source" with individual packages. Jigdo will use this tree to build CD/DVD images. If no file is found, it can also use the ISO image itself.

Jigdo can download one by one all the files necessary to build an exact equivalent of the CD or DVD. Since the download is done one file at a time, it can be stopped and started again as often as wanted. Jigdo uses wget for the download work, which is known to be a very good download client.

How Jigdo works

Jigdo needs two special files.

a ".jigdo" one to know where are the files necessary to build the images. It's a pretty small file (SUSE Linux 10.0 .jigdo file is 170kb);
a ".template" file. This file is roughtly a map of the iso file and the md5sum of the files. The size of the template file can be quite high, but still much less than the ISO.

Given these two files, jigdo begin to build an empty file the exact size of the final ISO.

Then it launches several wget sessions, each for one file and retreive it. As soon as the file is retreived, it is md5sum verified and written in the image file. There is no necessity to have the files written in any defined order, as Jigdo know where the file must go.

This way there is very little disk space overhead. One have the CD/DVD file (hudge, for the DVD), jigdo and template files and the biggest size file included, usually less than 100Mb in SUSE.

Jigdo is present in SUSE distribution and on the net. There is even a Jigdo for Windows (not documented here).

With the 10.0 .jigdo file layout, Jigdo uses different servers for each wget session, giving a very fast download.

Using Jigdo

You first need Jigdo from Yast or from http://atterer.net/jigdo/ (Linux and Windows version available) or special packages for SUSE LINUX 9.2, 9.3, or 10.0 from http://pi3.informatik.uni-mannheim.de/~schiele/suse/.

To find the file server, go to the Download page, ftp stable version, and go up in the ftp directory to find "SL-10.0-OSS/jigdo/". The files are there. Keep this URL, you will need it right now.

on summary, with the Jigdo binaries found on SUSE 10.0:

Don't dowload anything, Jigdo will do.

In the directory where you want to have the final image, type:

jigdo-lite <server><path to the .jidgo folder><.jigdo file>

For the DVD, this could be:

jigdo-lite http://ft(...)i.cz/
pub/linux/opensuse/distribution/SL-10.0-OSS/jigdo/
SUSE-10.0-DVD-OSS-i386-GM.iso.jigdo

all on the same line.

Press enter. Press enter a second time to answer yes to jigdo-lite question. Wait. That's all. On a good connection, two hours after you have the image.

More complete instructions here.

Here you can find another nice description of the tool in German.

How much does this save?

I did this for SUSE Linux 10.1 alpha1 with the provided CD sets.

The Original CD Sets

601M    SUSE-10.1-CD-OSS-i386-Alpha1-CD1.iso
649M    SUSE-10.1-CD-OSS-i386-Alpha1-CD2.iso
685M    SUSE-10.1-CD-OSS-i386-Alpha1-CD3.iso
665M    SUSE-10.1-CD-OSS-i386-Alpha1-CD4.iso
525M    SUSE-10.1-CD-OSS-i386-Alpha1-CD5.iso
3.1G    total

662M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD1.iso
688M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD2.iso
694M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD3.iso
679M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD4.iso
536M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD5.iso
3.2G    total

643M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD1.iso
686M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD2.iso
687M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD3.iso
662M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD4.iso
536M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD5.iso
3.2G    total

The Jigsaw Download Images

i386

188K    SL-10.1-OSS-alpha1.jigdo
8.6M    SUSE-10.1-CD-OSS-i386-Alpha1-CD1.template
132K    SUSE-10.1-CD-OSS-i386-Alpha1-CD2.template
14M     SUSE-10.1-CD-OSS-i386-Alpha1-CD3.template
156K    SUSE-10.1-CD-OSS-i386-Alpha1-CD4.template
116K    SUSE-10.1-CD-OSS-i386-Alpha1-CD5.template
23M     total

This is obviously enormous. Note that this is only 0.72% of the original size!

ppc

188K    SL-10.1-OSS-alpha1.jigdo
124M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD1.template
200M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD2.template
134M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD3.template
76M     SUSE-10.1-CD-OSS-ppc-Alpha1-CD4.template
195M    SUSE-10.1-CD-OSS-ppc-Alpha1-CD5.template
728M    total

These are 22% of the original size. This is not as good as for i386. Why is that the case? For some reason, SUSE builds the noarch packages on all platforms separately. Because of that the ppc version has noarch packages that are different to those on the FTP server and thus the savings for ppc are lower than for the i386 platform. If they used the same noarch packages for all platforms you could expect a total size of about 23M instead of 728M.

x86_64

188K    SL-10.1-OSS-alpha1.jigdo
118M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD1.template
200M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD2.template
134M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD3.template
77M     SUSE-10.1-CD-OSS-x86_64-Alpha1-CD4.template
177M    SUSE-10.1-CD-OSS-x86_64-Alpha1-CD5.template
704M    total

These are 22% of the original size for the same reasons as for ppc.

Creating the Images

After installing the Jigdo package you can use the following script.

#!/bin/bash
#
# mkjigdo - make jigdo files for all ISO images in one directory
# Copyright (C) 2005  Robert Schiele <rschiele@uni-mannheim.de>
#
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation; either version 2 of the License, or (at your option)
# any later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
# more details.
#
# You should have received a copy of the GNU General Public License along with
# this program; if not, write to the Free Software Foundation, Inc., 51
# Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
#

set -eu

usage()
{
    cat << EOT

Usage: $0 [OPTIONS]
Options:
  -b WORD  Basename of the jigdo file (=BASENAME)
  -c FILE  Configuration file to load
  -i PATH  Path for image files (=ISODIR)
  -j PATH  Path for jigdo files (=JIGDODIR)
  -m WORD  List of mirror servers separated by \`,' (=MIRRORS)
  -r PATH  Relative path on all mirrors (=RELDIR)
  -s PATH  Path for file storage (=SRCDIR)
  -t PATH  Path for temporary files and cache (=TMPDIR)
  -h       Output this help
  -v       Output version info
EOT
}

# set some plain stupid defaults
BASENAME=a
RELDIR=.
ISODIR=.
JIGDODIR=.
TMPDIR=.
SRCDIR=.
MIRRORS="ftp://localhost/pub"

# read standard config file
test -f ~/.$(basename "$0")rc && . ~/.$(basename "$0")rc

# parse command line
while [ "$*" ]; do
    case "$1" in
        -b) BASENAME="$2"; shift;;
        -c) . "$2"; shift;;
        -h) usage; exit;;
        -i) ISODIR="$2"; shift;;
        -j) JIGDODIR="$2"; shift;;
        -m) MIRRORS="${2//,/ }"; shift;;
        -r) RELDIR="$2"; shift;;
        -s) SRCDIR="$2"; shift;;
        -t) TMPDIR="$2"; shift;;
        -v) echo mkj version 0.0; exit 0;;
        *)  usage >&2; exit 1;;
    esac
    shift
done

# run jigdo on all ISO files
JIGDOLIST=
for ISO in "$ISODIR"/*.iso; do
    BASEFILE="${ISO/*\/}"
    BASEFILE="${BASEFILE/.iso}"
    case "$BASEFILE" in
        *.delta) ;;
        *) jigdo-file mt --bzip2 -i "$ISO" -j "$TMPDIR/$BASEFILE.jigdo.tmp" \
            -t "$JIGDODIR/$BASEFILE.template" -c "$TMPDIR/$BASENAME.cache" \
            "$SRCDIR"
            JIGDOLIST="$JIGDOLIST $TMPDIR/$BASEFILE.jigdo.tmp";;
    esac
done

# merge all temporary jigdo files into one file
{
    awk '/^[[]/{if(a==1)a=2}/^\[Image\]/{a=1}{if(a<2)print}' $JIGDOLIST
    echo '[Servers]'
    for i in $MIRRORS; do
        echo A="$i/$RELDIR/"
    done
    echo
    echo '[Parts]'
    awk '/^[[#]/{a=0}/^\[Parts\]/{a=1}/^[^[]/{if(a==1)print}' $JIGDOLIST | \
        sort -t= -k2 -u
} | gzip -c9 > "$JIGDODIR/$BASENAME.jigdo"

You can either specify all options on the command line or put them into the configuration file ~/.mkjigdorc. This is an example I used to make jigdo files for SUSE Linux 10.1 alpha1:

BASENAME=SL-10.1-OSS-alpha1
RELDIR=distribution/SL-OSS-factory
BASEDIR=/pub/opensuse/$RELDIR/
ISODIR=/pub/opensuse/distribution/$BASENAME/iso
JIGDODIR=res
TMPDIR=tmp
SRCDIR=$BASEDIR/inst-source
MIRRORS="ftp://ftp.opensuse.org/pub/opensuse
         ftp://ftp.gwdg.de/pub/opensuse"

Now just put the .jigdo and all the .template files to the FTP server and you are done.