2008/11/21

Simple packaging system



Overview

In this post I want to describe the filopack bash script, very simple (and stupid) packaging system I have prepared to ease the package manipulation. It is implemented specifically as a helper for packages presented here, but it could be used for any other system as well. It aims to help potential users with the installation process. It would be definitely better to base such a system on a well established package system like for example aptitude, rather than create my own (primitive) system from scratch, but unfortunately I had no time to study the aptitude yet (probably it would be much easier than I imagine) and started to practically learn the bash scripting instead ;-)

Following examples should explain the basic usage of the filopack packaging system. This tool serves for package enumeration, downloading, installation and un-instalation as well as package creation and uploading. However it does not handle dependencies at all, it just simplifies the installation routine, YOU have to know what packages you need.

The system is suitable for usage on the box itself as well as on the chroot-ed "mirror" system. Also besides the downloading and installation of pre-built packages, it can be used for a creation user-built packages from source code. I consider the latter two scenarios as "advanced" and will cover separately from the "basic" usage scenario - it means downloading and installing pre-built packages directly to the box.

Basic usage

In this scenario a user just wants to download and extract a package directly to the WMU box. This section will cover this scenario step by step.

Bash shell

In order to use the filopack script you need to have the bash shell installed on the system. You can use the JoKeR's prebuilt package:
box# cd /mnt/C/
box# wget http://mgb111.pradnik.net/addons/soft-shells/bash-3.2.33-3.tar.gz
box# tar xzvf bash-3.2.33-3.tar.gz
...
Now you can possibly set it as root's and users' shell:
box# nano /mnt/C/sys/etc/env-ng 
...

# change root shell to bash (if exist)
ROOTSHELL=1

# change users shell to bash (if exist)
USERSHELL=1


Busybox extended

box# cd /mnt/C/
box# wget http://mgb111.pradnik.net/addons/busybox2-extended.tar.gz
box# tar xzvf busybox2-extended.tar.gz
...
Make sure you have the /usr/bin (symlink to /mnt/C/sys/bin) in your PATH environment variable.

Download

The script can be downloaded from the Packages section or you can get it from console via wget as follows (of course you have to make it executable via chmod command):
box# cd /mnt/C
box# wget http://filodej.ic.cz/filopack/filopack.sh
box# chmod 755 filopack.sh

Usage

Now we have the script ready to be used. When we start it with no arguments it just prints the command line documentation. It covers "basic" usage which is suitable for most users.
box# ./filopack.sh
Stupid packaging system (version 0.3.6)

Usage:
 
  filopack.sh [OPTIONS] [<action>] [<package-name>]

Basic usage:

  Basic actions: 
    (typically used when you just download and install/uninstall pre-built packages)

    --help     ... prints detailed help message
    --version  ... prints version of the script
    --config   ... perform system (re)configuration
    --packages ... lists all packages available for download
    --download ... downloads a selected package
    --print    ... just prints the files of selected package
    --install  ... unpacks (installs) the package  
    --delete   ... deletes the given package archive
    --remove   ... removes (uninstalls) the package
    --summary  ... generates a summary of installed packages
In case one wants to create own packages or perform remote package installation there is also the "advanced" mode supported. I'll come back to this mode later.

Initial configuration

Now we have to configure the package destination directory (/mnt/C by default) as well as box IP/hostname and username. Latter two parameters are used just for remote packaging and so "basic" users can simply ignere it and leave the default values. With --config action we can re-configure system any time later.
box# ./filopack.sh --config
Destination address and user not configured (.filopack/.config file not found)
Insert destination directory [/mnt/C]: 
Insert box IP address or hostname (remote packaging only)[storage]: 
Insert box user account (remote packaging only) [root]: 
Settings stored to .filopack/.config file


Package listing

Now we are ready to retrieve the list of available packages. Individual package names are used as identifiers for package installation/uninstallation.
box# ./filopack.h --packages
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Available packages:
Mesa-7.0.3
MesaDemos-7.0.3
atk-1.22.0
audiofile-0.2.6
binutils-2.18
boost-1.35.0
cairo-1.6.4
code2html-0.9.1
cvstrac-2.0.1-3
dbus-1.2.1
dbus-glib-0.76
dbus-python-0.82.4
deluge-0.9.03
didiwiki-0.5
faad2-2.6.1
file-4.24
flac-1.2.1
fontconfig-2.6.0
gcc-4.1.2-sjlj
git-1.5.5.3
glade3-3.4.5
glib-2.17.2
gperf-3.0.3
gtk+-2.13.3
highlight-2.6.10
htop-0.8
httpd-2.2.8
jasper-1.701.0-2
jpeg-6b
libao-0.8.8
libglade-2.6.2
libid3tag-0.15.1b
libmikmod-3.2.0-beta2
libogg-1.1.3
libvorbis-1.2.0
libxcb-1.1
libxml2-2.6.30
libxslt-1.1.22
lsof-4.78
mod_python-3.3.1
mpd-0.13.2
pango-1.21.3
pycairo-1.4.12
pygobject-2.14.2
pygtk-2.12.1
python-2.5.2
pyxdg-0.15
source-highlight-1.9
sqlite-3.5.8
strace-4.5.15
streamripper-1.63.1
subversion-1.5.0
tiff-3.8.2
x11-7.1
xcb-util-0.2

Package retrieval

Let's say we are interested in the htop-0.8 package and we want to learn the contents.
box# ./filopack.sh --print htop-0.8
.filopack/htop-0.8.lst: No such file or directory
Unfortunately in this version we have to download the whole package in order to learn the contents (Maybe it will be changed in a future version).
We can use the following action in order to download the package:
box# ./filopack.sh --download htop-08 
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Package htop-08 not available (try to use --packages switch)
Once more, now with the correct package name:
box# ./filopack.sh --download htop-0.8
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Downloading package htop-0.8 from http://filodej.ic.cz ...
connected!

Length: 102 [text/plain]
connected!

Length: 224,259 [application/x-gtar]
Now we are ready to print the package contents:
box# ./filopack.sh --print htop-0.8
sys/bin/htop
sys/share/man/man1/htop.1
sys/share/applications/htop.desktop
sys/share/pixmaps/htop.png

Package installation
box# ./filopack.sh --install htop-0.8
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Sure to unpack htop-0.8 locally at /mnt/C (y/n)?y
sys/bin/htop
sys/share/man/man1/htop.1
sys/share/applications/htop.desktop
sys/share/pixmaps/htop.png
Now if you want to spare some disk space, you can delete the package archive file:
box# ./filopack.sh --delete htop-0.8 
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Are you sure to delete the htop-0.8.tgz archive (y/n)?y

Package removal

If later on you change your mind and decide to remove the package, you can do it as follows:
box# ./filopack.sh --remove htop-0.8
Configuration file .filopack/.config file found and used
Sure to remove htop-0.8 locally at /mnt/C (y/n)?y
... and now if you try to uninstall the same package again, operation fails (since there are no such files).
box# ./filopack.sh --remove htop-0.8
Configuration file .filopack/.config file found and used
Sure to remove htop-0.8 locally at /mnt/C (y/n)?y
rm: cannot remove `sys/bin/htop': No such file or directory
rm: cannot remove `sys/share/man/man1/htop.1': No such file or directory
rm: cannot remove `sys/share/applications/htop.desktop': No such file or directory
rm: cannot remove `sys/share/pixmaps/htop.png': No such file or directory
You may also want to get a summary what packages are available and what package archives are downloaded and/or installed.

Package summary
box# ./filopack.sh --summary
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)

Filodej package summary:
Package name                          downloaded [#installed/#total]
--------------------------------------------------------------------
Mesa-7.0.3                               yes           [  49/  49]
MesaDemos-7.0.3                          yes           [ 129/ 129]
atk-1.22.0-new                           yes           [ 160/ 160]
binutils-2.18-latest                     yes           [ 140/ 140]
boost-1.35.0-gcc4.1.2                    yes           [5195/5195]
boost-1.35                               yes           [5133/5195]
cairo-1.6.4                              yes           [  76/  76]
code2html-0.9.1                          yes           [   2/   2]
cvstrac-2.0.1-3                           no           [   3/   3]
dbus-1.2.1                               yes           [  38/  41]
dbus-glib-0.76                           yes           [  32/  32]
dbus-python-0.82.4                       yes           [  66/  66]
deluge-0.9.03-latest                     yes           [ 651/ 674]
deluge-torrent-0.5.9.3-new               yes           [   0/1543]
deluge-torrent-0.5.9.3                   yes           [   0/  99]
didiwiki-0.5                              no           [   1/   1]
file-4.24                                yes           [  11/  11]
fontconfig-2.6.0                         yes           [ 429/ 429]
gcc-4.1.2-sjlj                           yes           [  99/ 549]
git-1.5.5.3                               no           [ 228/ 228]
glade3-3.4.5                             yes           [ 323/ 323]
glib-2.17.2                              yes           [ 501/ 501]
gperf-3.0.3                              yes           [   4/   4]
gtk+-2.10.14                             yes           [1155/1155]
gtk+-2.13.3                              yes           [1233/1233]
highlight-2.6.10                          no           [ 206/ 206]
htop-0.8                                  no           [   0/   4]
httpd-2.2.8                               no           [ 174/ 174]
jasper-1.701.0-2                         yes           [  27/  27]
jpeg-6b                                  yes           [  10/  10]
libglade-2.6.2                           yes           [  34/  34]
libogg-1.1.3                              no           [  83/  83]
libvorbis-1.2.0                           no           [ 125/ 125]
libxcb-1.1                               yes           [ 160/ 160]
libxml2-2.6.30                           yes           [ 322/ 322]
libxslt-1.1.22                           yes           [ 142/ 142]
lsof-4.78                                yes           [   1/   1]
mod_python-3.3.1                          no           [  26/  26]
pango-1.21.3                             yes           [ 146/ 146]
pycairo-1.4.12                           yes           [   7/   7]
pygobject-2.14.2                         yes           [  44/  44]
pygtk-2.12.1                             yes           [ 647/ 649]
python-2.5.2                             yes           [3648/3648]
pyxdg-0.15                               yes           [  25/  25]
source-highlight-1.9                      no           [  49/  49]
sqlite-3.5.8                              no           [   8/   8]
strace-4.5.15                            yes           [   3/   3]
streamripper-1.63.1                       no           [  13/  13]
subversion-1.4.6-py-pl                    no           [ 211/ 211]
subversion-1.5.0                         yes           [ 319/ 319]
tiff-3.8.2                                no           [ 231/ 231]
x11-7.1                                  yes           [7791/7791]
xcb-util-0.2                              no           [  70/  70]
So that is all for now. In a following post I extend the tutorial with examples of "advanced" usage.

7 comments:

Denier said...

Hi. Thanks for taking the time to build this filopack thingie, although everytime I attempt to use the chmod command to "make it executable" I get a "Operation not permitted" error. Anything to say to that?

Cheers.

filodej said...

Hello Aqualung,

At first I had no idea what could cause your problem. The problem is definitely not specific to particular filopack.sh file (accidentally containing my script ;).
Later on I started to be horrified by an idea that you are trying to install deluge on a partition formatted with FAT32 file system (see for example this post).

If it is the case then please, reconsider such decision, take this comparison as an argument. Utilizing FAT32 for this purpose is really not a good idea (lack of support for symbolic links, file ownership, file permission, case sensitiveness... to say the least).

Let me know if I am wrong.
Regards,
Filodej

Denier said...

Alright, I proceeded entirely according to your instructions--that is, I downloaded the filopack.sh script directly onto the box using the wget command, and that worked fine. I then ran the "chmod 755 filopack.sh" line, and that didn't complain. I got no comments back, but at least it didn't complain, so that's gotta be fine, right?

Anyway, I then ran the "./filopack.sh --download deluge-1.0.5" line, and I got this: "-ash: ./filopack.sh: not found." Now, that's not good, right?

Any idea what it is that I'm doing wrong please?

(Sorry for being such a drag!)

filodej said...

Hi,
what changed that previously you were getting the "Operation not permitted" error and now the chmod command is silent?
(when I tried to chmod some files on a samba mounted Fat32 partition - obviously file permissions did not change but I did not get any error message)

Anyway, you still did not answer my question, what filesystem you are using?

If you do not know, on console try for example:

$ mount
/dev/rom0 on / type squashfs (ro)
none on /proc type proc (rw)
tmpfs on /tmp type tmpfs (rw)
/dev/hdc1 on /tmp/mnt/C type ext2 (rw)
/dev/hdc2 on /tmp/mnt/D type ext2 (rw)
/dev/hdc1 on /tmp/share/Public type ext2 (rw)
tmpfs on /tmp/share/USB type tmpfs (rw)
/dev/hdc2 on /tmp/Virtual/filodej/filodej type ext2 (rw)
/dev/hdc1 on /tmp/Virtual/filodej/Public type ext2 (rw)
...

Please, sometimes it is better to dump the exact console session to make sure we are talking the same language.

One more thing, The filopack script is a bash shell script and as such it needs to have the bash installed on the system. I am going to update this post with info how to install it.

Filodej

filodej said...

Hi again,
I just read your post on the Mascat forum. No matter what was your previous problem with chmod, now it seems to work (on ext2fs there should be no problem).

Now I am almost sure that the file not found message is there since you have not bash shell installed on your system. For details please see this post.

Regards,
Filodej

Denier said...

Many thanks. That must be it, I'll try it tomorrow. Anyway, I guess, then, that you should've listed this bash shell among the prerequisites... When talking to me, you should assume that all I have on my machine is what you've listed as prerequisites/dependencies, and *only* that.

Many thanks for a speedy reply, and for putting up with Linux virgins like me. Very much appreciate the patience!

Cheers!

filodej said...

You are right regarding the prerequisites. Fixed.

Filodej