2008/11/24

WMU-6500FS - Deluge torrent 1.0.5 (part IV)

Screenshots

Finally I found time to build the full version of deluge bittorrent client (version 1.0.5) and create a package for it.

Deluge 1.0.5 GTK client
(running on CoLinux with Debian 4.0, XMing server running on WinXP) Deluge 1.0.5 GTK client
(running on WinXP) Deluge 1.0.5 GTK client
(running on Ubuntu 8.04)

Build result

[binary] [file list]
Prerequisites: [JoKeR's minimal config] [JoKeR's essential libs]
For filopack system: [JoKeR's bash shell] [JoKeR's busybox extended]
Dependencies: [gcc libraries] [glib] [readline] [boost] [python] [setuptools] [DBus] [DBus-glib] [DBus-python] [pygobject] [pyxdg]
GUI dependencies: [X.Org] [Gtk+] [pycairo] [pygtk] [atk] [glade]
Edit: Regarding to this post it seems that [pygobject] and [pyxdg] packages are needed even for non-gui mode.

Overview

First I'll demonstrate the shortest path (given all the needed dependencies are already installed). Then I describe the build and package creation process, since I feel that such information can be helpful when something goes wrong.

Download & Install

We can utilize the filopack system for that purpose. In case we do not have it we can simply download it (and make executable) as follows:
box# cd /mnt/C
box# wget http://filodej.ic.cz/filopack/filopack.sh
box# chmod 755 filopack.sh
First action we have to do is to uninstall the previous Deluge version (assumed we have both Deluge 0.9.03 and ./filopack.sh installed on the box):
box# cd /mnt/C
box# ./filopack.sh --remove deluge-0.9.03
Configuration file .filopack/.config file found and used
Sure to remove deluge-0.9.03 locally at /mnt/C (y/n)?y
...
Now we can download the package:
box# cd /mnt/C
box# ./filopack.sh --download deluge-1.0.5
[root@storage C]# ./filopack.sh --download deluge-1.0.5
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Downloading package deluge-1.0.5 from http://filodej.ic.cz ...
connected!

Length: 68,332 [text/plain]
connected!

Length: 15,645,450 [application/x-tar]
And the last step is to install it:
box# ./filopack.sh --install deluge-1.0.5
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Sure to unpack deluge-1.0.5 locally at /home/petr (y/n)?
...
Now, we are done (if we are lucky ;-)

I added scripts (deluge.sh and deluged.sh) which we can run in place of original loader scripts (deluge and deluged) which are working around the weird problems with dynamic loader missing some symbols.
box# cat /mnt/C/sys/bin/deluged.sh
LD_PRELOAD="/usr/lib/libssl.so.0.9.7 /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0" deluged
So if you already have the /mnt/C/sys/bin/ directory in your PATH now instead of:
box# deluged
/usr/bin/python: can't resolve symbol '__cxa_pure_virtual'
... you can simple type:
box# deluged.sh
It is great that the daemon mode is now supported and we can run the GTK GUI remotely (even so the demands of python runtime are relatively high for the box).
In following figures you can see the clients running on Ubuntu and WinXP both connected to the daemon running on the box. In previous figure there is also the client running on the box - it can be seen that its progress controls are way behind the real time and even the PC running client does not receive much information from the daemon. In such situations everything becomes reaaaalllllly sloooowwww as the box can't catch the train.

It is obvious that it is not a good idea to run GTK client directly on the box, since it pushes its HW resources really to its limits, but in is nice that it (no matter how slowly) works.

Build & Pack

Everything necessary for successful build was already described in previous three parts of this series, this part just collects the information and describes the whole building process step-by-step.

Same as in the previous case we have to uninstall the old Deluge version (0.9.03)
dev# cd /mnt/C
dev# ./filopack.sh --remove deluge-0.9.03
Configuration file .filopack/.config file found and used
Sure to remove deluge-0.9.03 locally at /mnt/C (y/n)?
...
Now we are ready to start the building process. First thing we have to do is to create a timestamp file. With help of this file the packaging system determines what files are included to the resulting package.
dev# ./filopack.sh --init  deluge-1.0.5
Configuration file .filopack/.config file found and used
Timestamp written to file .filopack/deluge-1.0.5.ts
Now we can download and extract the sources.
dev# cd /usr/local/src/
dev# wget http://download.deluge-torrent.org/source/1.0.5/deluge-1.0.5.tar.bz2
...
dev# tar xjvf deluge-1.0.5.tar.bz2
dev# cd /usr/local/src/deluge-1.0.5
We have to do the following steps in order to make a successful build:
(For explanations see previous parts of this series)
Add proper paths to boost header files and libraries
dev# export CFLAGS=-I/mnt/C/sys/include/boost-1_35
dev# export LDFLAGS=-L/mnt/C/sys/lib
Add missing IPV6 related macro to socket.hpp header file:
dev# nano libtorrent/include/libtorrent/socket.hpp
    add:
      #ifndef IPV6_V6ONLY
      #define IPV6_V6ONLY 26
      #endif
Disable the memory debugging (file libtorrent/src/memdebug.cpp) from the build. You can:
- either modify the Makefile
- or clear the memdebug.cpp
- or use the preprocessor condition:
      #if 0
        ...
      #endif
Now we are ready to start the build and installation:
dev# python setup.py build 
...
dev# python setup.py install --prefix=/mnt/C/sys/
...
If the build and installation succeeded, we are almost done. Now, when we try to start the deluge daemon, the dynamic loader is missing some symbols (it was same with previous version, I am possibly doing something wrong but do not know what).
dev# deluged
dev# /usr/bin/python: can't resolve symbol '__cxa_pure_virtual'
At least we have the workaround - library preload, with following triar and error method we get the set of libraries to be rpe-loaded:
dev# LD_PRELOAD=/usr/lib/libstdc++.so.6 deluged
dev# /usr/bin/python: can't resolve symbol 'BIO_free'

dev# LD_PRELOAD="/usr/lib/libstdc++.so.6 /usr/lib/libssl.so.0.9.7" deluged
dev# /usr/bin/python: can't resolve symbol '_ZN5boost6system18get_posix_categoryEv'

dev# LD_PRELOAD="/usr/lib/libssl.so.0.9.7 /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0" deluged
OK, now it works. We can prepare the loader script with previous command-line:
dev# echo LD_PRELOAD=\"/usr/lib/libssl.so.0.9.7 /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0\" deluged \"\$@\" > /mnt/C/sys/bin/deluged.sh
dev# chmod +x /mnt/C/sys/bin/deluged.sh 
Now (if we have the /mnt/C/sys/bin in our PATH) we can run the deluge daemon as follows:
dev# deluged.sh
For deluge client the symbol problem is the same, so we prepare a loader script as well:
dev# echo LD_PRELOAD=\"/usr/lib/libssl.so.0.9.7 /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0\" \"\$@\" deluge > /mnt/C/sys/bin/deluge.sh
dev# chmod +x /mnt/C/sys/bin/deluge.sh 
Now we try to launch the GUI (GTK client) but some problems still remain:
dev# deluge.sh
...
  File "/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/common.py", line 142, in get_logo
    size, size)
gobject.GError: Unrecognized image file format
We already solved this problem with previous version, so we know the receipt:
dev# sed -i 's/deluge.svg/deluge.png/g' /mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/common.py
Let's try again:
dev# deluge.sh
...
  self.window.show()
/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/ui/gtkui/mainwindow.py:95: GtkWarning: gtk_icon_theme_load_icon: assertion `GTK_IS_ICON_THEME (icon_theme)' failed
  self.window.show()
Segmentation fault
We forgot the set up the DISPLAY environment variable properly (Actually I do not know why it crashes in such case, but the truth is that proper DISPLAY setup solves the problem):
dev# export DISPLAY=<x-server-ip>
One more thing we have to do is to let python know the path to the deluge egg:
dev# cd /mnt/C/sys/lib/python2.5/site-packages
dev# echo "./deluge-1.0.5-py2.5-linux-i686.egg" > deluge.pth 
Finally we are ready to create the deluge-1.0.5 package:
dev# cd /mnt/C
dev# ./filopack.sh --pack deluge-1.0.5

Remote install

Now we have the package created locally on the "mirror" system (colinux debian in my case). But we have to get it to the box. The filopack script can help us with this task as well.

First we have to uninstall the previous deluge version from the box. We can definitely do it locally on the box itself, but we can also do it remotely:
dev# ./filopack.sh -r --remove deluge-0.9.03
Configuration file .filopack/.config file found and used
Sure to remove deluge-0.9.03 remotely at root@storage:/mnt/C (y/n)?y
...
If we want to make sure that the previously installed software is gone, we can look at the package summary (unfortunately this feature is supported only locally). So from the box shell we can do following:
box# ./filopack.sh --summary
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)

Filodej package summary:
Package name                          downloaded [#installed/#total]
--------------------------------------------------------------------
Mesa-7.0.3                                no           [   0/  49]
MesaDemos-7.0.3                           no           [ 129/ 129]
atk-1.22.0                                no           [ 160/ 160]
audiofile-0.2.6                           no           [   0/  13]
binutils-2.18                             no           [ 140/ 140]
boost-1.35.0                              no           [5195/5195]
cairo-1.6.4                               no           [  76/  76]
code2html-0.9.1                           no           [   2/   2]
cvstrac-2.0.1-3                           no           [   3/   3]
dbus-1.2.1                                no           [  38/  38]
dbus-glib-0.76                            no           [  32/  32]
dbus-python-0.82.4                        no           [  66/  66]
deluge-0.9.03                        no          [   0/ 673]
didiwiki-0.5                              no           [   1/   1]
faad2-2.6.1                               no           [   0/   8]
...
Now we are back in the "mirror" system console and we can upload the (newly created) package to the box:
dev# ./filopack.sh -r --upload deluge-1.0.5 
Configuration file .filopack/.config file found and used
Sure to upload deluge-1.0.5 to root@storage:/mnt/C (y/n)?y
root@storage's password: 
deluge-1.0.5.tar.bz2                                                100%   15MB 318.3KB/s   00:48    
And perform the remote installation as well:
dev# ./filopack.sh -r --install deluge-1.0.5
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Module deluge-1.0.5 not found in package list - use 'tar.bz2' extension
Sure to unpack deluge-1.0.5 remotely at root@storage:/mnt/C (y/n)?y
root@storage's password: 
...
Deluge daemon (deluged.sh) now works OK on the box.

The only remaining problem is following. When we run the GTK GUI client (deluge.sh) we get following error:
box# deluge.sh
...
gobject.GError: Failed to open file '/tmp/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/data/pixmaps/inactive16.png': No such file or directory
When we look to the /tmp/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/data/pixmaps/ directory there is not a inactive16.png there is just inactive16.png0000644 file and the same is with checking16.png file.
It seems it is a bug in tar implementation, the file permissions are "leaking out" to the name (see this issue description or this forum).

The solution is easy in this case:
box# mv /tmp/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/data/pixmaps/inactive16.png{0000644,}
box# mv /tmp/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/data/pixmaps/checking16.png{0000644,}
When you run the deluge and it looks similar like this: ... then there are some missing links to font directories:
box# mkdir /usr/share/fonts
box# ln -sn /mnt/C/sys/X11/lib/X11/fonts/OTF /usr/share/fonts/X11-OTF
box# ln -sn /mnt/C/sys/X11/lib/X11/fonts/TTF /usr/share/fonts/X11-TTF

Setup

Now (after the update of the python package) we are able to run the console client right on the box: We can read the configuration options:
>  configs             

* add_paused: False
* allow_remote: False
* auto_managed: True
* autoadd_enable: False
* autoadd_location: C
* compact_allocation: False
* config_location: /mnt/C/sys/root/.config/deluge
...
* daemon_port: 58846
...
If we want access the daemon remotely, we can set it up as follows (of course it is possible to change the port as well):
> config-set allow_remote True

* Configuration value successfully updated.

>  quit 

Thanks
box#
Now we have to restart the daemon (if it is running):
box# killall deluged
box# deluged.sh
... and we are set up to access the daemon remotely (from PC running either Linux or Windows) as can be seen on following screenshots:
Deluge 1.0.5 GTK client
(running on WinXP) Deluge 1.0.5 GTK client
(running on Ubuntu 8.04)
That's all for now.

Read more...

2008/11/21

Simple packaging system

Overview

In this post I want to describe the filopack bash script, very simple (and stupid) packaging system I have prepared to ease the package manipulation. It is implemented specifically as a helper for packages presented here, but it could be used for any other system as well. It aims to help potential users with the installation process. It would be definitely better to base such a system on a well established package system like for example aptitude, rather than create my own (primitive) system from scratch, but unfortunately I had no time to study the aptitude yet (probably it would be much easier than I imagine) and started to practically learn the bash scripting instead ;-)

Following examples should explain the basic usage of the filopack packaging system. This tool serves for package enumeration, downloading, installation and un-instalation as well as package creation and uploading. However it does not handle dependencies at all, it just simplifies the installation routine, YOU have to know what packages you need.

The system is suitable for usage on the box itself as well as on the chroot-ed "mirror" system. Also besides the downloading and installation of pre-built packages, it can be used for a creation user-built packages from source code. I consider the latter two scenarios as "advanced" and will cover separately from the "basic" usage scenario - it means downloading and installing pre-built packages directly to the box.

Basic usage

In this scenario a user just wants to download and extract a package directly to the WMU box. This section will cover this scenario step by step.

Bash shell

In order to use the filopack script you need to have the bash shell installed on the system. You can use the JoKeR's prebuilt package:
box# cd /mnt/C/
box# wget http://mgb111.pradnik.net/addons/soft-shells/bash-3.2.33-3.tar.gz
box# tar xzvf bash-3.2.33-3.tar.gz
...
Now you can possibly set it as root's and users' shell:
box# nano /mnt/C/sys/etc/env-ng 
...

# change root shell to bash (if exist)
ROOTSHELL=1

# change users shell to bash (if exist)
USERSHELL=1


Busybox extended

box# cd /mnt/C/
box# wget http://mgb111.pradnik.net/addons/busybox2-extended.tar.gz
box# tar xzvf busybox2-extended.tar.gz
...
Make sure you have the /usr/bin (symlink to /mnt/C/sys/bin) in your PATH environment variable.

Download

The script can be downloaded from the Packages section or you can get it from console via wget as follows (of course you have to make it executable via chmod command):
box# cd /mnt/C
box# wget http://filodej.ic.cz/filopack/filopack.sh
box# chmod 755 filopack.sh

Usage

Now we have the script ready to be used. When we start it with no arguments it just prints the command line documentation. It covers "basic" usage which is suitable for most users.
box# ./filopack.sh
Stupid packaging system (version 0.3.6)

Usage:
 
  filopack.sh [OPTIONS] [<action>] [<package-name>]

Basic usage:

  Basic actions: 
    (typically used when you just download and install/uninstall pre-built packages)

    --help     ... prints detailed help message
    --version  ... prints version of the script
    --config   ... perform system (re)configuration
    --packages ... lists all packages available for download
    --download ... downloads a selected package
    --print    ... just prints the files of selected package
    --install  ... unpacks (installs) the package  
    --delete   ... deletes the given package archive
    --remove   ... removes (uninstalls) the package
    --summary  ... generates a summary of installed packages
In case one wants to create own packages or perform remote package installation there is also the "advanced" mode supported. I'll come back to this mode later.

Initial configuration

Now we have to configure the package destination directory (/mnt/C by default) as well as box IP/hostname and username. Latter two parameters are used just for remote packaging and so "basic" users can simply ignere it and leave the default values. With --config action we can re-configure system any time later.
box# ./filopack.sh --config
Destination address and user not configured (.filopack/.config file not found)
Insert destination directory [/mnt/C]: 
Insert box IP address or hostname (remote packaging only)[storage]: 
Insert box user account (remote packaging only) [root]: 
Settings stored to .filopack/.config file


Package listing

Now we are ready to retrieve the list of available packages. Individual package names are used as identifiers for package installation/uninstallation.
box# ./filopack.h --packages
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Available packages:
Mesa-7.0.3
MesaDemos-7.0.3
atk-1.22.0
audiofile-0.2.6
binutils-2.18
boost-1.35.0
cairo-1.6.4
code2html-0.9.1
cvstrac-2.0.1-3
dbus-1.2.1
dbus-glib-0.76
dbus-python-0.82.4
deluge-0.9.03
didiwiki-0.5
faad2-2.6.1
file-4.24
flac-1.2.1
fontconfig-2.6.0
gcc-4.1.2-sjlj
git-1.5.5.3
glade3-3.4.5
glib-2.17.2
gperf-3.0.3
gtk+-2.13.3
highlight-2.6.10
htop-0.8
httpd-2.2.8
jasper-1.701.0-2
jpeg-6b
libao-0.8.8
libglade-2.6.2
libid3tag-0.15.1b
libmikmod-3.2.0-beta2
libogg-1.1.3
libvorbis-1.2.0
libxcb-1.1
libxml2-2.6.30
libxslt-1.1.22
lsof-4.78
mod_python-3.3.1
mpd-0.13.2
pango-1.21.3
pycairo-1.4.12
pygobject-2.14.2
pygtk-2.12.1
python-2.5.2
pyxdg-0.15
source-highlight-1.9
sqlite-3.5.8
strace-4.5.15
streamripper-1.63.1
subversion-1.5.0
tiff-3.8.2
x11-7.1
xcb-util-0.2

Package retrieval

Let's say we are interested in the htop-0.8 package and we want to learn the contents.
box# ./filopack.sh --print htop-0.8
.filopack/htop-0.8.lst: No such file or directory
Unfortunately in this version we have to download the whole package in order to learn the contents (Maybe it will be changed in a future version).
We can use the following action in order to download the package:
box# ./filopack.sh --download htop-08 
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Package htop-08 not available (try to use --packages switch)
Once more, now with the correct package name:
box# ./filopack.sh --download htop-0.8
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Downloading package htop-0.8 from http://filodej.ic.cz ...
connected!

Length: 102 [text/plain]
connected!

Length: 224,259 [application/x-gtar]
Now we are ready to print the package contents:
box# ./filopack.sh --print htop-0.8
sys/bin/htop
sys/share/man/man1/htop.1
sys/share/applications/htop.desktop
sys/share/pixmaps/htop.png

Package installation
box# ./filopack.sh --install htop-0.8
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Sure to unpack htop-0.8 locally at /mnt/C (y/n)?y
sys/bin/htop
sys/share/man/man1/htop.1
sys/share/applications/htop.desktop
sys/share/pixmaps/htop.png
Now if you want to spare some disk space, you can delete the package archive file:
box# ./filopack.sh --delete htop-0.8 
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Are you sure to delete the htop-0.8.tgz archive (y/n)?y

Package removal

If later on you change your mind and decide to remove the package, you can do it as follows:
box# ./filopack.sh --remove htop-0.8
Configuration file .filopack/.config file found and used
Sure to remove htop-0.8 locally at /mnt/C (y/n)?y
... and now if you try to uninstall the same package again, operation fails (since there are no such files).
box# ./filopack.sh --remove htop-0.8
Configuration file .filopack/.config file found and used
Sure to remove htop-0.8 locally at /mnt/C (y/n)?y
rm: cannot remove `sys/bin/htop': No such file or directory
rm: cannot remove `sys/share/man/man1/htop.1': No such file or directory
rm: cannot remove `sys/share/applications/htop.desktop': No such file or directory
rm: cannot remove `sys/share/pixmaps/htop.png': No such file or directory
You may also want to get a summary what packages are available and what package archives are downloaded and/or installed.

Package summary
box# ./filopack.sh --summary
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)

Filodej package summary:
Package name                          downloaded [#installed/#total]
--------------------------------------------------------------------
Mesa-7.0.3                               yes           [  49/  49]
MesaDemos-7.0.3                          yes           [ 129/ 129]
atk-1.22.0-new                           yes           [ 160/ 160]
binutils-2.18-latest                     yes           [ 140/ 140]
boost-1.35.0-gcc4.1.2                    yes           [5195/5195]
boost-1.35                               yes           [5133/5195]
cairo-1.6.4                              yes           [  76/  76]
code2html-0.9.1                          yes           [   2/   2]
cvstrac-2.0.1-3                           no           [   3/   3]
dbus-1.2.1                               yes           [  38/  41]
dbus-glib-0.76                           yes           [  32/  32]
dbus-python-0.82.4                       yes           [  66/  66]
deluge-0.9.03-latest                     yes           [ 651/ 674]
deluge-torrent-0.5.9.3-new               yes           [   0/1543]
deluge-torrent-0.5.9.3                   yes           [   0/  99]
didiwiki-0.5                              no           [   1/   1]
file-4.24                                yes           [  11/  11]
fontconfig-2.6.0                         yes           [ 429/ 429]
gcc-4.1.2-sjlj                           yes           [  99/ 549]
git-1.5.5.3                               no           [ 228/ 228]
glade3-3.4.5                             yes           [ 323/ 323]
glib-2.17.2                              yes           [ 501/ 501]
gperf-3.0.3                              yes           [   4/   4]
gtk+-2.10.14                             yes           [1155/1155]
gtk+-2.13.3                              yes           [1233/1233]
highlight-2.6.10                          no           [ 206/ 206]
htop-0.8                                  no           [   0/   4]
httpd-2.2.8                               no           [ 174/ 174]
jasper-1.701.0-2                         yes           [  27/  27]
jpeg-6b                                  yes           [  10/  10]
libglade-2.6.2                           yes           [  34/  34]
libogg-1.1.3                              no           [  83/  83]
libvorbis-1.2.0                           no           [ 125/ 125]
libxcb-1.1                               yes           [ 160/ 160]
libxml2-2.6.30                           yes           [ 322/ 322]
libxslt-1.1.22                           yes           [ 142/ 142]
lsof-4.78                                yes           [   1/   1]
mod_python-3.3.1                          no           [  26/  26]
pango-1.21.3                             yes           [ 146/ 146]
pycairo-1.4.12                           yes           [   7/   7]
pygobject-2.14.2                         yes           [  44/  44]
pygtk-2.12.1                             yes           [ 647/ 649]
python-2.5.2                             yes           [3648/3648]
pyxdg-0.15                               yes           [  25/  25]
source-highlight-1.9                      no           [  49/  49]
sqlite-3.5.8                              no           [   8/   8]
strace-4.5.15                            yes           [   3/   3]
streamripper-1.63.1                       no           [  13/  13]
subversion-1.4.6-py-pl                    no           [ 211/ 211]
subversion-1.5.0                         yes           [ 319/ 319]
tiff-3.8.2                                no           [ 231/ 231]
x11-7.1                                  yes           [7791/7791]
xcb-util-0.2                              no           [  70/  70]
So that is all for now. In a following post I extend the tutorial with examples of "advanced" usage.

Read more...