2008/12/08

WMU-6500FS - built sw -Python 2.5.2

[Homepage]

[files] [binary]

First I briefly write about the initial build:
dev# cd /usr/local/src
dev# wget http://python.org/ftp/python/2.5.2/Python-2.5.2.tgz
dev# tar xvzf Python-2.5.2.tgz
dev# cd Python-2.5.2
dev# ./configure --prefix=/mnt/C/sys --enable-shared
dev# make
There was an error with missing 'yperr_string' symbol (I am not sure whether it was during the build or when I launched python console). The symbol was needed by the nis extension module. I do not know how on earth I would utilize an Interface to Sun's NIS (Yellow pages) so as a workaround I simply disabled it.
dev# make
dev# make install
dev# python
Another failure - error while loading shared libraries: libpython2.5.so.1.0 (it is located in /mnt/C/sys/lib).
As a solution, if there was a dynamic loader configuration on the box (/etc/ld.so.conf.d) we could do following:
dev# echo /usr/local/lib > /etc/ld.so.conf.d/python2.5.conf
dev# ldconfig
As there is not, we can use the LD_LIBRARY_PATH variable instead:
dev# nano /mnt/C/sys/etc/rc-local
Add following:
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/mnt/C/sys/lib
Now everything works ok.

After some time I realized that there is a readline extension module missing on the system. This module is needed for interactive console applications, and particularly is needed for the deluge console:
box# deluge --ui=null
[INFO    ] 21:58:04 main:93 Deluge ui 1.0.5
[DEBUG   ] 21:58:04 main:94 options: {'config': None, 'logfile': None, 'ui': 'null'}
[DEBUG   ] 21:58:04 main:95 args: []
[DEBUG   ] 21:58:05 configmanager:44 ConfigManager started..
[INFO    ] 21:58:05 main:98 Starting ui..
[DEBUG   ] 21:58:05 ui:44 UI init..
[DEBUG   ] 21:58:05 configmanager:88 Getting config 'ui.conf'
[DEBUG   ] 21:58:05 config:47 Config created with filename: ui.conf
[DEBUG   ] 21:58:05 config:48 Config defaults: {'default_ui': 'gtk'}
[INFO    ] 21:58:05 ui:68 Starting NullUI..
[DEBUG   ] 21:58:06 client:54 CoreProxy init..
Traceback (most recent call last):
  File "/usr/bin/deluge", line 8, in <module>
    load_entry_point('deluge==1.0.5', 'console_scripts', 'deluge')()
  File "/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/main.py", line 99, in start_ui
    UI(options, args)
  File "/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/ui/ui.py", line 69, in __init__
    from deluge.ui.null.deluge_shell import NullUI
  File "/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/ui/null/deluge_shell.py", line 29, in <module>
    import readline
ImportError: No module named readline
We can replicate this problem directly from the python console:
dev# python
Python 2.5.2 (r252:60911, May  3 2008, 16:19:27)
[GCC 3.3.6] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named readline
>>>
So let's look at the old log of python's configure script to see why the readline extension package was not built:
dev# cat config.log | grep readline
configure:20980: checking for readline in -lreadline
configure:21015: gcc -pthread -o conftest -g -O2   conftest.c -lreadline  -lpthread -ldl  -lutil >&5
/usr/bin/ld: cannot find -lreadline
| char readline ();
| return readline ();
configure:21126: checking for rl_callback_handler_install in -lreadline
configure:21161: gcc -pthread -o conftest -g -O2   conftest.c -lreadline  -lpthread -ldl  -lutil >&5
/usr/bin/ld: cannot find -lreadline
configure:21254: checking for rl_pre_input_hook in -lreadline
configure:21289: gcc -pthread -o conftest -g -O2   conftest.c -lreadline  -lpthread -ldl  -lutil >&5
/usr/bin/ld: cannot find -lreadline
configure:21325: checking for rl_completion_matches in -lreadline
configure:21360: gcc -pthread -o conftest -g -O2   conftest.c -lreadline  -lpthread -ldl  -lutil >&5
/usr/bin/ld: cannot find -lreadline
ac_cv_lib_readline_readline=no
ac_cv_lib_readline_rl_callback_handler_install=no
ac_cv_lib_readline_rl_completion_matches=no
ac_cv_lib_readline_rl_pre_input_hook=no
We can see that the library simply was not on the system and so the python configure script decided to exclude it's python wrapper from the build.

Before we proceed, we have to build the GNU readline and create a new package for it. How it was done is described in this post.

Now we have the libreadline.so library on the system and so can proceed to re-configure and re-build the python package. First we create a new timestamp file for python package and touch all the old files (in order to be included in the package). We can use the filopack script for that purpose:
dev# cd /mnt/C
dev# ./filopack.sh --init python-2.5.2
dev# ./filopack.sh --touch python-2.5.2
Configuration file .filopack/.config file found and used
Sure to touch all files of the python-2.5.2 locally at /mnt/C (y/n)?y
...
Now we can move to the python source directory and re-configure the build:
dev# cd /usr/local/src/Python-2.5.2
dev# ./configure --prefix=/mnt/C/sys --enable-shared
...
Let's check whether the readline package was found and included to the build:
dev# cat config.log | grep readline
configure:20980: checking for readline in -lreadline
configure:21015: gcc -pthread -o conftest -I/mnt/C/sys/include -I/mnt/C/sys/X11/include  -L/mnt/C/sys/lib -L/mnt/C/sys/X11/lib conftest.c -lreadline  -lpthread -ldl  -lutil >&5
/mnt/C/sys/lib/libreadline.so: undefined reference to `tgetnum'
/mnt/C/sys/lib/libreadline.so: undefined reference to `tgoto'
/mnt/C/sys/lib/libreadline.so: undefined reference to `tgetflag'
/mnt/C/sys/lib/libreadline.so: undefined reference to `BC'
/mnt/C/sys/lib/libreadline.so: undefined reference to `tputs'
/mnt/C/sys/lib/libreadline.so: undefined reference to `PC'
/mnt/C/sys/lib/libreadline.so: undefined reference to `tgetent'
/mnt/C/sys/lib/libreadline.so: undefined reference to `UP'
/mnt/C/sys/lib/libreadline.so: undefined reference to `tgetstr'
...
ac_cv_lib_readline_readline=no
ac_cv_lib_readline_rl_callback_handler_install=no
ac_cv_lib_readline_rl_completion_matches=no
ac_cv_lib_readline_rl_pre_input_hook=no
The situation changed, the readline library is (presumably) no longer missing, but still there are some missing symbols and thus the configure test failed.

After a short googling (for query "libreadline.so: undefined reference to BC PC UP") I found this page.
It seems that the missing symbols should be present in ncurses library.

Let's look what symbols are at the library in our system:
dev# nm /lib/libncurses.so | grep "tgetnum\|tgoto\|tgetflag\|BC\|PC\|UP"
0003a544 B BC
0003a770 B PC
0003a540 B UP
000248e1 T tgetflag
00024958 T tgetnum
00024b0c T tgoto
It means that the symbols are there, but the python configuration script did not manage to find the library.

For quick feedback for our experiments we can create a dummy C file and use the same compiler command line as the configure script did:
dev# echo "int main() { return 0; }" > conftest.c
dev# gcc -pthread -o conftest -I/mnt/C/sys/include -I/mnt/C/sys/X11/include  -L/mnt/C/sys/lib -L/mnt/C/sys/X11/lib conftest.c -lreadline  -lpthread -ldl  -lutil
/mnt/C/sys/lib/gcc/i386-linux-uclibc/4.1.2/../../../../i386-linux-uclibc/lib/crt1.o: In function `_start':
crt1.S:(.text+0x27): undefined reference to `main'
/mnt/C/sys/lib/libreadline.so: undefined reference to `tgetnum'
/mnt/C/sys/lib/libreadline.so: undefined reference to `tgoto'
/mnt/C/sys/lib/libreadline.so: undefined reference to `tgetflag'
/mnt/C/sys/lib/libreadline.so: undefined reference to `BC'
/mnt/C/sys/lib/libreadline.so: undefined reference to `tputs'
/mnt/C/sys/lib/libreadline.so: undefined reference to `PC'
/mnt/C/sys/lib/libreadline.so: undefined reference to `tgetent'
/mnt/C/sys/lib/libreadline.so: undefined reference to `UP'
/mnt/C/sys/lib/libreadline.so: undefined reference to `tgetstr'
collect2: ld returned 1 exit status
We see that the linker ended up with the same error. Now we can try to fix the command line. There is missing ncurses library on the command line; what about adding it explicitly?
dev# gcc -pthread -o conftest -I/mnt/C/sys/include -I/mnt/C/sys/X11/include  -L/mnt/C/sys/lib
 -L/mnt/C/sys/X11/lib conftest.c -lncurses -lreadline  -lpthread -ldl  -lutil
The test build now works, so now we can remove the temporary C-file and re-configure the python build:
dev# rm conftest.c

dev# LIBS="-lncurses" ./configure --prefix=/mnt/C/sys --enable-shared
...
Let's make sure that readline package passed the configure test:
 
dev# cat config.log | grep readline
configure:20980: checking for readline in -lreadline
configure:21015: gcc -pthread -o conftest -I/mnt/C/sys/include -I/mnt/C/sys/X11/include  -L/mnt/C/sys/lib -L/mnt/C/sys/X11/lib conftest.c -lreadline  -lpthread -ldl -lncurses -lutil >&5
configure:21126: checking for rl_callback_handler_install in -lreadline
configure:21161: gcc -pthread -o conftest -I/mnt/C/sys/include -I/mnt/C/sys/X11/include  -L/mnt/C/sys/lib -L/mnt/C/sys/X11/lib conftest.c -lreadline  -lreadline -lpthread -ldl -lncurses -lutil >&5
configure:21254: checking for rl_pre_input_hook in -lreadline
configure:21289: gcc -pthread -o conftest -I/mnt/C/sys/include -I/mnt/C/sys/X11/include  -L/mnt/C/sys/lib -L/mnt/C/sys/X11/lib conftest.c -lreadline  -lreadline -lpthread -ldl -lncurses -lutil >&5
configure:21325: checking for rl_completion_matches in -lreadline
configure:21360: gcc -pthread -o conftest -I/mnt/C/sys/include -I/mnt/C/sys/X11/include  -L/mnt/C/sys/lib -L/mnt/C/sys/X11/lib conftest.c -lreadline  -lreadline -lpthread -ldl -lncurses -lutil >&5
ac_cv_lib_readline_readline=yes
ac_cv_lib_readline_rl_callback_handler_install=yes
ac_cv_lib_readline_rl_completion_matches=yes
ac_cv_lib_readline_rl_pre_input_hook=yes
Yes, everything is ok now, we can build the source and install the binaries:
dev# make
...
/mnt/C/sys/include/sqlite3.h: version 3.5.8
INFO: Can't locate Tcl/Tk libs and/or headers
building 'readline' extension
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I/usr/local/src/Python-2.5.2/./Include -I/mnt/C/sys/include -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/src/Python-2.5.2/Include -I/usr/local/src/Python-2.5.2 -c /usr/local/src/Python-2.5.2/Modules/readline.c -o build/temp.linux-i686-2.5/usr/local/src/Python-2.5.2/Modules/readline.o
gcc -pthread -shared -L/mnt/C/sys/lib -L/mnt/C/sys/X11/lib -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes build/temp.linux-i686-2.5/usr/local/src/Python-2.5.2/Modules/readline.o -L/usr/lib/termcap -L/mnt/C/sys/lib -L/mnt/C/sys/X11/lib -L/usr/local/lib -L. -lreadline -lncurses -lpython2.5 -o build/lib.linux-i686-2.5/readline.so
building 'nis' extension
gcc -pthread -fPIC -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I/usr/local/src/Python-2.5.2/./Include -I/mnt/C/sys/include -I. -IInclude -I./Include -I/usr/local/include -I/usr/local/src/Python-2.5.2/Include -I/usr/local/src/Python-2.5.2 -c /usr/local/src/Python-2.5.2/Modules/nismodule.c -o build/temp.linux-i686-2.5/usr/local/src/Python-2.5.2/Modules/nismodule.o
gcc -pthread -shared -L/mnt/C/sys/lib -L/mnt/C/sys/X11/lib -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes build/temp.linux-i686-2.5/usr/local/src/Python-2.5.2/Modules/nismodule.o -L/mnt/C/sys/lib -L/mnt/C/sys/X11/lib -L/usr/local/lib -L. -lnsl -lpython2.5 -o build/lib.linux-i686-2.5/nis.so
*** WARNING: renaming "nis" since importing it failed: dynamic module does not define init function (initnis)
running build_scripts

dev# make install
...
Now, let's test the new installation:
dev# python
Python 2.5.2 (r252:60911, Dec  8 2008, 07:26:01)
[GCC 4.1.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
This is fine; and what about the deluge console:
dev# deluge --ui=null
[INFO    ] 07:28:18 main:93 Deluge ui 1.0.5
[DEBUG   ] 07:28:18 main:94 options: {'config': None, 'logfile': None, 'ui': 'null'}
[DEBUG   ] 07:28:18 main:95 args: []
[DEBUG   ] 07:28:18 configmanager:44 ConfigManager started..
[INFO    ] 07:28:18 main:98 Starting ui..
[DEBUG   ] 07:28:18 ui:44 UI init..
[DEBUG   ] 07:28:18 configmanager:88 Getting config 'ui.conf'
[DEBUG   ] 07:28:18 config:47 Config created with filename: ui.conf
[DEBUG   ] 07:28:18 config:48 Config defaults: {'default_ui': 'gtk'}
[INFO    ] 07:28:18 ui:68 Starting NullUI..
[DEBUG   ] 07:28:18 client:54 CoreProxy init..
Welcome to deluge-shell. Type 'help' to see a list of available commands.
> help

Available commands:
        * add: Add a torrent
        * config-set: Change a configuration setting
        * configs: Show configuration values
        * connect: Connect to a new deluge server.
        * del: Remove a torrent
        * exit: Exit from the client.
        * halt: Shutdown the deluge server.
        * help: Show help
        * info: Show information about the torrents
        * pause: Pause a torrent
        * quit: Exit from the client.
        * resume: Resume a torrent
        * rm: Remove a torrent
> quit

Thanks
Everything is now ok, we can create a new python package:
dev# cd /mnt/C
dev# ./filopack.sh --pack python-2.5.2
...
I realized that there are many files which are not a necessary part of the resulting package. Thhose are files with .pyc and .pyo extensions - just in time compiled python files. So I removed them from the list and re-created the package:
dev# ./filopack.sh -R --pack python-2.5.2
...
I have uploaded the resulting package on the web site. Now we can fix the problem on the box.

First let's look at the summary of installed files:
box# cd /mnt/C
box# ./filopack.sh --summary
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)

Filodej package summary:
Package name                          downloaded [#installed/#total]
--------------------------------------------------------------------
...
python-2.5.2                              no           [3525/3648]
...
... so the python package is installed; let's remove it:
box# ./filopack.sh --remove python-2.5.2
Configuration file .filopack/.config file found and used
Sure to remove python-2.5.2 locally at /mnt/C (y/n)?y
...
Now we can download and install both readline-5.2:
box# ./filopack.sh --download readline-5.2
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Downloading package readline-5.2 from http://filodej.ic.cz ...
connected!

Length: 573 [text/plain]
connected!

Length: 265,962 [application/x-tar]

box# ./filopack.sh --install readline-5.2
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Sure to unpack readline-5.2 locally at /mnt/C (y/n)?y
sys/lib/libreadline.a
sys/lib/libhistory.a
sys/lib/libhistory.so.5
sys/lib/libhistory.so.5.2
sys/lib/libhistory.so
sys/lib/libreadline.so.5.2
sys/lib/libreadline.so.5
sys/lib/libreadline.so
sys/man/man3/readline.3
sys/man/man3/history.3
sys/include/readline/rltypedefs.h
sys/include/readline/chardefs.h
sys/include/readline/history.h
sys/include/readline/keymaps.h
sys/include/readline/readline.h
sys/include/readline/rlconf.h
sys/include/readline/rlstdc.h
sys/include/readline/tilde.h
sys/info/readline.info
sys/info/rluserman.info
sys/info/history.info
... and python-2.5.2 package:
box# ./filopack.sh --download python-2.5.2
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Downloading package python-2.5.2 from http://filodej.ic.cz ...
connected!

Length: 134,205 [text/plain]
connected!

Length: 12,984,464 [application/x-tar]

box# ./filopack.sh --install python-2.5.2
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Sure to unpack python-2.5.2 locally at /mnt/C (y/n)?y
...
After the installation everything seemed ok:
box# python
Python 2.5.2 (r252:60911, Dec  8 2008, 07:26:01)
[GCC 4.1.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>>
... but surprisingly one more problem arose when I ran the deluge console:
box# deluged.sh

box# deluge --ui=null
[INFO    ] 16:56:44 main:93 Deluge ui 1.0.5
[DEBUG   ] 16:56:44 main:94 options: {'config': None, 'logfile': None, 'ui': 'null'}
[DEBUG   ] 16:56:44 main:95 args: []
[DEBUG   ] 16:56:45 configmanager:44 ConfigManager started..
[INFO    ] 16:56:45 main:98 Starting ui..
[DEBUG   ] 16:56:45 ui:44 UI init..
[DEBUG   ] 16:56:45 configmanager:88 Getting config 'ui.conf'
[DEBUG   ] 16:56:45 config:47 Config created with filename: ui.conf
[DEBUG   ] 16:56:45 config:48 Config defaults: {'default_ui': 'gtk'}
[INFO    ] 16:56:45 ui:68 Starting NullUI..
[DEBUG   ] 16:56:46 client:54 CoreProxy init..
Welcome to deluge-shell. Type 'help' to see a list of available commands.
Traceback (most recent call last):
  File "/usr/bin/deluge", line 8, in 
    load_entry_point('deluge==1.0.5', 'console_scripts', 'deluge')()
  File "/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/main.py", line 99, in start_ui
    UI(options, args)
  File "/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/ui/ui.py", line 70, in __init__
    ui = NullUI(args)
  File "/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/ui/null/deluge_shell.py", line 470, in __init__
    readline.read_init_file()
IOError: [Errno 2] No such file or directory
When we look at the python readline documentation we see the following:
readline.read_init_file([filename])
    Parse a readline initialization file. The default filename is the last filename used.
It seems that there is a readline initialization file missing on the box (probably it was outside the /mnt/C/sys directory tree and so was not transferred to the box).

Let's try to replicate the problem directly from python console:
box# python
Python 2.5.2 (r252:60911, Dec  8 2008, 07:26:01)
[GCC 4.1.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>> readline.read_init_file()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory
When I tried it on the colinux system, everything was ok:
dev# python
Python 2.5.2 (r252:60911, Dec  8 2008, 07:26:01)
[GCC 4.1.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>> readline.read_init_file()
>>>
Hopefully the system call trace will help:
box# strace -o readline-init.txt python
Python 2.5.2 (r252:60911, Dec  8 2008, 07:26:01)
[GCC 4.1.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>> readline.read_init_file()
Traceback (most recent call last):
  File "<stdin>", line 1, in 
IOError: [Errno 2] No such file or directory
>>>
Let's look at the end of the trace log:
box# tail -n 100 readline-init.txt
...
rt_sigprocmask(SIG_BLOCK, [INT], [RTMIN], 8) = 0
ioctl(0, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig icanon echo ...}) = 0
rt_sigprocmask(SIG_SETMASK, [RTMIN], NULL, 8) = 0
rt_sigaction(SIGWINCH, {SIG_DFL}, {0x40139066, [], SA_RESTORER|SA_RESTART, 0x401c45a8}, 8) = 0
time(NULL)                              = 1228752774
stat("/mnt/C/sys/root/.inputrc", 0xbffff4ec) = -1 ENOENT (No such file or directory)
stat("/etc/inputrc", 0xbffff4ec)        = -1 ENOENT (No such file or directory)
write(2, "Traceback (most recent call last"..., 35) = 35
...
write(2, "  File \"<stdin>\", line 1, in <mo"..., 38) = 38
write(2, "IOError", 7)                  = 7
write(2, ": ", 2)                       = 2
write(2, "[Errno 2] No such file or direct"..., 35) = 35
write(2, "\n", 1)                       = 1
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
rt_sigprocmask(SIG_BLOCK, [INT], [RTMIN], 8) = 0
ioctl(0, TIOCGWINSZ, {ws_row=31, ws_col=84, ws_xpixel=0, ws_ypixel=0}) = 0
ioctl(0, TIOCSWINSZ, {ws_row=31, ws_col=84, ws_xpixel=0, ws_ypixel=0}) = 0
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(0, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig -icanon -echo ...}) = 0
rt_sigprocmask(SIG_SETMASK, [RTMIN], NULL, 8) = 0
rt_sigaction(SIGWINCH, {0x40139066, [], SA_RESTORER|SA_RESTART, 0x401c45a8}, {SIG_DFL}, 8) = 0
write(1, ">>> ", 4)                     = 4
select(1, [0], NULL, NULL, NULL)        = 1 (in [0])
rt_sigprocmask(SIG_BLOCK, NULL, [RTMIN], 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [RTMIN], 8) = 0
read(0, "\4", 1)                        = 1
rt_sigprocmask(SIG_BLOCK, [INT], [RTMIN], 8) = 0
ioctl(0, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig icanon echo ...}) = 0
rt_sigprocmask(SIG_SETMASK, [RTMIN], NULL, 8) = 0
rt_sigaction(SIGWINCH, {SIG_DFL}, {0x40139066, [], SA_RESTORER|SA_RESTART, 0x401c45a8}, 8) = 0
write(2, "\n", 1)                       = 1
rt_sigaction(SIGINT, {SIG_DFL}, {0x40139066, [], SA_RESTORER, 0x401c45a8}, 8) = 0
munmap(0x402ad000, 16188)               = 0
munmap(0x402b1000, 216100)              = 0
_exit(0)                                = ?
So we found that neither /etc/inputrc nor /mnt/C/sys/root/.inputrc is present on the box and thus the readline initialization complains.

As a solution I choosen to copy the initialization file to root's user directory and re-build the readline package:
dev# cp /etc/inputrc /mnt/C/sys/root/.inputrc
dev# echo "sys/root/.inputrc" >> .filopack/readline-5.2.lst
dev# ./filopack.sh -R --pack readline-5.2
After the re-deployment to the box everything works ok:
dev# deluge --ui=null
[INFO    ] 17:22:53 main:93 Deluge ui 1.0.5
[DEBUG   ] 17:22:53 main:94 options: {'config': None, 'logfile': None, 'ui': 'null'}
[DEBUG   ] 17:22:53 main:95 args: []
[DEBUG   ] 17:22:53 configmanager:44 ConfigManager started..
[INFO    ] 17:22:53 main:98 Starting ui..
[DEBUG   ] 17:22:53 ui:44 UI init..
[DEBUG   ] 17:22:53 configmanager:88 Getting config 'ui.conf'
[DEBUG   ] 17:22:53 config:47 Config created with filename: ui.conf
[DEBUG   ] 17:22:53 config:48 Config defaults: {'default_ui': 'gtk'}
[INFO    ] 17:22:53 ui:68 Starting NullUI..
[DEBUG   ] 17:22:55 client:54 CoreProxy init..
Welcome to deluge-shell. Type 'help' to see a list of available commands.
> help

Available commands:
        * add: Add a torrent
        * config-set: Change a configuration setting
        * configs: Show configuration values
        * connect: Connect to a new deluge server.
        * del: Remove a torrent
        * exit: Exit from the client.
        * halt: Shutdown the deluge server.
        * help: Show help
        * info: Show information about the torrents
        * pause: Pause a torrent
        * quit: Exit from the client.
        * resume: Resume a torrent
        * rm: Remove a torrent

> connect


> info

*** ID: 33820db6dd5e5928d23bc811bbac2f4ae94cb882
*** Name: ubuntu-8.10-desktop-i386.iso
*** Path: torrentfiles
*** Completed: 0.0 KiB/698.8 MiB
*** Status: Paused


> resume

Usage: resume  [ ...]


> resume 33820db6dd5e5928d23bc811bbac2f4ae94cb882


> info

*** ID: 33820db6dd5e5928d23bc811bbac2f4ae94cb882
*** Name: ubuntu-8.10-desktop-i386.iso
*** Path: torrentfiles
*** Completed: 2.1 MiB/698.8 MiB
*** Status: Downloading
*** Download Speed: 60.9 KiB/s
*** Upload Speed: 0.0 KiB/s
*** ETA: 3h 15m


> exit

Thanks
box#
That's all for now.

Read more...

WMU-6500FS - built sw - readline-5.2

[Homepage]

[files] [binary]

First we initialize the package timestamp:
dev# cd /mnt/C/
dev# ./filopack.sh --init readline-5.2
Configuration file .filopack/.config file found and used
Timestamp written to file .filopack/readline-5.2.ts
Now we can download and extract the source to the /usr/local/src directory:
dev# cd /usr/local/src
dev# wget ftp://ftp.gnu.org/gnu/readline/readline-5.2.tar.gz
--06:08:23--  ftp://ftp.gnu.org/gnu/readline/readline-5.2.tar.gz
           => `readline-5.2.tar.gz'
Resolving ftp.gnu.org... 140.186.70.20
Connecting to ftp.gnu.org[140.186.70.20]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /gnu/readline ... done.
==> PORT ... done.    ==> RETR readline-5.2.tar.gz ... done.
Length: 2,037,705 (unauthoritative)

100%[===============================================================================================================================>] 2,037,705      4.38K/s    ETA 00:00

06:20:28 (2.75 KB/s) - `readline-5.2.tar.gz' saved [2037705]

dev# tar xzvf readline-5.2.tar.gz
...
Configuration and build process is straightforward:
dev# cd readline-5.2
dev# ./configure --prefix=/mnt/C/sys
...
dev# make
...
dev# make install
...
Now we go back to the /mnt/C directory and finish the package creation:
dev# cd /mnt/C/
dev# ./filopack.sh --pack readline-5.2
Configuration file .filopack/.config file found and used
-rw-r--r-- root/root    308780 2008-12-08 06:26:38 sys/lib/libreadline.a
-rw-r--r-- root/root     37294 2008-12-08 06:26:38 sys/lib/libhistory.a
lrwxrwxrwx root/root         0 2008-12-08 06:26:38 sys/lib/libhistory.so.5 -> libhistory.so.5.2
-r-xr-xr-x root/root     34696 2008-12-08 06:26:38 sys/lib/libhistory.so.5.2
lrwxrwxrwx root/root         0 2008-12-08 06:26:38 sys/lib/libhistory.so -> libhistory.so.5
-r-xr-xr-x root/root    249866 2008-12-08 06:26:38 sys/lib/libreadline.so.5.2
lrwxrwxrwx root/root         0 2008-12-08 06:26:38 sys/lib/libreadline.so.5 -> libreadline.so.5.2
lrwxrwxrwx root/root         0 2008-12-08 06:26:38 sys/lib/libreadline.so -> libreadline.so.5
-rw-r--r-- root/root     38459 2008-12-08 06:26:38 sys/man/man3/readline.3
-rw-r--r-- root/root     21769 2008-12-08 06:26:38 sys/man/man3/history.3
-rw-r--r-- root/root      2977 2008-12-08 06:26:38 sys/include/readline/rltypedefs.h
-rw-r--r-- root/root      4684 2008-12-08 06:26:38 sys/include/readline/chardefs.h
-rw-r--r-- root/root     10207 2008-12-08 06:26:38 sys/include/readline/history.h
-rw-r--r-- root/root      3529 2008-12-08 06:26:38 sys/include/readline/keymaps.h
-rw-r--r-- root/root     34515 2008-12-08 06:26:38 sys/include/readline/readline.h
-rw-r--r-- root/root      2305 2008-12-08 06:26:38 sys/include/readline/rlconf.h
-rw-r--r-- root/root      1503 2008-12-08 06:26:38 sys/include/readline/rlstdc.h
-rw-r--r-- root/root      3167 2008-12-08 06:26:38 sys/include/readline/tilde.h
-rw-r--r-- root/root    192320 2008-12-08 06:26:38 sys/info/readline.info
-rw-r--r-- root/root     72955 2008-12-08 06:26:38 sys/info/rluserman.info
-rw-r--r-- root/root     58583 2008-12-08 06:26:38 sys/info/history.info
One more thing we have to do is to copy the readline initialization file to the root's directory, add it to the .lst file and re-build the package:
dev# cp /etc/inputrc /mnt/C/sys/root/.inputrc
dev# echo "sys/root/.inputrc" >> .filopack/readline-5.2.lst
dev# ./filopack.sh -R --pack readline-5.2

Read more...

2008/12/04

Questions and answers


This post will contain answers for some questions, which can be interesting for others and which are more elaborated so that their scope exceeds a limit reasonable for ordinary comments.

Deluge installation, missing pkg_resources package

Q:

Hi Filodej,
thanks a lot for your reply. I got stuck when i was building the Apache files.
I don't know much about linux, but i'll try and i'll let you know if i get Deluge running

I'm planning to do this:

1.- Flash WMU box with Joker Firmware (already done w/o problems)

2.- Uncompress the binary files that you already build inside the WMU box (i'm not gonna use the chroot-ed system with uClibc library under Kubuntu)
For example:
tar xvzf binary_file

where binary_file are the following dependencies

X.Org
Gtk+
boost
python
pycairo
pygtk
atk
glade
pygobject
pyxdg
DBus
DBus-glib
DBus-python

I'll follow the order you set in your blog

3.- Modify configuration files or set environment variables (if any)

And that's all. I don't know if i can do this, or if i'm missing something. Hope this works! 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Hi Filodej,

i tried what i told you before. I uncompress the binaries and i got an error like "can't load library 'librt.so.0'"
So, i uncompress libs-essential-20080502.tar.bz2 and that fixed the error. But now, when i try to run Deluge i get this error:


[root@Nas_Storage site-packages]# deluge
Traceback (most recent call last):
File "/usr/bin/deluge", line 5, in < module >
from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

I couldn't see that error on your blog and couldn't find any clue to help me solve this, do you have any idea about how to dix this?
Thanks a lot, and sorry for my qustions

Edgard 

A:

The pkg_resources is a part of python setuptools package, you have to install it. I have install it during the Trac installation and so did not encounter this dependency, sorry for that.

When I try it on a relatively clean debian installation (with python 2.5 already installed), I get the same error:
debian:~# python2.5
Python 2.5 (release25-maint, Jul 20 2008, 20:47:25)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named pkg_resources
>>>
In order to install the setuptools package we can follow the installation instructions or as described in the Trac installation (setuptools section) use the ez_setup.py script:

First, we have to download it:
debian:~# cd /tmp
debian:/tmp# wget http://peak.telecommunity.com/dist/ez_setup.py
--04:10:59--  http://peak.telecommunity.com/dist/ez_setup.py
           => `ez_setup.py'
Resolving peak.telecommunity.com... 209.190.5.234
Connecting to peak.telecommunity.com|209.190.5.234|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9,716 (9.5K) [text/plain]

100%[=======================================================================================================>] 9,716         24.97K/s

04:11:00 (24.97 KB/s) - `ez_setup.py' saved [9716/9716]
Now we can try to launch it:
debian:/tmp# python2.5 ez_setup.py
Downloading http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg
error: invalid Python installation: unable to open /usr/include/python2.5/pyconfig.h (No such file or directory)
It seems we are missing some python C headers, we have to install the package with python 2.5 header files:
debian:/tmp# apt-get install python2.5-dev
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
  python2.5-dev
0 upgraded, 1 newly installed, 0 to remove and 36 not upgraded.
Need to get 0B/1676kB of archives.
After unpacking 5349kB of additional disk space will be used.
Selecting previously deselected package python2.5-dev.
(Reading database ... 25024 files and directories currently installed.)
Unpacking python2.5-dev (from .../python2.5-dev_2.5-5+etch1_i386.deb) ...
Setting up python2.5-dev (2.5-5+etch1) ...
Now try it again:
debian:/tmp# python2.5 ez_setup.py
Downloading http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg
Processing setuptools-0.6c9-py2.5.egg
Copying setuptools-0.6c9-py2.5.egg to /usr/lib/python2.5/site-packages
Adding setuptools 0.6c9 to easy-install.pth file
Installing easy_install script to /usr/bin
Installing easy_install-2.5 script to /usr/bin

Installed /usr/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg
Processing dependencies for setuptools==0.6c9
Finished processing dependencies for setuptools==0.6c9
Now we can test whether the installation was sucessful:
debian:/tmp# python2.5
Python 2.5 (release25-maint, Jul 20 2008, 20:47:25)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>>

GCC package is huge, is it really necessary?

Q:

Anonymous said... 
    ... 
    I tried to download 'gcc' from you site but the files seems to be corrupted. Do you have nay ideas ?

    below is a screen-shot from my machine:

    $ tar xjvf gcc-4.1.2-sjlj.tar.bz2.tar
    sys/bin/c++
    sys/bin/cpp
    sys/bin/g++
    sys/bin/gcc
    sys/bin/gccbug
    sys/bin/gcov
    sys/bin/i386-linux-uclibc-c++
    sys/bin/i386-linux-uclibc-g++
    sys/bin/i386-linux-uclibc-gcc
    sys/bin/i386-linux-uclibc-gcc-4.1.2
    sys/lib/gcc/i386-linux-uclibc/4.1.2/install-tools/include/README
    sys/lib/gcc/i386-linux-uclibc/4.1.2/install-tools/include/float.h
    sys/lib/gcc/i386-linux-uclibc/4.1.2/install-tools/include/iso646.h
    sys/lib/gcc/i386-linux-uclibc/4.1.2/install-tools/include/stdarg.h
    sys/lib/gcc/i386-linux-uclibc/4.1.2/install-tools/include/stdbool.h
    sys/lib/gcc/i386-linux-uclibc/4.1.2/install-tools/include/stddef.h
    sys/lib/gcc/i386-linux-uclibc/4.1.2/install-tools/include/varargs.h
    sys/lib/gcc/i386-linux-uclibc/4.1.2/install-tools/include/mmintrin.h
    sys/lib/gcc/i386-linux-uclibc/4.1.2/install-tools/include/mm3dnow.h
    sys/lib/gcc/i386-linux-uclibc/4.1.2/install-tools/include/xmmintrin.h
    sys/lib/gcc/i386-linux-uclibc/4.1.2/install-tools/include/emmintrin.h
    tar: Decompression failed
    tar: Unable to read all data
    tar: Unable to read all data

    regards,
    Christos

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Anonymous said...

    oops...sorry file was corrupted during download - apologies.
    Christos

A:

The truth is that the gcc package is huge (~25MB) though there are just few libraries for deluge. I am going to address this issue so that I cut just a few relevant libraries out of the huge gcc package and provide a new package (~1MB) for it. Following text is by no means necessary for the successful installation, I am just going to describe the process of cutting out relevant dependencies for interested...

First we can list the installed packages. It is possible to use the filopack script for it:
box# cd /mnt/C
box# ./filopack --summary
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)

Filodej package summary:
Package name                          downloaded [#installed/#total]
--------------------------------------------------------------------
...
gcc-4.1.2-sjlj                            no           [ 546/ 549]
...
Now we can uninstall the package:
box# ./filopack.sh --remove gcc-4.1.2-sjlj
Configuration file .filopack/.config file found and used
Sure to remove gcc-4.1.2-sjlj locally at /mnt/C (y/n)?y
Let's make sure the package is actually removed:
box# ./filopack.sh --summary
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)

Filodej package summary:
Package name                          downloaded [#installed/#total]
--------------------------------------------------------------------
...
gcc-4.1.2-sjlj                            no           [   0/ 549]
...
Ok, now when we try to run deluge, we get the "missing library" error message:
box# deluged.sh
/usr/bin/python: can't load library 'libstdc++.so.6'
It seems we need at least the libstdc++.so.6 library. Let's find all the related files:
box# cat .filopack/gcc-4.1.2-sjlj.lst | grep  libstdc++
sys/lib/libstdc++.a
sys/lib/libstdc++.so
sys/lib/libstdc++.so.6.0.8
sys/lib/libstdc++.so.6
sys/lib/libstdc++.la
We actually do not need the static libraries (unless we have a plan to build something directly on the box), so we are interested just in so files:
box# cat .filopack/gcc-4.1.2-sjlj.lst | grep  libstdc++*.so
sys/lib/libstdc++.so
sys/lib/libstdc++.so.6.0.8
sys/lib/libstdc++.so.6
Let's make a new lst file representing the new package deluge-1.0.5-gcclibs we are going to create. This file will contain a list of all the relevant libraries:
box# cat .filopack/gcc-4.1.2-sjlj.lst | grep  libstdc++*.so > .filopack/deluge-1.0.5-gcclibs.lst
... and extract just those files from the gcc-4.1.2-sjlj package:
box# cat .filopack/gcc-4.1.2-sjlj.lst | grep libstdc++*.so | xargs tar -xjvf gcc-4.1.2-sjlj.tar.bz2
sys/lib/libstdc++.so
sys/lib/libstdc++.so.6.0.8
sys/lib/libstdc++.so.6
When we try to start deluge daemon again:
box# deluged.sh
/usr/bin/python: Can't resolve symbol '_Unwind_SjLj_Register'
... it seems there are still some missing libraries (or more likely some missing symbols in an older version of the same library). Let's look at the dependencies more closer:
box# LD_TRACE_LOADED_OBJECTS=1 deluged.sh
        /usr/lib/libssl.so.0.9.7 => /usr/lib/libssl.so.0.9.7 (0x40007000)
        /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0 => /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0 (0x40030000)
        libpython2.5.so.1.0 => /mnt/C/sys/lib//libpython2.5.so.1.0 (0x4003f000)
        libpthread.so.0 => /mnt/C/sys/lib//libpthread.so.0 (0x40166000)
        libdl.so.0 => /lib/libdl.so.0 (0x40178000)
        libutil.so.0 => /mnt/C/sys/lib//libutil.so.0 (0x4017b000)
        libm.so.0 => /lib/libm.so.0 (0x4017d000)
        libc.so.0 => /lib/libc.so.0 (0x4018b000)
        libcrypto.so.0.9.7 => /mnt/C/sys/lib//libcrypto.so.0.9.7 (0x4021f000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x402f4000)
        libboost_system-gcc41-mt-1_35.so.1.35.0 => /mnt/C/sys/lib//libboost_system-gcc41-mt-1_35.so.1.35.0 (0x402fc000)
        librt.so.0 => /mnt/C/sys/lib//librt.so.0 (0x40300000)
        libstdc++.so.6 => /mnt/C/sys/lib//libstdc++.so.6 (0x40302000)
        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
Here is a list of libraries present in the gcc package:
dev# cat .filopack/gcc-4.1.2-sjlj.lst | grep "\.so"
sys/lib/libgcc_s.so
sys/lib/libgcc_s.so.1
sys/lib/libstdc++.so
sys/lib/libstdc++.so.6.0.8
sys/lib/libstdc++.so.6
sys/lib/libmudflap.so.0.0.0
sys/lib/libmudflap.so.0
sys/lib/libmudflap.so
sys/lib/libmudflapth.so.0.0.0
sys/lib/libmudflapth.so.0
sys/lib/libmudflapth.so
sys/lib/libssp.so.0.0.0
sys/lib/libssp.so.0
sys/lib/libssp.so
The libgcc_s looks like a good candidate. We can look for the missing symbol:
box# nm /lib/libgcc_s.so.1
nm: /lib/libgcc_s.so.1: no symbols

box# nm --dynamic /lib/libgcc_s.so | grep Unwind
00003ebc T _Unwind_Backtrace
00003e98 T _Unwind_DeleteException
00002800 T _Unwind_FindEnclosingFunction
00004e2c T _Unwind_Find_FDE
00003c50 T _Unwind_ForcedUnwind
000027dc T _Unwind_GetCFA
00002838 T _Unwind_GetDataRelBase
00003f60 T _Unwind_GetGR
00003f84 T _Unwind_GetIP
000027e8 T _Unwind_GetLanguageSpecificData
000027f4 T _Unwind_GetRegionStart
00002844 T _Unwind_GetTextRelBase
00003a08 T _Unwind_RaiseException
00003d00 T _Unwind_Resume
00003dc8 T _Unwind_Resume_or_Rethrow
00003f70 T _Unwind_SetGR
00003f90 T _Unwind_SetIP
The /lib/libgcc_s.so is an "old" version of the library being replaced by the new one contained in the gcc package. Apparently the symbol '_Unwind_SjLj_Register' is missing in the list, but there are similar symbols.
Let's list the libgcc related files:
box# cat .filopack/gcc-4.1.2-sjlj.lst | grep libgcc_s*.so
sys/lib/libgcc_s.so
sys/lib/libgcc_s.so.1
Now we can extract the files from the package tarball:
box# cat .filopack/gcc-4.1.2-sjlj.lst | grep libgcc_s*.so | xargs tar -xjvf gcc-4.1.2-sjlj.tar.bz2
sys/lib/libgcc_s.so
sys/lib/libgcc_s.so.1
Make sure that the symbol is actually there:
box# nm --dynamic /mnt/C/sys/lib/libgcc_s.so | grep Unwind
00005361 T _Unwind_Backtrace
00005345 T _Unwind_DeleteException
0000522e T _Unwind_FindEnclosingFunction
00004e4b T _Unwind_Find_FDE
000051d6 T _Unwind_GetCFA
00005235 T _Unwind_GetDataRelBase
000051c5 T _Unwind_GetGR
000051fb T _Unwind_GetIP
0000521a T _Unwind_GetLanguageSpecificData
00005227 T _Unwind_GetRegionStart
0000523c T _Unwind_GetTextRelBase
000051e7 T _Unwind_SetGR
00005209 T _Unwind_SetIP
000054a5 T _Unwind_SjLj_ForcedUnwind
000054ef T _Unwind_SjLj_RaiseException
0000511c T _Unwind_SjLj_Register
00005452 T _Unwind_SjLj_Resume
0000557d T _Unwind_SjLj_Resume_or_Rethrow
000051ba T _Unwind_SjLj_Unregister
Now we can append the files to the lst file.
box# cat .filopack/gcc-4.1.2-sjlj.lst | grep libgcc_s*.so >> .filopack/deluge-1.0.5-gcclibs.lst
And test the deluge again:
[root@storage C]# deluged.sh
... now the daemon starts without problems.
The last step is to create the deluge-1.0.5-gcclibs package.
box# cat .filopack/deluge-1.0.5-gcclibs.lst
sys/lib/libstdc++.so
sys/lib/libstdc++.so.6.0.8
sys/lib/libstdc++.so.6
sys/lib/libgcc_s.so
sys/lib/libgcc_s.so.1
We can use the --pack action of the filopack script with -R option in order to reuse the (already created) lst file:
box# ./filopack.sh -R --pack deluge-1.0.5-gcclibs
Configuration file .filopack/.config file found and used
tar: unrecognized option `--files-from=.filopack/deluge-1.0.5-gcclibs.lst'
BusyBox v1.90-JKR (2008.04.09-06:53 CET) multi-call binary

Usage: tar -[czjZxtvO] [-f TARFILE] [-C DIR] [FILE(s)] ...

Create, extract, or list files from a tar file.

Options:
        c               create
        x               extract
        t               list

Archive format selection:
        z               Filter the archive through gzip
        j               Filter the archive through bzip2
        Z               Filter the archive through compress

File selection:
        f               name of TARFILE or "-" for stdin
        O               extract to stdout
        C               change to directory DIR before operation
        v               verbosely list files processed

... oops, it seems that I have different versions of tar on my box and on my dev# system (Chroot-ed environment on top of the Colinux with Debian), there the --files-from works ok.
Let's make sure. First find the tar on the box:
box# which tar
/bin/tar
box# ls -l /bin/tar
lrwxrwxrwx 1 root root 7 2008-05-21 13:40 /bin/tar -> busybox
And now on the dev# system:
dev# tar --version
tar (GNU tar) 1.15.1

dev# which tar
/bin/tar

dev# ls -l /bin/tar
-rwxr-xr-x  1 root root 166880 Apr 16  2008 /bin/tar*
As a solution I have updated the filopack script to be compliant with the busybox tar version. I changed the following statement:
tar $PACK_CMD $PACKAGE.$EXT --files-from $CONFIG_DIR/$PACKAGE.lst 2>&1 | tee $CONFIG_DIR/$PACKAGE.err
to the following:
cat $CONFIG_DIR/$PACKAGE.lst | xargs tar $PACK_CMD $PACKAGE.$EXT 2>&1 | tee $CONFIG_DIR/$PACKAGE.err
.. the only thing I am not sure of is whether the number of command line argument is somewhat limited; such limitation could cause some problems for packages with many files.
Let's try it directly:
box# ./filopack.sh -R --pack deluge-1.0.5-gcclibs
Configuration file .filopack/.config file found and used
tar: Creating bzip2 compressed archives is not currently supported.
Equivalent command:
cat .filopack/deluge-1.0.5-gcclibs.lst | xargs tar cjvf deluge-1.0.5-gcclibs.tar.bz2

Bzip compression is not supported, so try the tgz:
box# ./filopack.sh -R -o tgz --pack deluge-1.0.5-gcclibs
Configuration file .filopack/.config file found and used
sys/lib/libgcc_s.so.1
sys/lib/libgcc_s.so
sys/lib/libstdc++.so.6
sys/lib/libstdc++.so.6.0.8
sys/lib/libstdc++.so
Equivalent command:
cat .filopack/deluge-1.0.5-gcclibs.lst | xargs tar czvf deluge-1.0.5-gcclibs.tgz

... this works. We have successfully created the package.
Note: Finally I created the bz2 version on my mirror system.
dev# ./filopack.sh -R --pack deluge-1.0.5-gcclibs
...
So, that's all for now. I have uploaded the package and now it is publicly available:
box# ./filopack.sh --packages
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Available packages:
...
deluge-1.0.5
deluge-1.0.5-deplibs
...
You can download it to your box:
box# ./filopack.sh --download deluge-1.0.5-gcclibs
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Downloading package deluge-1.0.5-gcclibs from http://filodej.ic.cz ...
connected!

Length: 113 [text/plain]
connected!

Length: 1,191,263 [application/x-tar]
And install it as follows:
box# ./filopack.sh --install deluge-1.0.5-gcclibs
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Sure to unpack deluge-1.0.5-gcclibs locally at /mnt/C (y/n)?y
sys/lib/libstdc++.so
sys/lib/libstdc++.so.6.0.8
sys/lib/libstdc++.so.6
sys/lib/libgcc_s.so
sys/lib/libgcc_s.so.1

Unresolved 'PyUnicodeUCS2_DecodeUTF8' symbol

Q:

$ deluged.sh
/usr/bin/python: Can't resolve symbol '_Unwind_SjLj_Register'
$
$
$ LD_PRELOAD="/usr/lib/libssl.so.0.9.7 /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0 /usr/lib/libgcc_s.so" deluged -d
[DEBUG ] 08:53:55 configmanager:44 ConfigManager started..
[INFO ] 08:53:55 daemon:44 Deluge daemon 1.0.5
[DEBUG ] 08:53:55 daemon:45 options: {'logfile': None, 'config': None, 'port': None, 'donot': True}
[DEBUG ] 08:53:55 daemon:46 args: []
/usr/bin/python: can't resolve symbol 'PyUnicodeUCS2_DecodeUTF8'
$

A:

Likely this symbol should be defined in python library. Let's look if it is the case:
box# nm /mnt/C/sys/lib/libpython2.5.so.1.0 | grep DecodeUTF8
00073f08 T PyUnicodeUCS4_DecodeUTF8
00073f30 T PyUnicodeUCS4_DecodeUTF8Stateful
So it seems there is a UCS4 vs. UCS2 mismatch on your system. When I googled PyUnicodeUCS4 PyUnicodeUCS2 I got the following explanation:
You are using a version of Python that uses a 4-byte representation for Unicode characters, but some C extension module you are importing was compiled using a Python that uses a 2-byte representation for Unicode characters (the default).
So it seems to be exactly the case on your system. Look at the python console just to make sure:
box# python
Python 2.5.2 (r252:60911, May  3 2008, 16:19:27)
[GCC 3.3.6] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.maxunicode > 65535 and 'UCS4 build' or 'UCS2 build'
UCS4 build
>>>
Now we have to find out what is the python extension to be blamed, but unfortunately I cannot do it on my own since it seems there is no such problem on my box. At least I can try to help you a bit how to investigate that.

You can use strace (system calls and signals tracer) for that. Here is the a short explanation how I built it and here you can directly download the package, or possibly if you are using the filopack system you can simply write:
$ cd /mnt/C
$ ./filopack.sh --download strace-4.5.15
$ ./filopack.sh --install strace-4.5.15
... and (hopefully) you are prepared.

To replicate a scenario similar to yours, I can run the deluge daemon without the LD_PRELOAD prefix, in such case I am also missing some symbols:
box# /usr/bin/deluged
box# /usr/bin/python: can't resolve symbol '__cxa_pure_virtual'
So now I can try to trace all the system calls:
box# strace /usr/bin/deluged
...
stat64("/mnt/C/sys/root/.config/deluge", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
fork()                                  = 17025
_exit(0)                                = ?
Process 17020 detached
box# /usr/bin/python: can't resolve symbol '__cxa_pure_virtual'
It seems that the loader problem was in another instance of deluged process. Let's look at strace help to see what we can do about it:
box# strace -h
usage: strace [-dffhiqrtttTvVxx] [-a column] [-e expr] ... [-o file]
              [-p pid] ... [-s strsize] [-u username] [-E var=val] ...
              [command [arg ...]]
   or: strace -c [-e expr] ... [-O overhead] [-S sortby] [-E var=val] ...
              [command [arg ...]]
-c -- count time, calls, and errors for each syscall and report summary
-f -- follow forks, -ff -- with output into separate files
-F -- attempt to follow vforks, -h -- print help message
-i -- print instruction pointer at time of syscall
-q -- suppress messages about attaching, detaching, etc.
-r -- print relative timestamp, -t -- absolute timestamp, -tt -- with usecs
-T -- print time spent in each syscall, -V -- print version
-v -- verbose mode: print unabbreviated argv, stat, termio[s], etc. args
-x -- print non-ascii strings in hex, -xx -- print all strings in hex
-a column -- alignment COLUMN for printing syscall results (default 40)
-e expr -- a qualifying expression: option=[!]all or option=[!]val1[,val2]...
   options: trace, abbrev, verbose, raw, signal, read, or write
-o file -- send trace output to FILE instead of stderr
-O overhead -- set overhead for tracing syscalls to OVERHEAD usecs
-p pid -- trace process with process id PID, may be repeated
-s strsize -- limit length of print strings to STRSIZE chars (default 32)
-S sortby -- sort syscall counts by: time, calls, name, nothing (default time)
-u username -- run command as username handling setuid and/or setgid
-E var=val -- put var=val in the environment for command
-E var -- remove var from the environment for command
Now we can give it another try, this time with the -f option:
box# strace -f /usr/bin/deluged
...
...
close(7)                                = 0
munmap(0x40006000, 4096)                = 0
open("./libbz2.so.1.0", O_RDONLY)       = -1 ENOENT (No such file or directory)
open("/mnt/C/sys/lib//libbz2.so.1.0", O_RDONLY) = 7
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40006000
read(7, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0$\21\0\000"..., 4096) = 4096
old_mmap(NULL, 57344, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40e57000
old_mmap(0x40e57000, 52556, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 7, 0) = 0x40e57000
old_mmap(0x40e64000, 3712, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 7, 0xd000) = 0x40e64000
close(7)                                = 0
munmap(0x40006000, 4096)                = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40006000
write(2, "", 0)                         = 0
write(2, "/usr/bin/python", 15/usr/bin/python)         = 15
write(2, ": can\'t resolve symbol \'", 24: can't resolve symbol ') = 24
write(2, "__cxa_pure_virtual", 18__cxa_pure_virtual)      = 18
write(2, "\'\n", 2'
)                     = 2
munmap(0x40006000, 4096)                = 0
_exit(1)                                = ?
Process 17039 detached
Most likely the libbz2.so module is requiring the unresolved __cxa_pure_virtual symbol.

Back to your problem, once we know the particular python extension requiring the 2 byte unicode version of the DecodeUTF8 method we can try to re-build or re-install it appropriately...

Part II.

Dear Filodej,

you are right! I have followed your instructions and indeed this is the problem. 
I hope that the console output after executing strace commandes would be found 
usefull for you.
Here is the complete output from executing strace, and here for the strace -f.

Thank you in advance, for helping me out :-) hope others will benefit as well.

Regards,
Christos
Good, so the relevent part of your trace is here:
$ strace -f /usr/bin/deluged
execve("/usr/bin/deluged", ["/usr/bin/deluged"], [/* 31 vars */]) = 0
...
ioctl(9, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbffeddc8) = -1 ENOTTY (Inappropriate ioctl for device)
fstat64(9, {st_mode=S_IFREG|0755, st_size=411371, ...}) = 0
open("/usr/lib/python2.5/site-packages/_xmlplus/parsers/pyexpat.so", O_RDONLY) = 10
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40006000
read(10, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0$7\0\000"..., 4096) = 4096
old_mmap(NULL, 180224, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4059c000
old_mmap(0x4059c000, 167720, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 10, 0) = 0x4059c000
old_mmap(0x405c5000, 9872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 10, 0x29000) = 0x405c5000
close(10)                               = 0
munmap(0x40006000, 4096)                = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40006000
write(2, "", 0)                         = 0
write(2, "/usr/bin/python", 15/usr/bin/python)         = 15
write(2, ": can\'t resolve symbol \'", 24: can't resolve symbol ') = 24
write(2, "PyUnicodeUCS2_DecodeUTF8", 24PyUnicodeUCS2_DecodeUTF8) = 24
write(2, "\'\n", 2'
)                     = 2
munmap(0x40006000, 4096)                = 0
_exit(1)                                = ?
Process 2748 detached
I wonder where did you get the pyexpat library since I did not build the PyXML as part of my python package. Let's look at the JoKeR's package (it seems as a good candidate):
box# wget http://mgb111.pradnik.net/addons/soft-lang/Python-25-PyXML-084.tar.gz
connected!

Length: 17,194,098 [application/x-gzip]

box# tar tzf Python-25-PyXML-084.tar.gz | grep pyexpat
./sys/lib/python2.5/site-packages/_xmlplus/parsers/pyexpat.so
./sys/lib/python2.5/site-packages/_xmlplus/sax/drivers/drv_pyexpat.py
./sys/lib/python2.5/site-packages/_xmlplus/sax/drivers/drv_pyexpat.pyc
./sys/lib/python2.5/site-packages/_xmlplus/sax/drivers2/drv_pyexpat.py
./sys/lib/python2.5/site-packages/_xmlplus/sax/drivers2/drv_pyexpat.pyc
./sys/lib/python2.5/test/output/test_pyexpat
./sys/lib/python2.5/test/test_pyexpat.py
./sys/lib/python2.5/test/test_pyexpat.pyc
./sys/lib/python2.5/test/test_pyexpat.pyo
./sys/lib/python2.5/lib-dynload/pyexpat.so
Bingo! You probably did not remove the JoKeR's installation and just extract my python package over it.
Since I have built python with different unicode settings that JoKeR (it is nothing like Microsoft's intended incompatibilities, I swear I had no idea about it at the time ;-) now you are experiencing the missing symbol(s).

Just to make sure I am right in my assumption, I try to replicate your problem on my box. Let's extract the _xmlplus directory from JoKeR's package:
box# tar xzvf Python-25-PyXML-084.tar.gz ./sys/lib/python2.5/site-packages/_xmlplus/*
...
Now I can try to do some python xml parsing and see what happens:
box# strace -o st.txt /usr/bin/python
Python 2.5.2 (r252:60911, May  3 2008, 16:19:27)
[GCC 3.3.6] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import xml.dom.minidom
>>> xml.dom.minidom.parseString( 'test')
/usr/bin/python: can't resolve symbol 'PyUnicodeUCS2_DecodeUTF8'
Great, now I am in the same trouble as you ;-)
Let's look at the trace log:
box# tail -n 20 st.txt
stat64("/usr/lib/python2.5/site-packages/_xmlplus/parsers/pyexpat", 0xbfff4a4c) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.5/site-packages/_xmlplus/parsers/pyexpat.so", O_RDONLY|O_LARGEFILE) = 5
ioctl(5, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfff49e8) = -1 ENOTTY (Inappropriate ioctl for device)
fstat64(5, {st_mode=S_IFREG|0755, st_size=411371, ...}) = 0
open("/usr/lib/python2.5/site-packages/_xmlplus/parsers/pyexpat.so", O_RDONLY) = 6
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40006000
read(6, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0$7\0\000"..., 4096) = 4096
old_mmap(NULL, 180224, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40310000
old_mmap(0x40310000, 167720, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 6, 0) = 0x40310000
old_mmap(0x40339000, 9872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0x29000) = 0x40339000
close(6)                                = 0
munmap(0x40006000, 4096)                = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40006000
write(2, "", 0)                         = 0
write(2, "/usr/bin/python", 15)         = 15
write(2, ": can\'t resolve symbol \'", 24) = 24
write(2, "PyUnicodeUCS2_DecodeUTF8", 24) = 24
write(2, "\'\n", 2)                     = 2
munmap(0x40006000, 4096)                = 0
_exit(1)                                = ?
and try to launch the deluge daemon:
box# deluged.sh
box# /usr/bin/python: can't resolve symbol 'PyUnicodeUCS2_DecodeUTF8'
Ok, now I am pretty sure this is exactly your situation. Let's remove the _xmlplus package again, we cannot use it anyway, since it is built incompatibly with the rest of the python framework:
box# rm -d -r ./sys/lib/python2.5/site-packages/_xmlplus/
Now let's try the xml parsing and launch the deluge daemon again:
box# python
Python 2.5.2 (r252:60911, May  3 2008, 16:19:27)
[GCC 3.3.6] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import xml.dom.minidom
>>> xml.dom.minidom.parseString( '<a>test</a>')
<xml.dom.minidom.Document instance at 0x4027940c>
>>>

box# deluged.sh
box# 
Everything works again.

The removal of the _xmlplus pyhon package should help in your case as well, but note that there could be other incompatible files. The ultimate solution would be to remove both python packages (mine and JoKeR's) and install just mine again. In case you needed the PyXML package (which seems unlikely to me, since it does not work for you anyway) I could sometimes build one compatible with 4-byte unicode.

Unresolved 'g_assertion_message_expr' symbol

Q:

Hello again,

I have erased everything from my box in order to perform a clean installation based on your instructions. 
Unfortunately, this didn't go as it was expected.
Initially I installed Deluge and all of the dependencies, apart from setuptools. 
This was due to the fact that I couldn't run python cause there was a library .so-something missing 
- unfortunately I did not write down the name.
In order to fix this (and hopefully it was fixed) I downloaded & installed Joker's "minimal-config" & "libs-essential".

Now when I run:
LD_PRELOAD="/usr/lib/libssl.so.0.9.7 /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0 /usr/lib/libgcc_s.so" deluged -d

I am getting the following error:
/usr/bin/python: can't resolve symbol 'g_assertion_message_expr'
...
regards,
Christos

A:

First I googled a bit and found that the g_assertion_message_expr symbol should be present in the libglib-2.0.so library. It seemed little bit weird that you are experiencing such error, since this library is listed in my list of dependencies. It is clear that it needs a deeper analysis...

First let's make sure the libglib library is really present on the system:
box# ls -l /mnt/C/sys/lib/libglib*
-rwxr-xr-x 1 root root     878 2008-06-30 00:06 /mnt/C/sys/lib/libglib-2.0.la
lrwxrwxrwx 1 root root      23 2008-12-03 23:04 /mnt/C/sys/lib/libglib-2.0.so -> libglib-2.0.so.0.1702.0
lrwxrwxrwx 1 root root      23 2008-12-03 23:04 /mnt/C/sys/lib/libglib-2.0.so.0 -> libglib-2.0.so.0.1702.0
-rwxr-xr-x 1 root root 2631967 2008-04-17 12:34 /mnt/C/sys/lib/libglib-2.0.so.0.1400.0
rwxr-xr-x 1 root root 1131687 2008-06-30 00:06 /mnt/C/sys/lib/libglib-2.0.so.0.1702.0
... as can be seen there are two of them but just one of them is linked. This fact could be related to your problem.

We can learn what is present in my glib package:
box# ./filopack.sh --download glib-2.17.2                                 
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Downloading package glib-2.17.2 from http://filodej.ic.cz ...
connected!

Length: 21,692 [text/plain]
connected!

Length: 1,947,202 [application/x-tar]

box# tar tjvf glib-2.17.2.tar.bz2 | grep libglib
-rwxr-xr-x 0/0   1131687 2008-06-30 00:06:55 sys/lib/libglib-2.0.so.0.1702.0
lrwxrwxrwx 0/0         0 2008-06-30 00:06:55 sys/lib/libglib-2.0.so.0 -> libglib-2.0.so.0.1702.0
lrwxrwxrwx 0/0         0 2008-06-30 00:06:55 sys/lib/libglib-2.0.so -> libglib-2.0.so.0.1702.0
-rwxr-xr-x 0/0       878 2008-06-30 00:06:55 sys/lib/libglib-2.0.la
So we see that the 2.17.2 version comes from my package. What about the other one?

You write that you have downloaded & installed Joker's "minimal-config" & "libs-essential", so let's look at that:
box# wget http://mgb111.pradnik.net/addons/libs-essential-20080502.tar.bz2
connected!

Length: 13,454,268 [application/x-bzip2]

box# tar tjvf libs-essential-20080502.tar.bz2 | grep libglib
-rwxr-xr-x 0/0       834 2008-04-17 12:34:55 ./sys/lib/libglib-2.0.la
lrwxrwxrwx 0/0         0 2008-05-10 20:41:07 ./sys/lib/libglib-2.0.so -> libglib-2.0.so.0.1400.0
lrwxrwxrwx 0/0         0 2008-05-10 20:41:07 ./sys/lib/libglib-2.0.so.0 -> libglib-2.0.so.0.1400.0
-rwxr-xr-x 0/0   2631967 2008-04-17 12:34:55 ./sys/lib/libglib-2.0.so.0.1400.0
... ok, so another one (2.14.0 version) is from JoKeR's package.

Let's dig a little bit deeper, what about symbols?
box# nm --dynamic /mnt/C/sys/lib/libglib-2.0.so.0.1702.0 | grep g_assertion_message_expr
0006409b T g_assertion_message_expr
box# nm --dynamic /mnt/C/sys/lib/libglib-2.0.so.0.1400.0 | grep g_assertion_message_expr
so it seems that the g_assertion_message_expr symbol was introduced right between 14.0 and 17.2 versions (or possibly the lack of it in the older version is caused by different build configuration). In any case the problem was caused by the fact that you extracted JoKeR's essential-libs package AFTER my glib package and thus effectively downgraded the glib from 2.17.2 version down to 2.14.0.

Sorry for that complication, I built my system on top of JoKeR's minimal-config and libs-essential and I mention it in the post about the box configuration; but it is sure my fault that I did not mention it explicitly among the deluge dependencies. I am doing it now to prevent further confusion.

That's all for now. You can now redirect the symlinks (libglib-2.0.so and libglib-2.0.so.0) by hand or better re-install my glib package, supposedly it should fix your problem. As can be seen such subtle details like installation order can sometimes make a big difference :-)

Missing bash shell

Q:

...
Anyway, I then ran the "./filopack.sh --download deluge-1.0.5" line, and I got this: "-ash: ./filopack.sh: not found." Now, that's not good, right?

Any idea what it is that I'm doing wrong please?
...

A:

In a bash session with bash shell installed on the system the filopack script is running fine:
box# ./filopack.sh
    
Stupid packaging system (version 0.3.7)

Usage:
 
  filopack.sh [OPTIONS] [] []

Basic usage:
  ...
When we look at the first line of the script there is bash shell specified as an interpreter:
box# head -n1 ./filopack.sh
#!/bin/bash
Let's look where is the bash shell installed:
box# which bash
/usr/bin/bash
The /bin directory is on a read-only file system, but /bin/bash is just a symlink to /usr/bin/bash:
box# ls -l /bin/bash 
lrwxrwxrwx 1 root root 19 2008-05-21 13:40 /bin/bash -> /mnt/C/sys/bin/bash
Let's try to "delete" the bash shell for a moment:
box# mv /usr/bin/bash{,.del}
Now when we try to run the filopack script:
 
box# ./filopack.sh           
-bash: ./filopack.sh: /bin/bash: bad interpreter: No such file or directory
The bash shell (we are still running the bash session even if the bash executable is removed) is sufficiently verbose in its error messages so the cause of the problem is obvious. Let's try the same in ash shell:
box# ash
box$ cd /mnt/C/
box$ ./filopack.sh
ash: ./filopack.sh: not found
Ok now we have replicated your problem. You have to install the bash shell in order to run the filopack script.

Read more...

2008/11/24

WMU-6500FS - Deluge torrent 1.0.5 (part IV)

Screenshots

Finally I found time to build the full version of deluge bittorrent client (version 1.0.5) and create a package for it.

Deluge 1.0.5 GTK client
(running on CoLinux with Debian 4.0, XMing server running on WinXP) Deluge 1.0.5 GTK client
(running on WinXP) Deluge 1.0.5 GTK client
(running on Ubuntu 8.04)

Build result

[binary] [file list]
Prerequisites: [JoKeR's minimal config] [JoKeR's essential libs]
For filopack system: [JoKeR's bash shell] [JoKeR's busybox extended]
Dependencies: [gcc libraries] [glib] [readline] [boost] [python] [setuptools] [DBus] [DBus-glib] [DBus-python] [pygobject] [pyxdg]
GUI dependencies: [X.Org] [Gtk+] [pycairo] [pygtk] [atk] [glade]
Edit: Regarding to this post it seems that [pygobject] and [pyxdg] packages are needed even for non-gui mode.

Overview

First I'll demonstrate the shortest path (given all the needed dependencies are already installed). Then I describe the build and package creation process, since I feel that such information can be helpful when something goes wrong.

Download & Install

We can utilize the filopack system for that purpose. In case we do not have it we can simply download it (and make executable) as follows:
box# cd /mnt/C
box# wget http://filodej.ic.cz/filopack/filopack.sh
box# chmod 755 filopack.sh
First action we have to do is to uninstall the previous Deluge version (assumed we have both Deluge 0.9.03 and ./filopack.sh installed on the box):
box# cd /mnt/C
box# ./filopack.sh --remove deluge-0.9.03
Configuration file .filopack/.config file found and used
Sure to remove deluge-0.9.03 locally at /mnt/C (y/n)?y
...
Now we can download the package:
box# cd /mnt/C
box# ./filopack.sh --download deluge-1.0.5
[root@storage C]# ./filopack.sh --download deluge-1.0.5
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Downloading package deluge-1.0.5 from http://filodej.ic.cz ...
connected!

Length: 68,332 [text/plain]
connected!

Length: 15,645,450 [application/x-tar]
And the last step is to install it:
box# ./filopack.sh --install deluge-1.0.5
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Sure to unpack deluge-1.0.5 locally at /home/petr (y/n)?
...
Now, we are done (if we are lucky ;-)

I added scripts (deluge.sh and deluged.sh) which we can run in place of original loader scripts (deluge and deluged) which are working around the weird problems with dynamic loader missing some symbols.
box# cat /mnt/C/sys/bin/deluged.sh
LD_PRELOAD="/usr/lib/libssl.so.0.9.7 /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0" deluged
So if you already have the /mnt/C/sys/bin/ directory in your PATH now instead of:
box# deluged
/usr/bin/python: can't resolve symbol '__cxa_pure_virtual'
... you can simple type:
box# deluged.sh
It is great that the daemon mode is now supported and we can run the GTK GUI remotely (even so the demands of python runtime are relatively high for the box).
In following figures you can see the clients running on Ubuntu and WinXP both connected to the daemon running on the box. In previous figure there is also the client running on the box - it can be seen that its progress controls are way behind the real time and even the PC running client does not receive much information from the daemon. In such situations everything becomes reaaaalllllly sloooowwww as the box can't catch the train.

It is obvious that it is not a good idea to run GTK client directly on the box, since it pushes its HW resources really to its limits, but in is nice that it (no matter how slowly) works.

Build & Pack

Everything necessary for successful build was already described in previous three parts of this series, this part just collects the information and describes the whole building process step-by-step.

Same as in the previous case we have to uninstall the old Deluge version (0.9.03)
dev# cd /mnt/C
dev# ./filopack.sh --remove deluge-0.9.03
Configuration file .filopack/.config file found and used
Sure to remove deluge-0.9.03 locally at /mnt/C (y/n)?
...
Now we are ready to start the building process. First thing we have to do is to create a timestamp file. With help of this file the packaging system determines what files are included to the resulting package.
dev# ./filopack.sh --init  deluge-1.0.5
Configuration file .filopack/.config file found and used
Timestamp written to file .filopack/deluge-1.0.5.ts
Now we can download and extract the sources.
dev# cd /usr/local/src/
dev# wget http://download.deluge-torrent.org/source/1.0.5/deluge-1.0.5.tar.bz2
...
dev# tar xjvf deluge-1.0.5.tar.bz2
dev# cd /usr/local/src/deluge-1.0.5
We have to do the following steps in order to make a successful build:
(For explanations see previous parts of this series)
Add proper paths to boost header files and libraries
dev# export CFLAGS=-I/mnt/C/sys/include/boost-1_35
dev# export LDFLAGS=-L/mnt/C/sys/lib
Add missing IPV6 related macro to socket.hpp header file:
dev# nano libtorrent/include/libtorrent/socket.hpp
    add:
      #ifndef IPV6_V6ONLY
      #define IPV6_V6ONLY 26
      #endif
Disable the memory debugging (file libtorrent/src/memdebug.cpp) from the build. You can:
- either modify the Makefile
- or clear the memdebug.cpp
- or use the preprocessor condition:
      #if 0
        ...
      #endif
Now we are ready to start the build and installation:
dev# python setup.py build 
...
dev# python setup.py install --prefix=/mnt/C/sys/
...
If the build and installation succeeded, we are almost done. Now, when we try to start the deluge daemon, the dynamic loader is missing some symbols (it was same with previous version, I am possibly doing something wrong but do not know what).
dev# deluged
dev# /usr/bin/python: can't resolve symbol '__cxa_pure_virtual'
At least we have the workaround - library preload, with following triar and error method we get the set of libraries to be rpe-loaded:
dev# LD_PRELOAD=/usr/lib/libstdc++.so.6 deluged
dev# /usr/bin/python: can't resolve symbol 'BIO_free'

dev# LD_PRELOAD="/usr/lib/libstdc++.so.6 /usr/lib/libssl.so.0.9.7" deluged
dev# /usr/bin/python: can't resolve symbol '_ZN5boost6system18get_posix_categoryEv'

dev# LD_PRELOAD="/usr/lib/libssl.so.0.9.7 /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0" deluged
OK, now it works. We can prepare the loader script with previous command-line:
dev# echo LD_PRELOAD=\"/usr/lib/libssl.so.0.9.7 /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0\" deluged \"\$@\" > /mnt/C/sys/bin/deluged.sh
dev# chmod +x /mnt/C/sys/bin/deluged.sh 
Now (if we have the /mnt/C/sys/bin in our PATH) we can run the deluge daemon as follows:
dev# deluged.sh
For deluge client the symbol problem is the same, so we prepare a loader script as well:
dev# echo LD_PRELOAD=\"/usr/lib/libssl.so.0.9.7 /usr/lib/libboost_filesystem-gcc41-mt-1_35.so.1.35.0\" \"\$@\" deluge > /mnt/C/sys/bin/deluge.sh
dev# chmod +x /mnt/C/sys/bin/deluge.sh 
Now we try to launch the GUI (GTK client) but some problems still remain:
dev# deluge.sh
...
  File "/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/common.py", line 142, in get_logo
    size, size)
gobject.GError: Unrecognized image file format
We already solved this problem with previous version, so we know the receipt:
dev# sed -i 's/deluge.svg/deluge.png/g' /mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/common.py
Let's try again:
dev# deluge.sh
...
  self.window.show()
/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/ui/gtkui/mainwindow.py:95: GtkWarning: gtk_icon_theme_load_icon: assertion `GTK_IS_ICON_THEME (icon_theme)' failed
  self.window.show()
Segmentation fault
We forgot the set up the DISPLAY environment variable properly (Actually I do not know why it crashes in such case, but the truth is that proper DISPLAY setup solves the problem):
dev# export DISPLAY=<x-server-ip>
One more thing we have to do is to let python know the path to the deluge egg:
dev# cd /mnt/C/sys/lib/python2.5/site-packages
dev# echo "./deluge-1.0.5-py2.5-linux-i686.egg" > deluge.pth 
Finally we are ready to create the deluge-1.0.5 package:
dev# cd /mnt/C
dev# ./filopack.sh --pack deluge-1.0.5

Remote install

Now we have the package created locally on the "mirror" system (colinux debian in my case). But we have to get it to the box. The filopack script can help us with this task as well.

First we have to uninstall the previous deluge version from the box. We can definitely do it locally on the box itself, but we can also do it remotely:
dev# ./filopack.sh -r --remove deluge-0.9.03
Configuration file .filopack/.config file found and used
Sure to remove deluge-0.9.03 remotely at root@storage:/mnt/C (y/n)?y
...
If we want to make sure that the previously installed software is gone, we can look at the package summary (unfortunately this feature is supported only locally). So from the box shell we can do following:
box# ./filopack.sh --summary
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)

Filodej package summary:
Package name                          downloaded [#installed/#total]
--------------------------------------------------------------------
Mesa-7.0.3                                no           [   0/  49]
MesaDemos-7.0.3                           no           [ 129/ 129]
atk-1.22.0                                no           [ 160/ 160]
audiofile-0.2.6                           no           [   0/  13]
binutils-2.18                             no           [ 140/ 140]
boost-1.35.0                              no           [5195/5195]
cairo-1.6.4                               no           [  76/  76]
code2html-0.9.1                           no           [   2/   2]
cvstrac-2.0.1-3                           no           [   3/   3]
dbus-1.2.1                                no           [  38/  38]
dbus-glib-0.76                            no           [  32/  32]
dbus-python-0.82.4                        no           [  66/  66]
deluge-0.9.03                        no          [   0/ 673]
didiwiki-0.5                              no           [   1/   1]
faad2-2.6.1                               no           [   0/   8]
...
Now we are back in the "mirror" system console and we can upload the (newly created) package to the box:
dev# ./filopack.sh -r --upload deluge-1.0.5 
Configuration file .filopack/.config file found and used
Sure to upload deluge-1.0.5 to root@storage:/mnt/C (y/n)?y
root@storage's password: 
deluge-1.0.5.tar.bz2                                                100%   15MB 318.3KB/s   00:48    
And perform the remote installation as well:
dev# ./filopack.sh -r --install deluge-1.0.5
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Module deluge-1.0.5 not found in package list - use 'tar.bz2' extension
Sure to unpack deluge-1.0.5 remotely at root@storage:/mnt/C (y/n)?y
root@storage's password: 
...
Deluge daemon (deluged.sh) now works OK on the box.

The only remaining problem is following. When we run the GTK GUI client (deluge.sh) we get following error:
box# deluge.sh
...
gobject.GError: Failed to open file '/tmp/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/data/pixmaps/inactive16.png': No such file or directory
When we look to the /tmp/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/data/pixmaps/ directory there is not a inactive16.png there is just inactive16.png0000644 file and the same is with checking16.png file.
It seems it is a bug in tar implementation, the file permissions are "leaking out" to the name (see this issue description or this forum).

The solution is easy in this case:
box# mv /tmp/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/data/pixmaps/inactive16.png{0000644,}
box# mv /tmp/mnt/C/sys/lib/python2.5/site-packages/deluge-1.0.5-py2.5-linux-i686.egg/deluge/data/pixmaps/checking16.png{0000644,}
When you run the deluge and it looks similar like this: ... then there are some missing links to font directories:
box# mkdir /usr/share/fonts
box# ln -sn /mnt/C/sys/X11/lib/X11/fonts/OTF /usr/share/fonts/X11-OTF
box# ln -sn /mnt/C/sys/X11/lib/X11/fonts/TTF /usr/share/fonts/X11-TTF

Setup

Now (after the update of the python package) we are able to run the console client right on the box: We can read the configuration options:
>  configs             

* add_paused: False
* allow_remote: False
* auto_managed: True
* autoadd_enable: False
* autoadd_location: C
* compact_allocation: False
* config_location: /mnt/C/sys/root/.config/deluge
...
* daemon_port: 58846
...
If we want access the daemon remotely, we can set it up as follows (of course it is possible to change the port as well):
> config-set allow_remote True

* Configuration value successfully updated.

>  quit 

Thanks
box#
Now we have to restart the daemon (if it is running):
box# killall deluged
box# deluged.sh
... and we are set up to access the daemon remotely (from PC running either Linux or Windows) as can be seen on following screenshots:
Deluge 1.0.5 GTK client
(running on WinXP) Deluge 1.0.5 GTK client
(running on Ubuntu 8.04)
That's all for now.

Read more...

2008/11/21

Simple packaging system

Overview

In this post I want to describe the filopack bash script, very simple (and stupid) packaging system I have prepared to ease the package manipulation. It is implemented specifically as a helper for packages presented here, but it could be used for any other system as well. It aims to help potential users with the installation process. It would be definitely better to base such a system on a well established package system like for example aptitude, rather than create my own (primitive) system from scratch, but unfortunately I had no time to study the aptitude yet (probably it would be much easier than I imagine) and started to practically learn the bash scripting instead ;-)

Following examples should explain the basic usage of the filopack packaging system. This tool serves for package enumeration, downloading, installation and un-instalation as well as package creation and uploading. However it does not handle dependencies at all, it just simplifies the installation routine, YOU have to know what packages you need.

The system is suitable for usage on the box itself as well as on the chroot-ed "mirror" system. Also besides the downloading and installation of pre-built packages, it can be used for a creation user-built packages from source code. I consider the latter two scenarios as "advanced" and will cover separately from the "basic" usage scenario - it means downloading and installing pre-built packages directly to the box.

Basic usage

In this scenario a user just wants to download and extract a package directly to the WMU box. This section will cover this scenario step by step.

Bash shell

In order to use the filopack script you need to have the bash shell installed on the system. You can use the JoKeR's prebuilt package:
box# cd /mnt/C/
box# wget http://mgb111.pradnik.net/addons/soft-shells/bash-3.2.33-3.tar.gz
box# tar xzvf bash-3.2.33-3.tar.gz
...
Now you can possibly set it as root's and users' shell:
box# nano /mnt/C/sys/etc/env-ng 
...

# change root shell to bash (if exist)
ROOTSHELL=1

# change users shell to bash (if exist)
USERSHELL=1


Busybox extended

box# cd /mnt/C/
box# wget http://mgb111.pradnik.net/addons/busybox2-extended.tar.gz
box# tar xzvf busybox2-extended.tar.gz
...
Make sure you have the /usr/bin (symlink to /mnt/C/sys/bin) in your PATH environment variable.

Download

The script can be downloaded from the Packages section or you can get it from console via wget as follows (of course you have to make it executable via chmod command):
box# cd /mnt/C
box# wget http://filodej.ic.cz/filopack/filopack.sh
box# chmod 755 filopack.sh

Usage

Now we have the script ready to be used. When we start it with no arguments it just prints the command line documentation. It covers "basic" usage which is suitable for most users.
box# ./filopack.sh
Stupid packaging system (version 0.3.6)

Usage:
 
  filopack.sh [OPTIONS] [<action>] [<package-name>]

Basic usage:

  Basic actions: 
    (typically used when you just download and install/uninstall pre-built packages)

    --help     ... prints detailed help message
    --version  ... prints version of the script
    --config   ... perform system (re)configuration
    --packages ... lists all packages available for download
    --download ... downloads a selected package
    --print    ... just prints the files of selected package
    --install  ... unpacks (installs) the package  
    --delete   ... deletes the given package archive
    --remove   ... removes (uninstalls) the package
    --summary  ... generates a summary of installed packages
In case one wants to create own packages or perform remote package installation there is also the "advanced" mode supported. I'll come back to this mode later.

Initial configuration

Now we have to configure the package destination directory (/mnt/C by default) as well as box IP/hostname and username. Latter two parameters are used just for remote packaging and so "basic" users can simply ignere it and leave the default values. With --config action we can re-configure system any time later.
box# ./filopack.sh --config
Destination address and user not configured (.filopack/.config file not found)
Insert destination directory [/mnt/C]: 
Insert box IP address or hostname (remote packaging only)[storage]: 
Insert box user account (remote packaging only) [root]: 
Settings stored to .filopack/.config file


Package listing

Now we are ready to retrieve the list of available packages. Individual package names are used as identifiers for package installation/uninstallation.
box# ./filopack.h --packages
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Available packages:
Mesa-7.0.3
MesaDemos-7.0.3
atk-1.22.0
audiofile-0.2.6
binutils-2.18
boost-1.35.0
cairo-1.6.4
code2html-0.9.1
cvstrac-2.0.1-3
dbus-1.2.1
dbus-glib-0.76
dbus-python-0.82.4
deluge-0.9.03
didiwiki-0.5
faad2-2.6.1
file-4.24
flac-1.2.1
fontconfig-2.6.0
gcc-4.1.2-sjlj
git-1.5.5.3
glade3-3.4.5
glib-2.17.2
gperf-3.0.3
gtk+-2.13.3
highlight-2.6.10
htop-0.8
httpd-2.2.8
jasper-1.701.0-2
jpeg-6b
libao-0.8.8
libglade-2.6.2
libid3tag-0.15.1b
libmikmod-3.2.0-beta2
libogg-1.1.3
libvorbis-1.2.0
libxcb-1.1
libxml2-2.6.30
libxslt-1.1.22
lsof-4.78
mod_python-3.3.1
mpd-0.13.2
pango-1.21.3
pycairo-1.4.12
pygobject-2.14.2
pygtk-2.12.1
python-2.5.2
pyxdg-0.15
source-highlight-1.9
sqlite-3.5.8
strace-4.5.15
streamripper-1.63.1
subversion-1.5.0
tiff-3.8.2
x11-7.1
xcb-util-0.2

Package retrieval

Let's say we are interested in the htop-0.8 package and we want to learn the contents.
box# ./filopack.sh --print htop-0.8
.filopack/htop-0.8.lst: No such file or directory
Unfortunately in this version we have to download the whole package in order to learn the contents (Maybe it will be changed in a future version).
We can use the following action in order to download the package:
box# ./filopack.sh --download htop-08 
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Package htop-08 not available (try to use --packages switch)
Once more, now with the correct package name:
box# ./filopack.sh --download htop-0.8
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Downloading package htop-0.8 from http://filodej.ic.cz ...
connected!

Length: 102 [text/plain]
connected!

Length: 224,259 [application/x-gtar]
Now we are ready to print the package contents:
box# ./filopack.sh --print htop-0.8
sys/bin/htop
sys/share/man/man1/htop.1
sys/share/applications/htop.desktop
sys/share/pixmaps/htop.png

Package installation
box# ./filopack.sh --install htop-0.8
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Sure to unpack htop-0.8 locally at /mnt/C (y/n)?y
sys/bin/htop
sys/share/man/man1/htop.1
sys/share/applications/htop.desktop
sys/share/pixmaps/htop.png
Now if you want to spare some disk space, you can delete the package archive file:
box# ./filopack.sh --delete htop-0.8 
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)
Are you sure to delete the htop-0.8.tgz archive (y/n)?y

Package removal

If later on you change your mind and decide to remove the package, you can do it as follows:
box# ./filopack.sh --remove htop-0.8
Configuration file .filopack/.config file found and used
Sure to remove htop-0.8 locally at /mnt/C (y/n)?y
... and now if you try to uninstall the same package again, operation fails (since there are no such files).
box# ./filopack.sh --remove htop-0.8
Configuration file .filopack/.config file found and used
Sure to remove htop-0.8 locally at /mnt/C (y/n)?y
rm: cannot remove `sys/bin/htop': No such file or directory
rm: cannot remove `sys/share/man/man1/htop.1': No such file or directory
rm: cannot remove `sys/share/applications/htop.desktop': No such file or directory
rm: cannot remove `sys/share/pixmaps/htop.png': No such file or directory
You may also want to get a summary what packages are available and what package archives are downloaded and/or installed.

Package summary
box# ./filopack.sh --summary
Configuration file .filopack/.config file found and used
Retrieving package index... (Connecting to http://filodej.ic.cz)

Filodej package summary:
Package name                          downloaded [#installed/#total]
--------------------------------------------------------------------
Mesa-7.0.3                               yes           [  49/  49]
MesaDemos-7.0.3                          yes           [ 129/ 129]
atk-1.22.0-new                           yes           [ 160/ 160]
binutils-2.18-latest                     yes           [ 140/ 140]
boost-1.35.0-gcc4.1.2                    yes           [5195/5195]
boost-1.35                               yes           [5133/5195]
cairo-1.6.4                              yes           [  76/  76]
code2html-0.9.1                          yes           [   2/   2]
cvstrac-2.0.1-3                           no           [   3/   3]
dbus-1.2.1                               yes           [  38/  41]
dbus-glib-0.76                           yes           [  32/  32]
dbus-python-0.82.4                       yes           [  66/  66]
deluge-0.9.03-latest                     yes           [ 651/ 674]
deluge-torrent-0.5.9.3-new               yes           [   0/1543]
deluge-torrent-0.5.9.3                   yes           [   0/  99]
didiwiki-0.5                              no           [   1/   1]
file-4.24                                yes           [  11/  11]
fontconfig-2.6.0                         yes           [ 429/ 429]
gcc-4.1.2-sjlj                           yes           [  99/ 549]
git-1.5.5.3                               no           [ 228/ 228]
glade3-3.4.5                             yes           [ 323/ 323]
glib-2.17.2                              yes           [ 501/ 501]
gperf-3.0.3                              yes           [   4/   4]
gtk+-2.10.14                             yes           [1155/1155]
gtk+-2.13.3                              yes           [1233/1233]
highlight-2.6.10                          no           [ 206/ 206]
htop-0.8                                  no           [   0/   4]
httpd-2.2.8                               no           [ 174/ 174]
jasper-1.701.0-2                         yes           [  27/  27]
jpeg-6b                                  yes           [  10/  10]
libglade-2.6.2                           yes           [  34/  34]
libogg-1.1.3                              no           [  83/  83]
libvorbis-1.2.0                           no           [ 125/ 125]
libxcb-1.1                               yes           [ 160/ 160]
libxml2-2.6.30                           yes           [ 322/ 322]
libxslt-1.1.22                           yes           [ 142/ 142]
lsof-4.78                                yes           [   1/   1]
mod_python-3.3.1                          no           [  26/  26]
pango-1.21.3                             yes           [ 146/ 146]
pycairo-1.4.12                           yes           [   7/   7]
pygobject-2.14.2                         yes           [  44/  44]
pygtk-2.12.1                             yes           [ 647/ 649]
python-2.5.2                             yes           [3648/3648]
pyxdg-0.15                               yes           [  25/  25]
source-highlight-1.9                      no           [  49/  49]
sqlite-3.5.8                              no           [   8/   8]
strace-4.5.15                            yes           [   3/   3]
streamripper-1.63.1                       no           [  13/  13]
subversion-1.4.6-py-pl                    no           [ 211/ 211]
subversion-1.5.0                         yes           [ 319/ 319]
tiff-3.8.2                                no           [ 231/ 231]
x11-7.1                                  yes           [7791/7791]
xcb-util-0.2                              no           [  70/  70]
So that is all for now. In a following post I extend the tutorial with examples of "advanced" usage.

Read more...