So I decided to see if I could get it working with the Debian supplied Python interpreter. One problem is that there’s a shared object file written in C which is part of the pkg application. That file is a 32 bit ELF too. So we’re going to download that source and recompile it for 64 bit. If you’re on a 32 bit system, you can skip that step (although it doesn’t hurt to do it anyway). First, we need to install the following packages:

apt-get install python2.5 python2.5-dev gcc python-cherrypy python-mako python-openssl python-ply python-pycurl python-simplejson

Now download the source for the _actions.c file from here (link to webpage, press download in the top).

Compile it with the following command:
gcc -I/usr/include/python2.5 -shared -fpic -O2 _actions.c -o _actions.so

Keep the resulting .so file, we’re going to replace it once we’ve downloaded GlassFish. Which is the next step, actually. So download GlassFish and set it up somewhere. I downloaded the tarball and unzipped it into /opt/glassfish.

The resulting directory contains several directories, including a pkg and a glassfish directory. The /opt/glassfish/glassfish directory is the actual GlassFish application. The pkg directory contains the pkg tool which is used to upgrade the GlassFish addons and systems and stuff (don’t ask me about the details, I’m not a Java developer, only a sysadmin). The first time you start it, it installs some stuff. So just run the following:
/opt/glassfish/bin/pkg

Next, mv the file /opt/glassfish/pkg/bin/pkg to /opt/glassfish/pkg/bin/pkg.orig. We do this since this script does all kinds of magic which we do not need. Also, it runs the pkg python code with the python2.4 interpreter that’s part of the package. We don’t want that. Let’s fix it.

Make a simple script to replace the one we moved away. I use the following, which works for me:

#!/bin/sh
python /opt/glassfish/pkg/bin/client.py

If you start it now, you’ll get an error:

$ python /opt/glassfish/pkg/bin/client.py
Traceback (most recent call last):
  File "/opt/glassfish/pkg/bin/client.py", line 60, in 
    import pkg
ImportError: No module named pkg

Ok, let’s fix that! Start with creating a directory called /opt/glassfish/pkg/custom-lib. You can change the name into anything you want, of course, as long as it’s clear that this is where you’re going to put the pkg python module. Actually, let’s do that immediatly: cp -r /opt/glassfish/pkg/vendor-packages/pkg /opt/glassfish/pkg/custom-lib

Change our script which we setup to run pkg into the following:

#!/bin/sh
PYTHONPATH="/opt/glassfish/pkg/custom-lib" python /opt/glassfish/pkg/bin/client.py

You need to have only the pkg module in there, because the PYTHONPATH variable takes precedence over the other modules installed via the Debian packages. Run the script and you’ll get a new error:

Traceback (most recent call last):
  File "/opt/glassfish/pkg/bin/client.py", line 61, in 
    import pkg.actions as actions
  File "/opt/glassfish/pkg/custom-lib/pkg/actions/__init__.py", line 144, in 
    from _actions import _fromstr
ImportError: /opt/glassfish/pkg/custom-lib/pkg/actions/_actions.so: wrong ELF class: ELFCLASS32

If you don’t get this error, you’re on a 32 bit system and you’re done! Congratulations! Otherwise, we’re going to copy the _actions.so file we compiled earlier over the one that’s packaged with GlassFish 3.0.1. Just copy it over the other file, like so: cp _actions.so /opt/glassfish/pkg/custom-lib/pkg/actions/_actions.so

Now we’re really done! You should be able to run /opt/glassfish/bin/pkg image-update now and update your currently installed GlassFish 3.0.1 with the latest modules and stuff. Awesome!

I’m in the process of creating a Debian package for GlassFish 3.0.1 which incorporated this fix. So if you’re not in a hurry or you’re reading this way after it was posted, you might want to check out debian.kumina.nl to see if the package is available.

Hope this helps someone!

There’s even a Debian package for it, but that doesn’t really contain everything I need. First of all, I need a package that runs on Etch. There’s a dependency in the Debian package (which is for Lenny) on an asciidoc of a version that’s available only in Lenny. I don’t really see why that dependency is there at all, but when removing the version, the package still works as expected.

Also, I made two patches of which one is not needed at all, but I left it in there anyway, because it doesn’t break anything. It simply adds the –innodb-file-per-table option to the invocation of the mysqld_safe daemon, but that’s not needed since it only affects MySQL when you change data. But since mysqld_safe is used to recover from logs, you won’t be changing data. I left it in there anyway. You can find the dpatch file here.

The second one might be more interesting. The mysql-server-5.0 package from Debian creates a file /etc/mysql/debian.cnf with login details for the debian-sys-maint user. This user also has enough permissions to do all the things necessary for manipulating the database when we want to make a backup with mylvmbackup. Normally, mylvmbackup needs you to give it user details for a privileged user in /etc/mylvmbackup.conf, which is a bit double, since the debian.cnf exists. So my patch adds a –defaults_file option which allows you to mention the debian.cnf. No need for more passwords in plaintext on the filesystem! You can either give this option no the commandline or in the mylvmbackup.conf in the [mysql] section. It only supports Debian style debian.cnf, though, since it searched for a [client] section in the file you mention. You can find the patch here.

A working package can be found in Kumina’s Debian repository for both etch and lenny. Please let me know if this works for you! It’s my first Perl, though, so be warned. If stuff breaks… Well… Sorry… So be sure to test it.

Tags: backup, configuration, debian, etch, lenny, mylvmbackup, mysql, package

Bart found a description of the problem and the solution at Jürgen Kreileder’s blog. Jürgen apparantly still has the repos online, which helped us resolve the problem on our machine. Much appreciated!

Tags: debian, lvm, sarge, solution, xen

Getting ColdFusionMX7 to run with the Etch supplied Apache 2.2 is a different matter. Took me about several weeks to find out (not full time, but still over 20 hours, I think). First of all, you need the modified wsconfig.jar that Adobe distributes via a knowledge base article about their hotfix. Don’t bother with their instructions, though, the mod_jrun22.so will not compile that way. Not because of missing apps or something, but because for some strange reason, the script thinks JRun is already up and running. I get an output like so:

Macromedia JRun 4.0 (Build 107948)
os.name: Linux
os.version: 2.6.18-6-xen-amd64
os.arch: i386
platform: intel-linux
Found port 2920 on host localhost
findServers(): found server coldfusion at 127.0.0.1:2920
Found JRun server coldfusion at 127.0.0.1:2920
this host is stage02:62.133.201.114
web server: Apache
web server directory: /etc/apache2
verbose connector logging: false
apialloc: false
force resource extract from jar: true
CFMX: true
mappings: .jsp,.jws,.cfm,.cfml,.cfc,.cfr,.cfswf
filter mapping prefix: false
Apache binary: /usr/sbin/apache2
Apache control script: /usr/sbin/apache2ctl
Apache apxs: true
This web server is already configured for JRun.

Not very helpful. I tried removing all files related to Apache and JRun that I could think of, including the part in /etc/apache2/httpd.conf that loads the module, but it still told me that the web server was already configured. Time for some heavier work.

First, start with unzipping the wsconfig.zip. It’ll yield a wsconfig.jar. Unzip that one too in a temporary directory, let’s say in /tmp/wsconfig. Now go to /tmp/wsconfig/controller/src. There’s a text file in there that contains the steps to install it. However, I need to install a few packages:

apt-get install apache2-prefork-dev gcc

You might need apache2-worker-dev, if you use that Apache flavour. You can check that with dpkg -l | grep apache2. You’ll find either apache2-mpm-prefork or apache2-mpm-worker. That’s your clue!

Now, execute the following commands:

sudo apxs2 -c -n jrun22 mod_jrun22.c  \
jrun_maptable_impl.c  jrun_property.c  jrun_session.c platform.c \
jrun_mutex.c jrun_proxy.c  jrun_utils.c

sudo apxs2 -i -n jrun22 mod_jrun22.la

sudo strip /usr/lib/apache2/modules/mod_jrun22.so

This will install the Apache 2.2 JRun module at /usr/lib/apache2/modules/mod_jrun22.so. Now change your Apache config accordingly, and you’re ready to go.

One thing I found, but that was just me trying to be hasty, was that you need to use:

LoadModule jrun_module /usr/lib/apache2/modules/mod_jrun22.so

Hope this helps someone.

Tags: apache22, coldfusion, coldfusionmx7, debian, etch, jrun, mod_jrun22, sarge

It is nigh impossible. I’ve tried several things. First, I added the option innodb_file_per_table. The idea was to optimize each table, so it would be rewritten into a separate file. Then delete the ibdata1 file and it’s logs and restart MySQL. This worked in a simple test setup, but failed miserably in the production environment. I have no idea what went wrong, but even when I restored the old ibdata1 and it’s logs, it wouldn’t work anymore. Chaos and mayhem all around. Joy.

So I resorted to the only other solution I could find: Make a dump of the complete database and remove all files in /var/lib/mysql, then restart MySQL and restore the dump. Not a very practical solution, but the only one that seems to work reliably.

Tags: debian, innodb, mysql

I did most of my debugging with a small perl script that I found on the HP forums. Simply edit the location and you’re ready to go. My problem was that I had no trouble at all getting it to work in another setup, but I couldn’t find at first what was different in the production setup. In the end it was very obvious (as it usually is): The network was way more complicated.

The setup I’m talking about is in a datacentre that houses several of our customers. They’re all daughter-companies from one large one, so they’re okay with sharing some infrastructure. But each company has it’s own domain name and prefers of course to use that one. Now, we name our virtual machines (we do everything virtualised) after their specific function, so a database master would be called something like dbm1.customera.com. Another customer with another database master would call the machine dbm1.customerb.com. But when we’re working in that network, it’s annoying to use those full names, so we have an internal DNS that resolves things like dbm1.custa and dbm1.custb. Just for convenience.

But that convenience is what made NFS locking break. Since the machine would report it’s hostname as being dbm1.customera.com, while it’s DNS name would be dbm1.custa. Apparantly, NFS can’t handle that very well.

Luckily, there’s an easy solution: Add a specific hostname to the NFS stat daemon. In Debian, you do that by editing /etc/default/nfs-common (after you installed the nfs-common package) and adding the following line: STATDOPTS="-n dbm1"

That worked for me. Now, I have working InnoDB tables on an NFS share.

Tags: debian, innodb, locking, mysql, nfs, statd

Apparantly, by default, the MySQL in Lenny created access rules that allow all local users to access the database without a password. These have priority over the specific user controls. A bit unexpected and I suspect others to run into this. Could make you search for a problem if you’re installing a shared webhosting machine with a local MySQL database.

Solution:
delete from mysql.user where User='';

Tags: a_bit_strange, credentials, debian, fix, lenny, mysql, permissions

Also described in Debian bugreport #438864. Took me a while to find it, so referencing it here.

Tags: debian, subversion, SubversionException, trac, workaround

Yesterday and today (well, only in the evening today), I’ve been trying to get MySQL working on an s3fs drive, but I’m afraid that’s not going to work. Well, part of it might be because I’m trying to set it up from home and not from an EC2 instance, but still. MySQL just refuses to start with a weird message:

Starting MySQL database server: mysqld . . . . . . . . . . . . . . failed!
invoke-rc.d: initscript mysql, action "start" failed.

After which it hangs, altough it seems like MySQL did start. Maybe it’s something with the initscript. I realised in the end that I don’t really need MySQL running in such a setup. Making dumps every so often should be enough for what we have in mind. So I’m going to script an automatic recovery of the database, based on a dump every 5 minutes. That should work without losing any speed.

Tags: debian, ec2, mysql, s3

As usual, you can find my files right here. I claim no copyright to the code whatsoever, since I only added about 8 lines and most of those were copied from some other programme. Use as you see fit.

• check_ldap_starttls (the programme)
• check_ldap_starttls.c (the code)

Enjoy!

Update Been fixed in the latest packages, I believe. Removed the links here.

Tags: debian, ldap, nagios, ssl