Tags: 64bit, debian, glassfish, howto, java, opensolaris, pkg, python
leave a comment
GlassFish 3.0.1′s pkg tool using Debian’s Python packages
While setting up GlassFish 3.0.1 for a customer on Debian Lenny using 64 bit machines, I ran into the problem that the update tool shipped with GlassFish (OpenSolaris‘s pkg tool) uses a Python interpreter which is part of the package. That Python interpreter however is 32 bit, which requires the ia32 libraries to be installed. Worse, it requires lididn in 32 bit, which is not part of the default Debian packages, so I had to get that one from the (very nice) Debian Multimedia repository. Although that’s a quick fix to get stuff working, we rather not use these repositories on production machines, due to security concerns and the like. Also, upgrades are easier if you only use the standard Debian repositories.
So I decided to see if I could get it working with the Debian supplied Python interpreter. One problem is that there’s a shared object file written in C which is part of the pkg application. That file is a 32 bit ELF too. So we’re going to download that source and recompile it for 64 bit. If you’re on a 32 bit system, you can skip that step (although it doesn’t hurt to do it anyway). First, we need to install the following packages:
apt-get install python2.5 python2.5-dev gcc python-cherrypy python-mako python-openssl python-ply python-pycurl python-simplejson
Now download the source for the _actions.c file from here (link to webpage, press download in the top).
Compile it with the following command:
gcc -I/usr/include/python2.5 -shared -fpic -O2 _actions.c -o _actions.so
Keep the resulting .so file, we’re going to replace it once we’ve downloaded GlassFish. Which is the next step, actually. So download GlassFish and set it up somewhere. I downloaded the tarball and unzipped it into /opt/glassfish.
The resulting directory contains several directories, including a pkg and a glassfish directory. The /opt/glassfish/glassfish directory is the actual GlassFish application. The pkg directory contains the pkg tool which is used to upgrade the GlassFish addons and systems and stuff (don’t ask me about the details, I’m not a Java developer, only a sysadmin). The first time you start it, it installs some stuff. So just run the following:
/opt/glassfish/bin/pkg
Next, mv the file /opt/glassfish/pkg/bin/pkg to /opt/glassfish/pkg/bin/pkg.orig. We do this since this script does all kinds of magic which we do not need. Also, it runs the pkg python code with the python2.4 interpreter that’s part of the package. We don’t want that. Let’s fix it.
Make a simple script to replace the one we moved away. I use the following, which works for me:
#!/bin/sh python /opt/glassfish/pkg/bin/client.py
If you start it now, you’ll get an error:
$ python /opt/glassfish/pkg/bin/client.py Traceback (most recent call last): File "/opt/glassfish/pkg/bin/client.py", line 60, inimport pkg ImportError: No module named pkg
Ok, let’s fix that! Start with creating a directory called /opt/glassfish/pkg/custom-lib. You can change the name into anything you want, of course, as long as it’s clear that this is where you’re going to put the pkg python module. Actually, let’s do that immediatly: cp -r /opt/glassfish/pkg/vendor-packages/pkg /opt/glassfish/pkg/custom-lib
Change our script which we setup to run pkg into the following:
#!/bin/sh PYTHONPATH="/opt/glassfish/pkg/custom-lib" python /opt/glassfish/pkg/bin/client.py
You need to have only the pkg module in there, because the PYTHONPATH variable takes precedence over the other modules installed via the Debian packages. Run the script and you’ll get a new error:
Traceback (most recent call last): File "/opt/glassfish/pkg/bin/client.py", line 61, inimport pkg.actions as actions File "/opt/glassfish/pkg/custom-lib/pkg/actions/__init__.py", line 144, in from _actions import _fromstr ImportError: /opt/glassfish/pkg/custom-lib/pkg/actions/_actions.so: wrong ELF class: ELFCLASS32
If you don’t get this error, you’re on a 32 bit system and you’re done! Congratulations! Otherwise, we’re going to copy the _actions.so file we compiled earlier over the one that’s packaged with GlassFish 3.0.1. Just copy it over the other file, like so: cp _actions.so /opt/glassfish/pkg/custom-lib/pkg/actions/_actions.so
Now we’re really done! You should be able to run /opt/glassfish/bin/pkg image-update now and update your currently installed GlassFish 3.0.1 with the latest modules and stuff. Awesome!
I’m in the process of creating a Debian package for GlassFish 3.0.1 which incorporated this fix. So if you’re not in a hurry or you’re reading this way after it was posted, you might want to check out debian.kumina.nl to see if the package is available.
Hope this helps someone!
Tags: backup, configuration, debian, etch, lenny, mylvmbackup, mysql, package
Comments Off
MyLVMbackup: Making it cooperate with Debian
For making a consistent backup of MySQL data, mylvmbackup seems to be a great tool. It’s written in Perl and does what you would expect: Lock the tables, flush them all to disk, make a LVM snapshot and release the tables, recover the InnoDB log,make the backup from the snapshot, remove the snapshot. Minimal downtime and still a consistent backup. Although it’s fairly new, it looks good.
There’s even a Debian package for it, but that doesn’t really contain everything I need. First of all, I need a package that runs on Etch. There’s a dependency in the Debian package (which is for Lenny) on an asciidoc of a version that’s available only in Lenny. I don’t really see why that dependency is there at all, but when removing the version, the package still works as expected.
Also, I made two patches of which one is not needed at all, but I left it in there anyway, because it doesn’t break anything. It simply adds the –innodb-file-per-table option to the invocation of the mysqld_safe daemon, but that’s not needed since it only affects MySQL when you change data. But since mysqld_safe is used to recover from logs, you won’t be changing data. I left it in there anyway. You can find the dpatch file here.
The second one might be more interesting. The mysql-server-5.0 package from Debian creates a file /etc/mysql/debian.cnf with login details for the debian-sys-maint user. This user also has enough permissions to do all the things necessary for manipulating the database when we want to make a backup with mylvmbackup. Normally, mylvmbackup needs you to give it user details for a privileged user in /etc/mylvmbackup.conf, which is a bit double, since the debian.cnf exists. So my patch adds a –defaults_file option which allows you to mention the debian.cnf. No need for more passwords in plaintext on the filesystem! You can either give this option no the commandline or in the mylvmbackup.conf in the [mysql] section. It only supports Debian style debian.cnf, though, since it searched for a [client] section in the file you mention. You can find the patch here.
A working package can be found in Kumina’s Debian repository for both etch and lenny. Please let me know if this works for you! It’s my first Perl, though, so be warned. If stuff breaks… Well… Sorry… So be sure to test it.
Debian Sarge: LVM2 snapshot (removal)
I had some troubles with removing an LVM snapshot from within a Debian Sarge domU. It would lock up the domU completely, without removing the snapshot. A bit of a problem, especially since this was a production machine. So I destroyed the domU and tried again, with the same results, unfortunately.
Bart found a description of the problem and the solution at Jürgen Kreileder’s blog. Jürgen apparantly still has the repos online, which helped us resolve the problem on our machine. Much appreciated!
Tags: apache22, coldfusion, coldfusionmx7, debian, etch, jrun, mod_jrun22, sarge
Comments Off
Upgrade Debian Sarge to Etch: ColdFusionMX7 is a pain
So we’re finally getting around to upgrading an old Debian Sarge server which runs ColdFusionMX7 to Etch. The Debian part is easy, I simply edit the sources.list and do an apt-get dist-upgrade and some questions later, presto, the machine is a working Debian Etch machine.
Getting ColdFusionMX7 to run with the Etch supplied Apache 2.2 is a different matter. Took me about several weeks to find out (not full time, but still over 20 hours, I think). First of all, you need the modified wsconfig.jar that Adobe distributes via a knowledge base article about their hotfix. Don’t bother with their instructions, though, the mod_jrun22.so will not compile that way. Not because of missing apps or something, but because for some strange reason, the script thinks JRun is already up and running. I get an output like so:
Macromedia JRun 4.0 (Build 107948) os.name: Linux os.version: 2.6.18-6-xen-amd64 os.arch: i386 platform: intel-linux Found port 2920 on host localhost findServers(): found server coldfusion at 127.0.0.1:2920 Found JRun server coldfusion at 127.0.0.1:2920 this host is stage02:62.133.201.114 web server: Apache web server directory: /etc/apache2 verbose connector logging: false apialloc: false force resource extract from jar: true CFMX: true mappings: .jsp,.jws,.cfm,.cfml,.cfc,.cfr,.cfswf filter mapping prefix: false Apache binary: /usr/sbin/apache2 Apache control script: /usr/sbin/apache2ctl Apache apxs: true This web server is already configured for JRun.
Not very helpful. I tried removing all files related to Apache and JRun that I could think of, including the part in /etc/apache2/httpd.conf that loads the module, but it still told me that the web server was already configured. Time for some heavier work.
First, start with unzipping the wsconfig.zip. It’ll yield a wsconfig.jar. Unzip that one too in a temporary directory, let’s say in /tmp/wsconfig. Now go to /tmp/wsconfig/controller/src. There’s a text file in there that contains the steps to install it. However, I need to install a few packages:
apt-get install apache2-prefork-dev gcc
You might need apache2-worker-dev, if you use that Apache flavour. You can check that with dpkg -l | grep apache2. You’ll find either apache2-mpm-prefork or apache2-mpm-worker. That’s your clue!
Now, execute the following commands:
sudo apxs2 -c -n jrun22 mod_jrun22.c \ jrun_maptable_impl.c jrun_property.c jrun_session.c platform.c \ jrun_mutex.c jrun_proxy.c jrun_utils.c sudo apxs2 -i -n jrun22 mod_jrun22.la sudo strip /usr/lib/apache2/modules/mod_jrun22.so
This will install the Apache 2.2 JRun module at /usr/lib/apache2/modules/mod_jrun22.so. Now change your Apache config accordingly, and you’re ready to go.
One thing I found, but that was just me trying to be hasty, was that you need to use:
LoadModule jrun_module /usr/lib/apache2/modules/mod_jrun22.so
Hope this helps someone.
Shrinking a MySQL ibdata1 file
MySQL’s InnoDB engine stores all tables by default in one large file which is located (in Debian) under /var/lib/mysql/ibdata1. This file allocates some diskspace (in Debian it defaults to 10M chunks), but never unallocates the disk space. So if you make a few large tables, your partition might get full. Removing those large tables doesn’t help a thing, since the ibdata1 file is never shrunken by MySQL’s InnoDB engine. I needed to shrink it, preferably with as little as possible downtime.
It is nigh impossible. I’ve tried several things. First, I added the option innodb_file_per_table. The idea was to optimize each table, so it would be rewritten into a separate file. Then delete the ibdata1 file and it’s logs and restart MySQL. This worked in a simple test setup, but failed miserably in the production environment. I have no idea what went wrong, but even when I restored the old ibdata1 and it’s logs, it wouldn’t work anymore. Chaos and mayhem all around. Joy.
So I resorted to the only other solution I could find: Make a dump of the complete database and remove all files in /var/lib/mysql, then restart MySQL and restore the dump. Not a very practical solution, but the only one that seems to work reliably.
NFS file locking
Sometimes you need file locking over NFS. In my case, I needed it for running MySQL tables via NFS. Even though you can tell MySQL to not do external locking, this setting is ignored when you use InnoDB. That’s not a very well documented feature. So you need to make sure locking is available.
I did most of my debugging with a small perl script that I found on the HP forums. Simply edit the location and you’re ready to go. My problem was that I had no trouble at all getting it to work in another setup, but I couldn’t find at first what was different in the production setup. In the end it was very obvious (as it usually is): The network was way more complicated.
The setup I’m talking about is in a datacentre that houses several of our customers. They’re all daughter-companies from one large one, so they’re okay with sharing some infrastructure. But each company has it’s own domain name and prefers of course to use that one. Now, we name our virtual machines (we do everything virtualised) after their specific function, so a database master would be called something like dbm1.customera.com. Another customer with another database master would call the machine dbm1.customerb.com. But when we’re working in that network, it’s annoying to use those full names, so we have an internal DNS that resolves things like dbm1.custa and dbm1.custb. Just for convenience.
But that convenience is what made NFS locking break. Since the machine would report it’s hostname as being dbm1.customera.com, while it’s DNS name would be dbm1.custa. Apparantly, NFS can’t handle that very well.
Luckily, there’s an easy solution: Add a specific hostname to the NFS stat daemon. In Debian, you do that by editing /etc/default/nfs-common (after you installed the nfs-common package) and adding the following line: STATDOPTS="-n dbm1"
That worked for me. Now, I have working InnoDB tables on an NFS share.
Tags: a_bit_strange, credentials, debian, fix, lenny, mysql, permissions
1 comment
MySQL on Lenny and permissions
Ok, had something strange that I think is worth a share. I just upgraded a virtual machine to Lenny and installed MySQL and phpMyAdmin and apache and the like. Created a user for a Django project I’m working on and it couldn’t connect. I tried resetting the password several times, didn’t help. Then, out of frustration, I tried connecting without a password and… it worked!
Apparantly, by default, the MySQL in Lenny created access rules that allow all local users to access the database without a password. These have priority over the specific user controls. A bit unexpected and I suspect others to run into this. Could make you search for a problem if you’re installing a shared webhosting machine with a local MySQL database.
Solution:
delete from mysql.user where User='';
Debian’s Trac: “SubversionException: (‘No such revision 1′, 160006)”
A bit annoying, but the workaround (fix?) is easy. Edit /var/lib/python-support/python2.4/trac/versioncontrol/cache.py, line 114. You’ll see pass there, change it into return. Fixed.
Also described in Debian bugreport #438864. Took me a while to find it, so referencing it here.
MySQL on S3… not yet
Ok, I’ve been playing with EC2 and S3. Using quite a long and complicated script to create an AMI (which is Amazon’s term for a disk image). The script is coming along nicely, using debootstrap to create a Debian etch image.
Yesterday and today (well, only in the evening today), I’ve been trying to get MySQL working on an s3fs drive, but I’m afraid that’s not going to work. Well, part of it might be because I’m trying to set it up from home and not from an EC2 instance, but still. MySQL just refuses to start with a weird message:
Starting MySQL database server: mysqld . . . . . . . . . . . . . . failed!
invoke-rc.d: initscript mysql, action "start" failed.
After which it hangs, altough it seems like MySQL did start. Maybe it’s something with the initscript. I realised in the end that I don’t really need MySQL running in such a setup. Making dumps every so often should be enough for what we have in mind. So I’m going to script an automatic recovery of the database, based on a dump every 5 minutes. That should work without losing any speed.
check_ldap with starttls for Nagios
Since we use Nagios to monitor services and hosts with customers and OpenLDAP for accounts, we’d like to check if LDAP is still active and working. But we had a problem here, because the check_ldap plugin that’s distributed in Debian‘s nagios-plugins package doesn’t speak STARTTLS. So I modified the source-code of check_ldap and created check_ldap_starttls. Some might (and have) argue that using ldaps:// would be easier to implement, but that’s a non-argument, for as far as we’re concerned, since ldaps:// is deprecated.
As usual, you can find my files right here. I claim no copyright to the code whatsoever, since I only added about 8 lines and most of those were copied from some other programme. Use as you see fit.
- check_ldap_starttls (the programme)
- check_ldap_starttls.c (the code)
Enjoy!
Update Been fixed in the latest packages, I believe. Removed the links here.
Can’t we all just get along?
There is an interesting read on Ian Murdock’s Weblog: Can’t we all just get along? It talks about the differences between Debian and Ubuntu, the latter being a derivative of the first. Although I can understand Ian’s problem, I do not share his views.
He’s correct in some respects, for example that Ubuntu’s packages often don’t work on Debian Sarge. But the question is, do we want them to? Debian Sarge is a very stable and secure platform, one we (as a company) prefer to use in a corporate environment. In fact, 45% of our deployments are Debian Woody and another 45% are Debian Sarge. (The rest are mostly different hardware routers and Cisco stuff.) I wouldn’t want those Ubuntu packages working on Sarge! As much as I love Debian, I never thought of it as a desktop distribution. It’s ideal for servers, because it’s stable and has slow update cycles. You know, when people say they choose for Red Hat because they can upgrade often, that’s just not logical. Servers we have are deployed and work without a glitch. They don’t need to be upgraded regularly, just patched for security holes! Why in the world would we want to upgrade a firewall/router? Patch the holes and give me those updates, that’s all I want. And that’s what Debian gives me. Sarge is due to be released very soon and the next version of Debian, Sid, will probably not be released for at least two years. And that’s the way I want it to be. Server protocols don’t change all that often.
On the other hand, a desktop is a machine that needs to be upgraded often. And when I take the time to upgrade, I want it to show. Not just some new buttons, but a nicer interface, new programmes, more features, better spam-detection, et cetera, et cetera. That’s what Ubuntu gives us. Fast release cycles that matter. Each six months we get improvements. Do we want those improvements on a server? Usually not. But when we do want them, it’s easier to backport an Ubuntu package than to build one from scratch.
Just my two cents.
Drupal on Sarge
It might be me (it most probably is), but I can’t get Drupal going on Debian Sarge. We’ve got Apache 2 running on Amstel, our machine in Amsterdam, and I simply installed the package Drupal with apt-get, the standard package tool from Debian. It tweaked all sorts of things, but I cannot get the site to work. I keep getting messages like these:
[Thu Mar 10 20:42:18 2005] [debug] /home/adconrad/apache2/apache2-2.0.53/build-tree/apache2/server/core.c(2749): [client xx.xx.xx.xx] redirected from r->uri = /cgi-bin/php/index.php
[Thu Mar 10 20:42:18 2005] [debug] /home/adconrad/apache2/apache2-2.0.53/build-tree/apache2/server/core.c(2749): [client xx.xx.xx.xx] redirected from r->uri = /index.php
[Thu Mar 10 20:42:18 2005] [debug] /home/adconrad/apache2/apache2-2.0.53/build-tree/apache2/server/core.c(2749): [client xx.xx.xx.xx] redirected from r->uri = /drupal/index.php
Maybe it’s because we’ve got PHP4 running as CGI, but I really think it should work. The error message is too cryptic for me to actually deduce anything out off, except that it seems to redirect itself in circles. The three lines above are repeated several times and result in a 500. I’m at a loss. I seem unable to solve this in a correct way. Any help is appreciated.
