GlassFish 3.0.1′s pkg tool using Debian’s Python packages

While setting up GlassFish 3.0.1 for a customer on Debian Lenny using 64 bit machines, I ran into the problem that the update tool shipped with GlassFish (OpenSolaris‘s pkg tool) uses a Python interpreter which is part of the package. That Python interpreter however is 32 bit, which requires the ia32 libraries to be installed. Worse, it requires lididn in 32 bit, which is not part of the default Debian packages, so I had to get that one from the (very nice) Debian Multimedia repository. Although that’s a quick fix to get stuff working, we rather not use these repositories on production machines, due to security concerns and the like. Also, upgrades are easier if you only use the standard Debian repositories.

So I decided to see if I could get it working with the Debian supplied Python interpreter. One problem is that there’s a shared object file written in C which is part of the pkg application. That file is a 32 bit ELF too. So we’re going to download that source and recompile it for 64 bit. If you’re on a 32 bit system, you can skip that step (although it doesn’t hurt to do it anyway). First, we need to install the following packages:

apt-get install python2.5 python2.5-dev gcc python-cherrypy python-mako python-openssl python-ply python-pycurl python-simplejson

Now download the source for the _actions.c file from here (link to webpage, press download in the top).

Compile it with the following command:
gcc -I/usr/include/python2.5 -shared -fpic -O2 _actions.c -o _actions.so

Keep the resulting .so file, we’re going to replace it once we’ve downloaded GlassFish. Which is the next step, actually. So download GlassFish and set it up somewhere. I downloaded the tarball and unzipped it into /opt/glassfish.

The resulting directory contains several directories, including a pkg and a glassfish directory. The /opt/glassfish/glassfish directory is the actual GlassFish application. The pkg directory contains the pkg tool which is used to upgrade the GlassFish addons and systems and stuff (don’t ask me about the details, I’m not a Java developer, only a sysadmin). The first time you start it, it installs some stuff. So just run the following:
/opt/glassfish/bin/pkg

Next, mv the file /opt/glassfish/pkg/bin/pkg to /opt/glassfish/pkg/bin/pkg.orig. We do this since this script does all kinds of magic which we do not need. Also, it runs the pkg python code with the python2.4 interpreter that’s part of the package. We don’t want that. Let’s fix it.

Make a simple script to replace the one we moved away. I use the following, which works for me:

#!/bin/sh
python /opt/glassfish/pkg/bin/client.py

If you start it now, you’ll get an error:

$ python /opt/glassfish/pkg/bin/client.py
Traceback (most recent call last):
  File "/opt/glassfish/pkg/bin/client.py", line 60, in 
    import pkg
ImportError: No module named pkg

Ok, let’s fix that! Start with creating a directory called /opt/glassfish/pkg/custom-lib. You can change the name into anything you want, of course, as long as it’s clear that this is where you’re going to put the pkg python module. Actually, let’s do that immediatly: cp -r /opt/glassfish/pkg/vendor-packages/pkg /opt/glassfish/pkg/custom-lib

Change our script which we setup to run pkg into the following:

#!/bin/sh
PYTHONPATH="/opt/glassfish/pkg/custom-lib" python /opt/glassfish/pkg/bin/client.py

You need to have only the pkg module in there, because the PYTHONPATH variable takes precedence over the other modules installed via the Debian packages. Run the script and you’ll get a new error:

Traceback (most recent call last):
  File "/opt/glassfish/pkg/bin/client.py", line 61, in 
    import pkg.actions as actions
  File "/opt/glassfish/pkg/custom-lib/pkg/actions/__init__.py", line 144, in 
    from _actions import _fromstr
ImportError: /opt/glassfish/pkg/custom-lib/pkg/actions/_actions.so: wrong ELF class: ELFCLASS32

If you don’t get this error, you’re on a 32 bit system and you’re done! Congratulations! Otherwise, we’re going to copy the _actions.so file we compiled earlier over the one that’s packaged with GlassFish 3.0.1. Just copy it over the other file, like so: cp _actions.so /opt/glassfish/pkg/custom-lib/pkg/actions/_actions.so

Now we’re really done! You should be able to run /opt/glassfish/bin/pkg image-update now and update your currently installed GlassFish 3.0.1 with the latest modules and stuff. Awesome!

I’m in the process of creating a Debian package for GlassFish 3.0.1 which incorporated this fix. So if you’re not in a hurry or you’re reading this way after it was posted, you might want to check out debian.kumina.nl to see if the package is available.

Hope this helps someone!

Sorting Apache log files

Last week I had to sort several months worth of Apache log files to feed them to Webalizer. These came from several servers, so it was fairly difficult to get the data together and in the correct format. I ended up using a small python sorting script, especially written for this. Maybe it’ll help someone else, so I’m sharing it here. Just pipe the logs through the script and direct the output to a file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

import sys
 
data = sys.stdin.readlines()
 
def compare_apache_dates (date1, date2):
	str1 = date1.split()[3]
	str2 = date2.split()[3]
	months = ["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"]
 
	if str1[8:12] > str2[8:12]:
		return 1
	elif str1[8:12] < str2[8:12]:
		return -1
	elif months.index(str1[4:7]) > months.index(str2[4:7]):
		return 1
	elif months.index(str1[4:7]) < months.index(str2[4:7]):
		return -1
	elif str1[1:3] > str2[1:3]:
		return 1
	elif str1[1:3] < str2[1:3]:
		return -1
	elif str1[13:15] > str2[13:15]:
		return 1
	elif str1[13:15] < str2[13:15]:
		return -1
	elif str1[16:18] > str2[16:18]:
		return 1
	elif str1[16:18] < str2[16:18]:
		return -1
	elif str1[19:21] > str2[19:21]:
		return 1
	elif str1[19:21] < str2[19:21]:
		return -1
	else:
		return 0
 
data.sort(compare_apache_dates)
for line in data:
	if line[1:4] == "var":
		# Small hack to fix output lines from egrep
		sys.stdout.write( line.split(":",1)[1] )
	else:
		sys.stdout.write( line )

Tags: apache, log, python, sort, webalizer

Kumishifts: The Release

[Repost from Kumiblog]

One of my personal missions within Kumina is to decrease the amount of noise. We work fairly event-driven, responding to everything that gets reported. Although I believe our customers like this very much, it can be a bit of a bother in the case where someone is not on helpdesk duty. One of my pet peeves was the fact that Nagios sent SMS alerts to everyone, instead of the person who was on duty at the time.

Since we have multiple Nagios instances, it would be a bit of a hassle to change the config everywhere all the time, so we stuck with the current way. At least you can be sure that someone received the message. But since we get some false positives too, at time, it does add to the bill.

So I decided to create Kumishifts. This little Python script takes a Google Calendar (or any ical you point it at) and distills who’s on duty. It generated a Nagios contacts file based on that information (and then some). We can now actually work with escalations to make sure that if the first person responsible doesn’t respond fast enough (bad person!), a second one will get the message after a little while, too.

We’re not yet deploying the script everywhere, but will soon, after it’s properly tested. I’d appreciated any feedback on the app!

Tags: application, calendar, google calender, helpdesk, ical, nagios, python, shifts

[FunkLoad] Random tests for benchmarking

I’ve recently started using FunkLoad to do benchmarking of websites. It works quite okay and I love the fact that it’s Python, since that’s my language of choice.

One thing that kept bugging me was that since FunkLoad is more a unittester, it can by default only run a single run multiple times against a site. That won’t do. You want several different kinds of users visiting your website when you’re benchmarking. But since everything is Python, I was able to solve that like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# -*- coding: iso-8859-15 -*-
"""loadtest FunkLoad test
 
$Id: $
"""
import unittest, random, sys
from funkload.FunkLoadTestCase import FunkLoadTestCase
from webunit.utility import Upload
from funkload.utils import Data
#from funkload.utils import xmlrpc_get_credential
 
class LoadTest(FunkLoadTestCase):
    """ Loadtest for website.
 
    This test use a configuration file LoadTest.conf.
    """
 
    def setUp(self):
        """Setting up test."""
        self.logd("setUp")
        self.server_url = self.conf_get('main', 'url')
 
    def doTest(self):
        # Picks a test at random
        tests = ["8081","8082","8083","8084","8085","8086"]
        rnd = random.choice(tests)
        return getattr(self, "test_"+str(rnd))()
 
    def test_8081(self):
        # The description should be set in the configuration file
        server_url = self.server_url
        # begin of test ---------------------------------------------
        # end of test---------------------------------------------
 
    def test_8082(self):
        # The description should be set in the configuration file
        server_url = self.server_url
        # begin of test ---------------------------------------------
        # end of test---------------------------------------------

(I left out the test themselves, but I’m sure you can find where to place them.)

The recorded tests are in the same class, and they’re called test_8081, test_8082, etc. up to test_8086. These corresponded to the proxy ports on which fl-record was listening when the session was recorded. Line 27 makes sure that the test that was chosen gets run.

You can of course expand this as much as you like, for example adding weights for certain tests. Kinda neat, python.

Tags: benchmarking, funkload, python, random, website, work

Python: My .vimrc

This is especially for working with Python. I welcome additions or better practises!

set term=builtin_ansi
syntax on
autocmd BufRead *.py set smartindent cinwords=if,elif,else,for,while,try,except,finally,def,class
autocmd BufRead *.py inoremap # X^H#
autocmd BufRead *.py set tabstop=4
autocmd BufRead *.py set shiftwidth=4
autocmd BufRead *.py set smarttab
autocmd BufRead *.py set expandtab
autocmd BufRead *.py set softtabstop=4
autocmd BufRead *.py set autoindent
autocmd BufRead *.py highlight BadWhitespace ctermbg=red guibg=red
autocmd BufRead *.py match BadWhitespace /^\t\+/
autocmd BufRead *.py match BadWhitespace /\s\+$/

(The ^H is typed by Ctrl+V Ctrl+H.)

Tags: python, vim

Python: IMAP IDLE with imaplib2

Had a little fun today trying to get IMAP IDLE from Python working. It’s not in the default imaplib, but Piers Lauder’s imaplib2 has support for it. (I read on another blog that Piers Lauder is also the writer of imaplib, so that’s kinda nice.)

I don’t really program a lot anymore, so I had some trouble trying to understand how it’s supposed to work. Luckily, the blog post that I linked above, also had a link to the git sources of offlineimap, which uses it. Their imapserver.py has a nice implementation that I kind of copied.

As a proof of concept, and hopefully so others can use this too, I created the script below to see if I could get it to work. It doesn’t do a whole lot, just print a line when the mailbox gets a new mail. But you can build on it from there. I need these kind of examples to get stuff like this, so I hope it helps someone else! Leave me a comment if it does.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

import imaplib2, time
from threading import *
 
# This is the threading object that does all the waiting on 
# the event
class Idler(object):
    def __init__(self, conn):
        self.thread = Thread(target=self.idle)
        self.M = conn
        self.event = Event()
 
    def start(self):
        self.thread.start()
 
    def stop(self):
        # This is a neat trick to make thread end. Took me a 
        # while to figure that one out!
        self.event.set()
 
    def join(self):
        self.thread.join()
 
    def idle(self):
        # Starting an unending loop here
        while True:
            # This is part of the trick to make the loop stop 
            # when the stop() command is given
            if self.event.isSet():
                return
            self.needsync = False
            # A callback method that gets called when a new 
            # email arrives. Very basic, but that's good.
            def callback(args):
                if not self.event.isSet():
                    self.needsync = True
                    self.event.set()
            # Do the actual idle call. This returns immediately, 
            # since it's asynchronous.
            self.M.idle(callback=callback)
            # This waits until the event is set. The event is 
            # set by the callback, when the server 'answers' 
            # the idle call and the callback function gets 
            # called.
            self.event.wait()
            # Because the function sets the needsync variable,
            # this helps escape the loop without doing 
            # anything if the stop() is called. Kinda neat 
            # solution.
            if self.needsync:
                self.event.clear()
                self.dosync()
 
    # The method that gets called when a new email arrives. 
    # Replace it with something better.
    def dosync(self):
        print "Got an event!"
 
# Had to do this stuff in a try-finally, since some testing 
# went a little wrong.....
try:
    # Set the following two lines to your creds and server
    M = imaplib2.IMAP4_SSL("mail.example.com")
    M.login("mylogin","mypassword")
    # We need to get out of the AUTH state, so we just select 
    # the INBOX.
    M.select("INBOX")
    # Start the Idler thread
    idler = Idler(M)
    idler.start()
    # Because this is just an example, exit after 1 minute.
    time.sleep(1*60)
finally:
    # Clean up.
    idler.stop()
    idler.join()
    M.close()
    # This is important!
    M.logout()

Tags: example, idle, imap, imaplib2, python

Python: Reverse reader

Sometimes you want to read files from the end to the front. You can find a lot of recipes online, but I’m using the one in the comments from here (scroll down). Bonus feature, I can use that one as an iterator, so csv.reader and other stuff like it can use it. Blogging it here, so I know where to find it again if I need it

Tags: file, python, reader, reverse

4 Years of Blogging

So I timed this post at exactly four years after I wrote my first post on this blog. A lot has changed since then. Maybe I should re-read some of those older messages.

Hehehe, I was still using Mandriva (then known as Mandrake) back then. And even then I was working on joining RSS feeds and bayesian filters! How cool is that? I keep coming back to that project and starting it anew every year, I guess. Hope I can get it to work one of these days. I still think the idea is great, if only I could get myself to spend a few days on the project, I know I could make it work. Ah well…

Tags: anniversary, bayes, history, linux, python

duplicity in MacPorts, backup to S3

Duplicity from MacPorts is currently broken. The problem is in the portfile, which wants to use Python 2.5. However, there is no OpenSSL package for Python 2.5 in MacPorts, which gives you this error if you try to backup to S3 through SSL:

File "/opt/local/lib/python2.5/httplib.py", line 1135, in connect
ssl = socket.ssl(sock, self.key_file, self.cert_file)
AttributeError: 'module' object has no attribute 'ssl'

The solution is to edit the Portfile (which lives in /opt/local/var/macports/sources/rsync.macports.org/release/ports/sysutils/duplicity/Portfile on my machine) and change “python25″ in “python24″ and halfway through the file change the py25-* packages in py-* packages. Then install the port. That fixes it.

Don’t forget to *not* upgrade the port without changing the Portfile…

Tags: duplicity, macports, python, s3

Isoformat in Python 2.4

Sometimes I don’t like python. Just sometimes, I love it most of the times. But just look what hoops I need to jump through to get a current timestamp in ISO format:

from datetime import datetime, tzinfo, timedelta
import time

class TZ(tzinfo):
        def utcoffset(self,dt): return timedelta(seconds=time.timezone)
        def dst(self,dt): return timedelta(0)

a = datetime.now(TZ())
a= a.replace(microsecond = 0)
print a.isoformat()

That prints:

asterisk:~ tim$ python test.py
2008-03-24T13:38:06-01:00

Fugly…

Tags: code, date, isoformat, python, time

PyQt4 on Windows

Yes, you read that correctly, on Windows. Wanted to make a .exe out of an app I’m working on, so I had to install PyQt4 on Windows. I never use Windows, so I was at a loss. But for those others who are looking for it:

You can change the env vars like %PATH% in Configuration screen > System > Advanced.

Finding that took me about 45 minutes. God, I hate Windows.

Tags: pyqt4, python, qt4, windows

PyQt4 on MacOSX with Macports

Yesterday evening, I created a working py-pyqt4 Portfile for Macports. At least, it worked for me. You can find it in ticket #14744 on the Macports Trac. Hope this helps someone.

Tags: macports, pyqt4, python, qt4

Fallen in love again…

… with programming (if you expected “my girlfriend” here, I’ve never stopped loving her). I’ve been going through the Django tutorial and I’m really liking doing webdevelopment again. Yes, I know, I was working with Ruby on Rails earlier, but Django is much more to my liking. I have to quit now with the tutorial because I’m sleepy, but I will continue with it tomorrow. Take a look at it… It’s really, really cool.

On a sidenote, maybe if Ruby on Rails had such a cool tutorial, I would fall for that language, but I’m struggling much more with Ruby on Rails. And I already know (some) Python, which makes Django much easier for me. What are you waiting for?? Go take a look!

Tags: django, programming, python, rant