Python: Reverse reader
Sometimes you want to read files from the end to the front. You can find a lot of recipes online, but I’m using the one in the comments from here (scroll down). Bonus feature, I can use that one as an iterator, so csv.reader and other stuff like it can use it. Blogging it here, so I know where to find it again if I need it 
MySQL: Backslash is a forbidden character in passwords
Just so you know. My co-worker Bart spent a while figuring that out.
4 Years of Blogging
So I timed this post at exactly four years after I wrote my first post on this blog. A lot has changed since then. Maybe I should re-read some of those older messages.
Hehehe, I was still using Mandriva (then known as Mandrake) back then. And even then I was working on joining RSS feeds and bayesian filters! How cool is that? I keep coming back to that project and starting it anew every year, I guess. Hope I can get it to work one of these days. I still think the idea is great, if only I could get myself to spend a few days on the project, I know I could make it work. Ah well…
Security is hard
Just got an email form one of our customers asking if it was possible to use an encrypted password in PHP for connecting to MySQL. So that instead of doing something like:
$connection = mysql_connect(host,user,pass);
He wanted to do something like:
$connection = mysql_connect(host,user,encrypted_pass);
Which is of course not very useful (since you could use the encrypted string just like any other normal password, so there’s no added security). We come across these kinds of notions quite often, people want to use encryption for security, but the way they use it makes it kind of useless.
A few years ago we had a customer who wanted a fully secured machine, from boot onwards. This so he could sell appliances without giving his customers easy access to the operation system and application. He wanted an encrypted hard disk. But if you use standard x86 based hardware, you have no way to store an initial secret. Even if you’d embed the password somewhere in the bootloader, it’s still somewhere on that machine.
Security is hard to do well. I wish people would start by simply applying best practises, like setting safe file permissions. Encryption is often not very useful if you want parts of an application to actually access the data without the user entering the password of that encrypted data.
Need
I got a need to write, but nothing to write about. Now how sad is that?
Advantage of being young
Last Saturday my 11 year old step-sister used my iMac. She had seen the interface before, but never used it. I opened Safari for her, since I reckoned a browser is an easy piece of software that’s the same on every OS, mostly. I left her alone after that (had to do the dishes).
After ten minutes I heard my iTunes playing and when I went to look at her screen I noticed that she had been browsing through my music and was writing a story in TextEditor. She showed me how she found out that she could minimise windows, enlarge them again, switch to another desktop (I use VirtueDesktop) and essentially where to find the software she needed in my dock.
Ten friggin’ minutes. I mean… like… wow.
Birthdayparty
Celebrated my birthday yesterday and I had a wonderful day. My step sister (of eleven) helped me the whole afternoon with the shopping and getting ready and most of my friends came for the evening and actualy stayed for a long time. Which seems to indicate that they had fun.
In all honesty, the party was probably more a “thank you” towards my friends, because they all helped me quite a lot these last few months. During the evening I mentally gave them all a separate role. My coworkers, who listened to me whine often enough, but never complained about it. My best friend, who always agreed with me about the issue, no matter what it was about. That sounds a bit shallow, but at times that’s exactly what I need, simply someone who just agrees. My best ladyfriend, who had her own troubles, but was still there to talk to me and dispell some negative notions I had about myself. My dad and his wife, who always had time to sit around the kitchen table and drink coffee with me. My dad’s best friend, who made some important decisions for later when they were needed and when no one else thought about it. And of course Aagje, who was always ready to cheer me up and who I almost repayed with considering to give her away to someone else.
I had a great evening and I think everyone else enjoyed themselves as well. That’s important, good times with good friends.
Hot hot Santa!
Depending on where you work, might be NSFW (mostly because it will probably break your monitor). Still, very funny.
*snip* Removed it because it started playing every single time! Look at it here: Hot hot Santa
Shrinking a MySQL ibdata1 file
MySQL’s InnoDB engine stores all tables by default in one large file which is located (in Debian) under /var/lib/mysql/ibdata1. This file allocates some diskspace (in Debian it defaults to 10M chunks), but never unallocates the disk space. So if you make a few large tables, your partition might get full. Removing those large tables doesn’t help a thing, since the ibdata1 file is never shrunken by MySQL’s InnoDB engine. I needed to shrink it, preferably with as little as possible downtime.
It is nigh impossible. I’ve tried several things. First, I added the option innodb_file_per_table. The idea was to optimize each table, so it would be rewritten into a separate file. Then delete the ibdata1 file and it’s logs and restart MySQL. This worked in a simple test setup, but failed miserably in the production environment. I have no idea what went wrong, but even when I restored the old ibdata1 and it’s logs, it wouldn’t work anymore. Chaos and mayhem all around. Joy.
So I resorted to the only other solution I could find: Make a dump of the complete database and remove all files in /var/lib/mysql, then restart MySQL and restore the dump. Not a very practical solution, but the only one that seems to work reliably.
NFS file locking
Sometimes you need file locking over NFS. In my case, I needed it for running MySQL tables via NFS. Even though you can tell MySQL to not do external locking, this setting is ignored when you use InnoDB. That’s not a very well documented feature. So you need to make sure locking is available.
I did most of my debugging with a small perl script that I found on the HP forums. Simply edit the location and you’re ready to go. My problem was that I had no trouble at all getting it to work in another setup, but I couldn’t find at first what was different in the production setup. In the end it was very obvious (as it usually is): The network was way more complicated.
The setup I’m talking about is in a datacentre that houses several of our customers. They’re all daughter-companies from one large one, so they’re okay with sharing some infrastructure. But each company has it’s own domain name and prefers of course to use that one. Now, we name our virtual machines (we do everything virtualised) after their specific function, so a database master would be called something like dbm1.customera.com. Another customer with another database master would call the machine dbm1.customerb.com. But when we’re working in that network, it’s annoying to use those full names, so we have an internal DNS that resolves things like dbm1.custa and dbm1.custb. Just for convenience.
But that convenience is what made NFS locking break. Since the machine would report it’s hostname as being dbm1.customera.com, while it’s DNS name would be dbm1.custa. Apparantly, NFS can’t handle that very well.
Luckily, there’s an easy solution: Add a specific hostname to the NFS stat daemon. In Debian, you do that by editing /etc/default/nfs-common (after you installed the nfs-common package) and adding the following line: STATDOPTS="-n dbm1"
That worked for me. Now, I have working InnoDB tables on an NFS share.
MySQL says: “ERROR 1030 (HY000): Got error -1 from storage engine”
In my case, this meant it was not allowed to write data (an INSERT) to a InnoDB table because innodb_force_recovery was set too high. Took me a while to figure out, so I hope this helps someone.
Solution is to simply disable the innodb_force_recovery. You should use that setting only for making an emergency dump anyway.
More about my wanderings through MySQL+InnoDB+NFS land soon.
A Different Project
The seatbelt of my car is stuck. Very annoying, I always drive with my seatbelt on (you risk a fine if you don’t here in NL) and it feels like driving naked when I can’t use it. So today I brought my car to the mechanic’s.
The car is there now, I just walked to my dad’s house (it’s closer to the shop than my own) and I came across this sign:
Seen in Puth
It’s a photo made with my cell phone, so not that clear. The whole sign is of a kind that we see here often, usually at construction sites. The contents are:
- Project: Rik
- Date of delivery: 23 nov 2008
- Building contractor: Daddy
- General foreman: Mommy
- Supervisor: Luuk
- Interested: www.j3kidsart.nl
Do not give to neighbour
So my neighbour was at my front door yesterday with a package from Apple. She was kind enough to accept it and sign for it. I thanked her and closed the door. I went inside and started opening the package. Then I noticed the following text on the UPS label:
“Do not deliver to neighbour.”
… Okay?
