Personal Email Server with Ubuntu and Losing Network on Virtual Appliances

Jan
31
2007

Yesterday, before leaving for work, I seriously FUBAR'ed my RSS reading email setup to the point where it wouldn't get past the "A's" in the feed list. Before that, a couple of weeks ago, I ended up with a huge regular email mess because:

  1. Some spammer used my email addresses as the "FROM" on a bunch of spam.
  2. A major bank in South Africa was the recipient of a LOT of those spam messages.
  3. Said bank decided to bounce each and every one of them to the "FROM" address: me.
  4. I only managed to get the sandbags in place after 31,000 of those bounced messages came through.

And, before that, I had my personal mail server needing to be reinstalled and messed with because the junk account to which all spam was directed reached a size that Mercury Mail couldn't handle. Finally, the Windows machine that is serving as my mail server (and a bunch of other uses) is going to be replaced as it moves into a role as a Call of Duty 2 server. I was really not looking forward to carrying the existing mail setup (the only thing on this machine that's problematic to move) into the new setup.

With all of that weighing heavily on the email portion of my geek setup, last night I decided to tackle something that's been on the TODO list for something like 3 years: a personal Linux email server to handle all of this crap in a more industrial approach. Further, I want all of this in a VMWare virtual appliance so I can easily back it up, move it between different hardware, take it with me on a portable drive, etc.

I've taken half-hearted stabs at this setup before, and stopped before any of it really worked, so I was deeply amazed when I was actually able to get to a basic functioning setup in a couple of hours last night. Here's the basic set of requirements I've got in order to call this thing "done".

  • Server does not have a domain of its own.
  • Fetches mail from an arbitrary number of POP3 mailboxes to appropriate user accounts.
  • IMAP access to MailDir setup of those mailboxes.
  • Outbound email sent via smarthost through outbound.mailhop.org.
  • My entire chain of tools for RSS reading installed and routing feeds to one of the IMAP mail accounts.
  • Spam and virus filtering (though I'm willing to let this slide until after migration if I can't do it the ideal way).
  • Automatic cleanup and archiving of email.
  • Access will be via SSH tunneled ports to localhost. All access will be "local".

Things I'd like to see, but don't need to happen before migrating off of the old setup.

  • LDAP server for contacts.
  • Whitelisting any email from someone in LDAP server.
  • Procmail rules to enhance pre-processing of my email.

I've used literally a dozen or more different guides to set up email servers on Linux. And, without fail, until last night, they all broke down at some point. Most of them focused on how to set it up if you are setting up a "real" mail server, i.e. one that has a domain and sends and receives email for that domain. I'm not going to register a new domain just for this because it's for collecting the email I already get. In other cases, by the time I'd reached step 2 or 3 and the instructions said: "you should see XYZ at this point and that means it's working", I'd see nothing of the sort and there were no followup instructions on what to do if you didn't see what the tutorial mentioned.

This time around, I went with the instructions from Spencer Stirling. They worked, but not because they matched up exactly. It's just that where they didn't match my reality, they were either close enough or it didn't matter and it still worked anyway. I didn't do any of the security bits yet, but plan to put them in place where it makes sense for my setup.

So, what's working as of tonight (after the 2 hours last night and a bit of futzing this morning before work) is:

  • Ubuntu Edgy Server in VMWare running and patched
  • Exim4 running
  • Fetchmail running
  • Courier IMAP running
  • Mutt is running and reading email
  • Squirrelmail installed and working with IMAP
  • LAMP stack installed for PHP tools for RSS
  • PHP-CLI installed for commandline scripting of RSS tools
  • Basic documentation page started in the server's web server for things I need to remember how to do down the road, like set up new users and links to the tutorials I used to set it up.
  • Email from one of my legacy email addresses is getting fetched and delivered appropriately by fetchmail.

All in all, looking good. Well before the migration needs to happen, I'll be there for regular email and the RSS setup should be close behind.

That is, until I tried to run this virtual appliance on another computer. Suddenly, the only network interface available was the loopback to 127.0.0.1. While there truly is no place like home, I'm not thrilled when it's the only place you can go.

Turns out that, unlike Ubuntu Workstation, Ubuntu Server does this thing where, if it gets a new MAC address on a network card, it won't use it on the same network config. Since VMWare hands out new MAC addresses on the virtual network cards like candy, this is likely to be a huge problem for anyone who intends to use VMWare appliances on more than one machine.

After a lot of brow furrowing, I finally found a solution. This forum posting explains what exactly is going on (though from the perspective of a rare move between hosts).

Here's the script mentioned in the forum posting that "resets" the MAC address/networking when this happens. Put it in /usr/sbin/ under something like "fixmac" and make it executable (chmod +x fixmac). Then, when this happens, log in and run it as root and reboot. It will be back to normal.

mac=`ifconfig -a | grep "HWaddr" | cut -d " " -f 11`
echo "eth0 mac $mac arp 1" > /etc/iftab

This is still a pain though and I'm looking for a better option long term as I don't want to have to go through that every time I move it (which is likely to be more than once per day). At any rate, given the difficulty I had in finding the source of the problem, I thought I'd share what a fix is.

Cleaning Up My Handwriting: The End of an Unannounced Challenge

Jan
28
2007

I have horrible penmanship. It's always been horrible. Back in elementary school, when it sat on my report cards along with things like "gets along with others", my otherwise glowing grades and "Exceeds Expectations" was a glaring exception. Under penmanship, it pretty much always read: "Needs Improvement".

Eventually, they stopped grading me on it, but from there it actually got worse. As I entered college and had nearly everything carrying a requirement of being typed, it both continued to get worse when I actually did write things down (because no one else was reading it and criticizing it) and the culture surrounding me enabled me to keep heading further and further down this path.

I started feeling more exposed recently when I wanted to hand-write some thank you notes and write a paragraph in a birthday card. It's downright embarrassing and, with the 1st of the year, I decided to do something about it.

I wasn't sure how far I'd get on it and so I didn't mention it on this site. Similarly, despite being convinced of the benefits of doing this, and feeling embarrassed by what I had, I still felt *really* weird doing penmanship exercises. I pretty much kept my activities on this under wraps for all of January.

After looking at some of the materials online and in books for teaching handwriting, I ended up working on something fairly close to "standard" italic handwriting. It seemed the most reasonable approach for an adult to take. It doesn't have nearly as many wasted strokes as the cursive I was taught as a kid, but still made the kinds of connections I was already making between letters in my chicken scratching.

I put in about 2 hours each weekend and made an attempt to use my "new" writing whenever I needed to write something down during the month.

A month or so into this, I thought I'd share my results. At the moment, it's not as neat as it can be when writing REALLY slowly, but the samples were written fast enough to be useful in real life. That means that I can write like the "After" for real at this point.

Here's the text I used for my before and after samples. It's a poem by Stephen Crane that I've always liked.

In the desert
I saw a creature, naked, bestial,
Who, squatting upon the ground,
Held his heart in his hands,
And ate of it.
I said: "Is it good, friend?"
"It is bitter - bitter," he answered;
"But I like it
Because it is bitter,
And because it is my heart."
-Stephen Crane

Here's what my handwriting looked like at the beginning of the year.

My Handwriting "Before"

You can imagine how difficult it is to read that 6 months or a year after you write it down. The letters aren't the same from word to word, some are on top of others, and others are little more than slightly darker spots on a line.

And, here's what it looks like now after 30 days of working on it. I used a book I picked up at Amazon: Write Now: The Complete Program For Better Handwriting as the guide for practice. It felt a bit silly as an adult to be tracing letters, but I think it was good for retraining the muscles.

My Handwriting "After"

I'm still going to keep working on it a bit here and there as an ongoing project. Given that it's now no longer humiliating to share handwritten notes and those of you who have to stare at a whiteboard that I'm writing on are no longer subject to the punishing experience you were before, I think further increases in speed and legibility are definitely feasible. I've also got plenty of room for training the muscle memory for words I'm not used to writing regularly.

Overall, this is one of the better 30 Day Challenges I've done. I'm really happy with the results.

MP3s Masquerading As WAVs Suitable for Ringtones

Jan
28
2007

A while back, I decided to switch back to TMobile from Sprint/Nextel and get Shelly and I back on the same plan. Part of it was that it's become clear over the last year or so that TMobile's service has gotten much better in our area. Still more if it has to do with the lopsided usage that has one of us (the royal "us") paying for extra minutes while the other doesn't use theirs, making it a much better prospect to be on a shared plan. Then there's the fact that Shelly's phone died. Lastly, I'd gotten accustomed to the ability to buy unlocked GSM phones instead of selling my soul and years of cellular loyalty for $150 off of a phone and Sprint/Nextel isn't GSM compatible.

So, about 6 weeks ago, we decided to do a bit of a telephonic shuffle. We added my number to Shelly's plan as a "new" line. This came with a basic "free" phone. We pulled the SIM card out of her dead one and put it into this new one and her dead phone problem was solved. We put my new SIM card into an old Sony Z600 that I had in the electronics-I-can't-bring-myself-to-throw-out box. It isn't a bad phone, but there is a reason why it was in the box: the hinge was sloppy and the battery life kind of sucks.

So, I found myself a cheap unlocked phone that might meet my actual needs (ignoring technolust). There was a decent refurbished smartphone for just over $75. When I looked back over the last 2 years of actual phone usage, my actual needs matched up pretty well.

  1. Outlook sync. I have all of my contacts, calendar and tasks in Outlook and this puts those things together in my phone without a PDA, which experience has shown to be not terribly well used in my life.
  2. Standard USB-mini connector for power and sync. Any one of the dozens of basic USB cables can be used to charge this thing. My phones ALWAYS go dead when I'm somewhere that I don't have a charger handy. I'm rarely somewhere where there's not a USB cable and a PC lying around.
  3. Automatic profile switching based on calendar. If there's a meeting scheduled, the phone goes on silent mode automatically.
  4. WAV ringtones. I hate having music as a ringtone. I usually get something that sounds like an old AT&T phone or something else that's more subtle than obnoxious music. Having WAV or MP3 ringtones is a way to get exactly the right sounds to let me know the phone is ringing, while being subtle enough to suit my sensibilities.

It's the last bit that led me to my problem on Friday. I went looking for WAV sound effects, hoping to find something that I'd be OK with having go off in a room.

I dug around a bit and found one of someone just clearing their throat. I thought that might work for me and figured it'd be a good test of how to set these up. It was then that I discovered a nasty little secret that all of our wonderful It Just Works™ software has been hiding.

See, when I put the WAV on the phone, it just plain didn't work. I figured it might be something like being at the wrong sample rate or something, so I looked at the properties of the file. It was there that I discovered that this file that the site (which was WAV only sound effects) wasn't actually a WAV, but an MPEG Layer 3. It was a stinking MP3 file, renamed to a .wav. A quick trip through Audacity and I had a *real* WAV that, sure enough, actually worked on the phone.

So, I wondered, how many other files on this site were labeled as WAV incorrectly. The answer? All of them. I tried a couple of dozen and all of them were actually MP3 files that had been renamed as WAV. I suspect that the person running the site doesn't even know it (which is part of why I'm not linking to the site). That's because all of the software we use for listening to audio files handles both kinds and silently cleans up a mess like this when we send it the file.

This is one of the reasons I prefer working a little lower with my tools than at that shiny, happy, user friendly layer. At least I can see what's going wrong when I don't get an expected result. That's why, even when I use a Mac, I'm not drooling over slick, simple applications that Just Work™. That's because, while they do work most of the time, when they fail (and EVERY piece of software I've ever used on every platform has something go wrong eventually), fixing them can be next to impossible because it's a sealed black box.

At any rate, I'm just going to run any file I want to use as a ringtone through Audacity from now on and sidestep the problem. And, that phone clearing its throat is mine. Just let it go to voicemail.

The Wynia Curse

Jan
26
2007

I'm home from work today with back pain that makes my moving around look like I belong in a nursing home. While I'm confident I'll be back to normal and at work on Monday, this caps off a week that is almost making me a believer in the Wynia Curse.

Officially (as officially as a family tradition gets), the Wynia Curse is mostly a coping mechanism for the series of unfortunate events that seem to happen with disturbing frequency to everyone in the family line. These events are not just strange for their frequency but for the circumstances that surround them.

For instance, when I broke my left wrist when I was 14 (in itself an unfortunate, but not uncommon occurrence), I didn't do it in any "normal" way. I broke it on the ferris wheel at an amusement park while on a youth trip for church. After I broke it, my friends left me (thinking I was faking) and I sat on a bench, in shock. As I sat there, a rain storm that ended up dumping 2 inches in a little less than an hour and a half started and my situation became wet in addition to pathetic.

More recently, my grandmother was sitting in her lift chair in the nursing home and the controller shorted out and started on fire and spread to her clothes and chair. She was alone in the room and couldn't reach the call button to get a nurse. So, by the time the screams brought someone to the room, she had 2nd degree burns from the fire.

When this kind of stuff happens to your family on a *regular* basis, but somehow it all ends up OK, you sort of develop a darker sense of humor than you might otherwise have and the Wynia Curse is part of that. When this stuff happens, it often gets just brushed off as being part of the curse and we move on. It's pretty much a coping mechanism for dealing with

Now, my rational brain does NOT believe in curses. Many of the unfortunate events in my own life could have been prevented by being more careful (if you're using a pitchfork in chest deep water, harvesting chopped seaweed, make sure you can keep track of where your feet are if you don't want to run the implement through your foot). Many others in my own case were just freakish and that sort of crap happens. My upper brain is 100% certain that there's no actual curse. However, the lower, less rational portions of my brain kicks in when this stuff happens and is is *just* as certain that not only is there a curse out there, but it's taken me and my family as a full-time job, occasionally taking on an assistant to do the job right.

Unfortunately, like all of the rest of these battles between instinctual emotion and rational thought, it's a more difficult battle than it ought to be. However, every time I battle through this and put the rational side in charge of the issue, things *always* turn out better.

I basically treat my brain as a computer with an old operating system that I can't upgrade, swap out or otherwise change. Running on top of that are various bits of software that I have to work on building and patching that compensate for the inadequacies of the lower level operating system.

Here's to writing a handler for the curse.

This is Why I Love Creative Commons

Jan
18
2007

In the last couple of days, I've seen a couple of really good examples of why I really love Creative Commons. People have been using CC-licensed for the exact kinds of things that they should be and in ways I hadn't planned for or couldn't have necessarily expected. Beyond that, the people involved spanned the globe in ways that just wouldn't have happened even a few years ago.

First, on Tuesday, a guy named Matías, from Argentina enjoyed my little humorous look at marriage and datatypes (much more than my wife did, I might add). He enjoyed it so much, he decided to translate it into Spanish and post it on his site. While he asked my permission, he didn't need to because of the license for all of the content on this site (unless otherwise noted, it's CC-BY).

Now, those on the geekier side of life who life life en español can dig the article too.

Second was Roy Blumenthal in South Africa, who, upon getting an art project to make something about "80s", went looking on the Flickr photo finder for a CC-BY image that matched his vision. From that image, he did a sketch and eventually a painting and a lineart version. He released the resulting art under another Creative Commons license (adding the non-commercial clause as he's allowed to do).

I'm finding it funny how Creative Commons is creating enough of an incentive to result in all of this extra expression, despite all of the work being under less restrictive terms than standard copyright. *MORE* stuff is getting created *and* society at large gets more access to it than when it's locked up to wait for someone to decide that it will make enough money to be worth releasing.

« Older Entries  

J Wynia

For better or worse, I'm the guy who runs things here. I'm a web consultant, software developer, writer and geek from Minneapolis, MN. This site is a fairly wide cross-section of the things I'm interested in and enjoy writing about.

Oh, and if you happen to be looking for hosting for your Subversion repositories or just web hosting in general, take a look at Dreamhost. It's what I use for Subversion and your signup helps me out.

Latest Microposts

jwynia: is ripping the first DVD on the new Thinkpad. Holy crap this DVD drive is quiet and smooth. No jet engine takeoff.
jwynia: is unsubscribing to a bunch of mailing lists that he's been deleting without reading for WAY too long.
jwynia: @bethdean if I ever get to the point of having an office and staff for my consulting, there WILL be a microwave popcorn ban.
jwynia: is wondering whether his intent to spend his stimulus check in Ireland is weird.
jwynia: is baffled by what a hot commodity the screen-cleaning spray has become in this office.
Follow Microposts on Twitter | Subscribe to Microposts

My Attendance At the Gym

Feeds and Links


www.flickr.com
This is a Flickr badge showing public photos from J Wynia. Make your own badge here.

Search


Pages

Archives

© 2007 J Wynia. All original content is licensed under the terms of the Creative Commons Attribution license unless otherwise noted. Content from other sources is licensed under its original terms.