Photo: Dan Zen

I make a swing past a bunch of sites like Freshmeat, C Sharp Source, CodeProject, Codeplex and Sourceforge
every week or 2, looking for new and updated C# (and PHP, etc) projects and libraries that are up to interesting stuff.

One article on CodeProject today caught my eye. The article is entitled: Generate Complete Web 2.0 Applications in Minutes. That sentiment is all over the software development industry. The Microsoft Windows Server 2008/SQL Server 2008/Visual Studio 2008 launch last week showed the attitude at several different points.

Basically, it boils down to excitement at the ability to quickly build "applications" with little more than a wizard. That exact sentiment was expressed to me multiple times at a consulting company I worked with for a while. The person in question believed that if we just used one of these magical tools, our projects would go from a couple of hundred hours down to a single day.

The problem with this whole approach to software development is not that the tools themselves aren't great or that they promise something they don't deliver. It's that they're asking a question that doesn't matter in the context it's getting asked.

That's because neither I nor any software developer that I know is collecting a paycheck or getting an invoice paid for "building applications". At least, that's not what the payment is in exchange for in economic terms. Rather, I am getting paid for *solving problems*.

And, the thing about getting paid to solve problems and magic tools that make some kinds of problems easier to solve this year than last year (which is what most of these tools actually do) is that that just changes the problems that people pay for.

When tools that do easy CRUD generation or data entry applications show up in the marketplace, that doesn't mean that software developers magically get to use those tools to only work 2 hours a week and then coast for the rest. It just changes the kinds of problems that earn developers money.

I love this chain of progress. That's because it lets me solve ever-increasingly difficult problems with the same effort. That's very cool. However, it doesn't promise easy street and it bothers me when that's what's implied by this whole attitude.

Photo: Kevin Dooley

If you were to go through my Visual Studio "Projects" directory or my personal development web server, you'll find about 2/3 of the directories are named SomethingPOC or SomethingExperiment. The former has become my convention over the last year or so and stands for Proof of Concept.

I spend a good portion of each day with one or more of these projects open. That's because I consider the use of the Proof of Concept to be integral to software development. Whether the official methodology of the project encourages them, is indifferent or actively discourages them (and I've worked in all of those environments), I will insist on using them.

First, a quick explanation of what exactly I mean by a POC. Basically, it's the simplest possible program that will answer a question that you have about the tasks in front of you.

Say, for instance, that you wanted to retrieve an RSS feed and store the individual entries into a SQL Server database. I'd probably do a quick POC to connect to the database and insert a record. Then I'd do one for fetching an RSS feed. I might also do one that checks a feed for new items vs one's seen before.

In other words, each POC tests out one concept and proves that you can do what was a question mark in your project approach. If you end up with more than one method in a POC, you're probably doing too much and it should be broken down into more than one POC.

I deliberately name the projects and classes with names that can NOT work in the final project. This helps to hedge against the inclination to do a quick copy and paste into your real project. This is important because POC's should be quick and loose. They don't have error handling, don't do validation (unless that's what you're proving) and generally don't follow many of the rules of good software development. That's a good thing.

That freedom means you can quickly explore the problem and work through some possible solutions in a "sandbox" without worrying about whether you're doing it "right". However, it's also a good thing that you throw the POC away or only use it as a reference.

If you do lots of POC's, name them to encourage disposable coding and then move on to do your "real" development, you'll find that you have often left those crappy early mistakes in the POC, have already run into and overcome many of the typical problems you run into in new solutions.

Once you're into your "real" development, lots of people abandon POC's. However, I keep using them throughout the project (even into the bugfixing and testing phases). Every time I'm asking myself a question about whether an idea will work, rather than trying the experimental code in the permanent code.

Over time, this ends up being a constant cycle. You ask yourself a question that can only be answered with code, do a POC to come up with an answer and then move back to the full project to implement it. If you aren't used to this kind of cycle, I'd recommend giving it a shot. I won't work without it.

For a while now, I've had a project on the back burner for a different set of tools for RSS reading, writing and publishing. I'd like a single toolchain that lets me keep everything together in one place. I've got piles of notes, a few proof of concept projects and the start of several of the components.

Last night, when I couldn't sleep, I decided to check something off of the list that I wanted to see as a proof of concept for the Atom Publishing Client part of the toolchain: HTML Tidy cleanup to XHTML of HTML before putting it into an Atom entry document.

I currently do most of my writing for this site in Windows Live Writer. However, that's more of a compromise than an ideal choice. While I could probably hack a plugin together that would make Live Writer a more suitable long-term choice, I really want a very specific set of features that includes getting away from the XML-RPC API that all of the server-side engines Live Writer works with are based on.

So, I've been tinkering with a multi-tabbed Windows app for editing posts. The WYSIWYG tab for quick editing uses the MSHTML engine from Internet Explorer. I've looked around and unless you're willing to pony up $299 for a commercial control, that's the most reasonable choice.

However, the HTML that MSHTML spits out is horrible and really needs to be cleaned up. So, I set out to figure out how to use HTMLTidy in a C# project.

I tried to find a .NET wrapper for HTMLTidy and thought I had scored right away when I found one here. However, when I tried to use it, not even the sample code would build without errors on my development machine.

So, I dropped back to trying the COM object version. The last update to it was back in 2000 or so, but it looked like all of the features I needed were in that version, so I decided to give it a shot.

To use the TidyCOM library, you add it as a reference and insert your "using TidyCOM" statement in your class. The actual usage is fairly straightforward.

Example:

TidyObject TidyObj = new TidyObject();
TidyObj.Options.Doctype = "strict";
TidyObj.Options.DropFontTags = true;
TidyObj.Options.OutputXhtml = true;
TidyObj.Options.Indent = TidyCOM.IndentScheme.AutoIndent;
TidyObj.Options.TabSize = 2;
String CleanHTML = TidyObj.TidyMemToMem(HTML);

That code assumes that the "HTML" variable has your messy HTML in it and at the end, "CleanHTML" has your cleaned up XHTML in it.

My little multi-tabbed prototype is using a buffer object to keep the "current" HTML in it. Whenever you switch tabs, the old content is scrubbed through this code before the new tab gets updated out of the buffer. That means that whether it's the WYSIWYG tab that messes it up or you in the HTML editor, you still get valid XHTML in the eventual output.

I also extended my CleanupHTML method (that contains the above code) to scrub out the HTML header tags, body tags, etc. Since the HTML will actually end up as one part of the Atom xml file and not as a standalone HTML file, I only want the content from the editor and both MSHTML and HTML Tidy will always put that stuff back in unless you strip it out.

While I'd still like an assembly that's a little more current, this clearly does the job well enough to check this feature off of my checklist. Now, on to RESTful services on IIS with C#.

Back a couple of weeks before Christmas, an unfortunate series of events resulted in my 4GB iPod nano going through the washing machine. And, as is the case with many such incidents, it rendered the device entirely non-functional.

I bit the bullet and picked up a replacement because it's fairly critical to my mental well-being. I need to be able to throw on a bit of music or a podcast when I'm driving or need to drown out the outside world. So, I ordered a new 4GB nano video.

Then, a couple of days after getting that nano and using it, the consulting company that I'm subcontracting through gave all of the consultants on my project a 30GB Zune.

That, of course, caused a quick re-org of my portable media strategy. I ended up making the nano a purely podcast and audiobook device and moved all of my music (and hopefully some video when I get it cooperating) to the Zune.

Unfortunately, revisiting nearly any process in one's life can quickly shine a light on previously ignored problems and make you re-question your solution. Such was the case with my podcasting listening. When I switched over to the nano, I missed the ability to listen at faster-than-normal speeds for spoken word podcasts. However, the other benefits outweighed that downside, so I moved on.

However, last night, I wondered if I couldn't just convert some of those podcasts into iPod audiobooks (the ones with .m4b as the extension). Several of the podcasts already distribute in that format and you get things like bookmarking of where you were in the audio as well as the ability to speed things up.

I'll leave it as an exercise for the reader to guess what I think about a device that allows only the speeding up of a specific class of audio files to the exclusion of other classes of audio files.

Regardless, I checked if the Swiss Army knife of audio/video: ffmpeg would be able to handle it and was happy to see it would. So, I wrote a really simple C# console app to convert these .mp3 files into .m4b files. The workflow is still a bit lacking as it doesn't easily tie in to the iPod/iTunes functionality of putting "the 1 latest unplayed" podcast from each feed on the device, but I can quit hoping that the people speaking would just hurry the hell up.

There were a couple of oddities that needed to be coded around. Most noteable is the proper quoting of file paths (which still might not work in all cases) and the fact that you need to actually convert from MP3 to M4A and then rename that M4A to M4B in order to be done.

At any rate, the code is below the fold if you would like to mess with it yourself. You call it by running

MP3toMB.exe input.mp3 128

It also works if you just drag an MP3 onto the exe by assuming a 96kbps bitrate. Obviously, this is coded for Windows, but the same principle could easily be accomplished on Mac with Applescript or Linux with shell scripting or just batch files on Windows given a commandline copy of ffmpeg.

I've also got a Windows app that combines lots of MP3's into a single audiobook file, but I wanted something that I could use to automate and run on a more nightly basis to convert stuff, which this gives me.
Read the rest of this entry »

Photo: Stian Martinsen

In several conversations recently with other software developers (yep, those are just as exciting as your wildest dreams) and their frustrations with the process, as implemented in modern corporate America, the same analogy kept popping into my head.

More and more, I feel like the things that businesses are after in their software development are similar to medieval alchemy. For 2500 years, the entire field that eventually became chemistry was obsessed with 3 basic questions:

1. How can we change lead (or other metals) into gold?
2. How can we create an elixir that will cure all diseases and prolong life indefinitely?
3. Can we discover a universal solvent?

All of these strike us as goals that weren't even attainable. Yet, the underlying desires often did get met when the focus shifted to what eventually became modern chemistry. By dropping the focus on the single, universal solution and just figuring out how to treat individual diseases or how to dissolve individual compounds or just fundamentally understand chemistry, many advances did happen.

Many/most of the diseases that the alchemists sought to cure or treat are under control today. There's very little in the world of chemistry that we can't tear apart and we can do things like convert coal or corn into one of the most sought after substances on earth: liquid fuel for transportation.

One of the consulting firms I worked with had a project manager that was constantly pushing the developers to find and use "automagical" tools to build our solutions. What he was after was the kind of IDE or tool that, with a few clicks, would just spit out a nearly complete solution.

That would, of course, result in the sales force being able to sell expensive solutions that could be fulfilled in minutes instead of days and weeks. It didn't matter how often I pointed out that, as a consulting company, if our clients' solutions were so simple that a few clicks and config options could solve them, they wouldn't bother coming to us: they'd just buy the software themselves.

This same person wasn't very excited about things like loosely-coupled systems and/or Service Oriented Architecture unless they also came with wizards that let you choose 4 or 5 options and they'd just spit out a fully-realized application. Yet, those approaches keep working for me as a way of looking for patterns in companies' problems and solving them quickly and completely.

Instead of looking for the tool that spits out C#, PHP, ColdFusion and Ruby, I'm looking for repeating problems like managing queues of objects to be processed. Once you have an approach to that general problem, a good developer can probably implement it in whatever language they're most comfortable with.

That's due, in large part, to the fact that the bulk of the work as a software developer is NOT in typing in the text of the programming language in question. Douglas Crockford said in one of his Yahoo video lectures something along the lines of: a developer could probably type up all of their code for an entire year in a day or 2.

Yet, many of these automagical tools really only seem to automate the stuff related to typing code, not for solving problems. And, like I said a couple of days ago, if you're in the consulting game or just looking to stay employed as a developer, the money and jobs are where the problems are.

That's why, when I hear someone looking for that quick and easy tool that will "just" take care of it this afternoon, I tend to interpret it as, "Can't we just change this lead into gold instead of getting real gold?"