I’ve been interested in the stuff that I’ve heard about git since hearing that Linus Torvalds was ditching BitKeeper and writing his own replacement to handle version control for the Linux kernel. I’ve taken a look at it a few different times as it’s profile grew in the Rails/Perl/etc communities.

Every time, there was something that kept me from adopting it: the state of Windows support. And, after this past week, I think that my focus on the msysGit version was part of the problem. I have been reluctant to use the Cygwin version. That is mostly because I didn’t want to *have* to install all of cygwin on any workstation where I want git support and the pile of files that adds.

The thing is, though, that when I took a good look last week, I already had cygwin running on every single Windows machine I’ve got, INCLUDING virtual machines. Never one to cling to a supposition after evidence invalidates it, I gave git on cygwin a shot.

Because I still have lots of Subversion repositories and am likely to continue using it on lots of projects, I insisted on getting the git/Subversion integration working. Also, I use a Dreamhost account to host my repositories and any project-related forums, project tracking, etc., as a sort of private Sourceforge/Codeplex/Google Code without any licensing issues to deal with. I wanted to add git support to that existing project space.

It turned out like most projects of this sort. The stuff I thought would be a pain (like getting git running on the Dreamhost account) turned out to be documented and straightforward, while stuff like getting the “git svn” command to work (which should just work off the cygwin install) thwarted me for far too long.

However, after I worked out the kinks, I was able to get it working and pointed at some of my Subversion repositories on several of my workstations. After cloning those repositories, what I'm most suprised by (even though I was told to expect it) is how git manages to copy ALL of the version history for the entire repository into a single .git directory (instead of hundreds or thousands of .svn directories) and end up with something SMALLER than the single version checkout of Subversion for the same source tree.

As an example, one of my repositories had 440 revisions across 10 branches. A checked out copy from Subversion of the current release branch is 2.51GB in 43,655 files across 23,361 folders.

After moving it to git, the checked out copy, which now includes the ENTIRE 440 revision archive of project history is 1.51GB in 15,356 files and 2,233 folders.

As far as the hiccups, it boiled down to a couple of things:

1. Make sure to install all of the “git” related packages in cygwin.
2. Install the subversion libraries for perl (they’re not anywhere near the “git” packages in the cygwin installer). This took care of it complaining about the SVN CPAN Perl library.
3. Be prepared to have to run “rebaseall” on the cygwin installation.  This took care of a LOT of cygwin complaints about various DLL’s.
4. Running “git svn clone http://example.com/repos/” can be agonizingly slow on large repositories. It’s not an issue if you’re working off of a tiny project, but that 1.5GB project took all night and then some. Ongoing work is quite fast, so I’m hoping they work the kinks out of this going forward or that they’ll ditch the tagline “The fast version control system”.

Given how efficient it is with storage, I’m interested in experimenting with running it in the background on my development directories and having a FileSystemWatcher commit whenever things change. I’d like to see how long it takes before things get out of control.

I’m also hoping that the GUI tools catch up. That’s not so much because I want them for myself (I prefer commandline for SVN as well), but because TortoiseSVN has been key in getting SVN adopted on projects and flushing SourceSafe on several of my projects.

My current big project is a rewrite of a platform that has 3 predecessors, all of which are still running, with implementations on them. The original gig was to dig into 2 of those platforms to get them back to running because all of the existing developers had left. Back in December, we were asked to tackle writing platform number 4.

One of my “rules” of software development on any project where I have a say in such matters is “You don’t get to rewrite it unless you understand it.”

That one made it on my list because of how common it is to watch software developers take a look at code some other developer write for 15 minutes only to declare it “crap” and in need of a complete rewrite. When challenged to explain what the offending library/package/application does, most can’t. That leads to a near 100% certainty that the rewrite will miss some large requirement or miss something subtle but important in how the original works.

So, as we set out to rewrite this mess, we wanted to make sure we understood what those older platforms do. We read all of the original requirements, dug through data structures and code, talked to the people who helped define the requirements on the original, etc.

One of the common threads that went through everything related to all of the implementations was the overly complicated ways that things were done in an obvious attempt to accommodate a wide variety of permutations. However, the complexity actually got in the way of accommodating those same permutations.

As we worked through the features that would be in the initial version of version 4, we pushed back on every assumption we could, hard. It’s something I’ve always done to one degree or another. On this project it’s been with more discipline than in the past.

Now, 4 months later, that has proven to be a critical strategy. That’s because optimizing your thinking about the problem and how you’ll solve it leads to orders of magnitude more productivity than most software tools.

If you add up the time spent actually typing code into an editor or IDE, compared with time spent trying to figure stuff out and otherwise spent thinking, it’s probably 30/70. That means that even if you had tools that could read your mind and instantly do exactly what you wanted, you’d only speed things up by 30%.

In trying to get other people to push back the same way, I’ve struggled to describe it in the kinds of rules that people un-used to doing this kind of thing (beginners in the Dreyfus Model). This week, one common factor in many of these situations popped out at me (Strategic Intuition at work). When subject matter experts asked us questions that would have led to complicated messes, they were almost always expecting a boolean answer.

They were questions like:

• Can this new platform do X?
• Will it have feature Y?
• As a user of this system will I be able to do this specific task?
• Can you guys make it do Z?

While those questions can be answered, often easily, doing so almost universally is the wrong thing to do.  Under those kinds of questions are giant piles of assumptions. When you answer these questions without probing deeper, you validate ALL of those assumptions without ever knowing what they are.

If, instead, you play the 3-year-old and ask “Why”, you can drill down into those assumptions. You can discover that many of them can be set aside or rendered moot by a much simpler design. And those that stand up to the incessant “Why” are worth keeping and now you have a much deeper understanding of what’s driving the feature request.

Of course, it’s an added perk that rejecting a black/white view of an issue fits in well with the Glass Too Big philosophy too.

One of the last things I posted here was how I was toying with flexible content and template solutions in ASP.NET MVC. For that project, we're going forward with the solution that emerged from that experimentation and I'm excited to see how it turns out. 

Along the way, an interesting question about how to create what are essentially plugins for the MVC framework. Basically, if what you're building is an MVC app that will be deployed in dozens, hundreds or thousands of independent implementations, it becomes obvious pretty quickly that you need most of the pieces of the application to be modular so they can added/removed/overridden at least to a minimal level.

I've come up with what I think is a workable solution, which is pretty much just gathering up some of the other ideas already out there into a quick-n-dirty working prototype. The basic idea is this:

Put an encapsulated set of model classes (or none if you depend on central model classes), controller classes and a set of views into its own DLL. The views (using the default WebForms view engine, they're aspx files) go in as resources.

Pull that DLL into your MVC application and the MVC stack inside your "plugin" will work just like if those bits had been a "natural" part of the base MVC application. In fact actually creating the pieces from inside the main app and then moving them over to their own project is the quickest way I found to actually create the bits for the plugins.

When the question came up, I vaguely remembered seeing a question on StackOverflow about how to put views into an assembly/DLL. That question along with one on VirtualPathProviders provided most of the kernel of what I needed.

So, on to how to actually do this.
Read the rest of this entry »

I've been using Twitter off and on since late 2006 and have racked up almost 2000 entries in that time. Earlier this year, when Twitter hit stretches where the service was out for entire days at a time, I started using Friendfeed to do the actual posting and pushed those posts over to Twitter.

Part of what sent me over to Friendfeed was a sinking feeling that most of those early posts would go down with the ship and I wanted to get some redundancy in where that ephemera was showing up online. Most of those posts were throwaway (see Sturgeon's Law) but I also know how much I enjoy going through slices of complete ordinariness after my memory of the events has faded. So, I want to keep a copy of all of this stuff and figure out later what's interesting.

At any rate, given Twitter's problems, I tried to get those old posts out whenever things would work for a patch. Unfortunately, even when things were working, if you went back through the pages of the archive, you'd get about 10 pages in and hit a wall. Given that my Twitter history is about 100 pages long, that left the vast majority of my posts behind that wall, possibly never to be seen again.

However, not wanting to resign myself to just having an archive going forward, I put an entry on my calendar to keep trying every few weeks, just in case they eventually fixed things. So, tonight, the reminder popped up and, what do you know? I could go through all 99 pages of my archive.

Not wanting to lose what very well may be a 1-time window of opportunity (they've pretty much said that the instant messaging isn't coming back), I banged out a quick console app to grab all of my posts into files for later processing.

It's mostly copy-n-paste stuff (don't judge, it was written while I was eating curly fries and a roast beef sandwich), but I did include bits to hold off for an hour and wait if it gets throttled by the 100 requests per hour limit from Twitter. If you've got a lot of stuff on Twitter, running this may best be done overnight.

At any rate, it grabs the posts 20 at a time and saves them to a directory as .atom files. You set the Twitter username and password in the app.config file like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="TwitterUsername" value="jwynia"/>
    <add key="TwitterPassword" value="IM_NOT_THAT_DUMB"/>
  </appSettings>
</configuration>

And then a basic console app (with some extra namespaces from stuff I threw away still in there), looks like this. I didn't bother with any of the API wrappers, just using basic .NET code to do it.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Configuration;
using System.Collections.Specialized;
using System.Security;
using System.Net;
using System.IO;
using System.Xml;
using System.Xml.XPath;

namespace TwitterArchiver
{
    class Program
    {
        public static String Username {get;set;}
        public static String Password { get; set; }
        public static NetworkCredential TwitterCredential { get; set; }
        static void Main(string[] args)
        {
            NameValueCollection appSettings = ConfigurationManager.AppSettings;
            Username = appSettings.Get("TwitterUsername");
            Password = appSettings.Get("TwitterPassword");
            TwitterCredential = new NetworkCredential(Username, Password);
            String OutputDir = Username + "-export";
            Directory.CreateDirectory(OutputDir);

            int NumberOfPages = GetUserPageCount();
            for (int i = 1; i <= NumberOfPages; i++)
            {
                String ArchivePage = GetArchivePage(i);
                WriteStringToFile(ArchivePage, OutputDir + "/" + Username + "-page-" + i.ToString() + ".atom");
                System.Threading.Thread.Sleep(1000);
            }

            Console.WriteLine("=====Press any key to continue=====");
            Console.ReadLine();

        }
        public static void WriteStringToFile(String InputString, String Filename)
        {
            StreamWriter sw = new StreamWriter(Filename);
            sw.WriteLine(InputString);
            sw.Close();
        }
        public static String GetArchivePage(int PageNumber)
        {
            Uri TwitterBaseUri = new Uri("http://twitter.com/statuses/user_timeline/" + Username + ".atom?page=" + PageNumber.ToString());
            CredentialCache myCache = new CredentialCache();
            myCache.Add(TwitterBaseUri, "Basic", TwitterCredential);

            WebRequest request = WebRequest.Create(TwitterBaseUri);
            request.Credentials = myCache;
            string responseFromServer;
            try
            {
                HttpWebResponse response = (HttpWebResponse)request.GetResponse();
                Console.WriteLine(response.StatusDescription);
                Stream dataStream = response.GetResponseStream();
                StreamReader reader = new StreamReader(dataStream);
                responseFromServer = reader.ReadToEnd();
                reader.Close();

                dataStream.Close();
                response.Close();
            }
            catch
            {
                System.Threading.Thread.Sleep(1000 * 3600);
                responseFromServer = GetArchivePage(PageNumber);
            }
            return responseFromServer;
        }
        public static int GetUserPageCount(){
            Uri TwitterBaseUri = new Uri(" http://twitter.com/users/show/" + Username + ".xml");
            CredentialCache myCache = new CredentialCache();
            myCache.Add(TwitterBaseUri, "Basic", TwitterCredential);

            WebRequest request = WebRequest.Create(TwitterBaseUri);
            request.Credentials = myCache;
            int NumberOfPages = 0;
            try
            {
                HttpWebResponse response = (HttpWebResponse)request.GetResponse();
                Console.WriteLine(response.StatusDescription);
                Stream dataStream = response.GetResponseStream();
                XPathDocument ResponseDoc = new XPathDocument(dataStream);
                XPathNavigator ResponseNavigator = ResponseDoc.CreateNavigator();
                XPathNodeIterator ResponseIterator = ResponseNavigator.Select("//user/statuses_count");
                ResponseIterator.MoveNext();
                int PostCount = int.Parse(ResponseIterator.Current.Value);
                dataStream.Close();
                response.Close();
                NumberOfPages = PostCount / 20;
                if ((PostCount % 20) > 0)
                {
                    NumberOfPages++;
                }
            }
            catch
            {
                System.Threading.Thread.Sleep(1000 * 3600);
                NumberOfPages = GetUserPageCount();
            }
            return NumberOfPages;
        }

    }
}

Build and run that and you, too, can see what you ate for dinner 2 years ago. I think I might put together a highlight post of my personal favorites from my archive to make this worthwhile.

At any rate, I need to get back to more pressing matters (the economy might be slow, but I'm busier than ever right now and what looks like into 2009, so I shouldn't complain). Later.

photo credit: j wynia

I carry a variety of notetaking devices. They all have their benefits and I tend to oscillate between pen/paper and electronic devices. However, the notebook nearly always tends to hang out around me and I dump ideas onto the pages as the day goes on.

Those notebooks have been filling up with ideas for projects. Combine that with some client projects and my shift in schedule (which hasn't freed up as much time as I'd hoped), and I find myself at the beginning of several projects at once.

This raises a few needs that have been hanging out there. Designing and starting a web application project involves a few things for which I've never been happy using the existing tools.

First, there's actual layout design. While Photoshop works fine for doing the "sketches", the people who are in a role to review those design ideas tend to get overly focused on things like font choice when you're trying to figure out whether you should use one screen or a wizard for registration only to suddenly notice that the wizard was chosen just as the app goes to production.

That's why my interest was piqued when Balsamiq came out a while back. It provided a way to do mockups of screens and keep people focused on the functional design instead of colors and fonts, just like I want. However, there are quite a few apps that I'd use more that also are in the $100 range (Balsamiq is $79) and I haven't bought those yet, so it's unlikely that I'll be buying it any time soon. Plus, I'm not exactly a fan of the Comic Sans font.

However, the other night, I was relaxing in front of the TV before bed and I figured I'd doodle a few sketches like the widgets in Balsamiq. As I was drawing, I also was thinking about Antonio Lupetti's technique for sketching on screenshots, which I've been wanting to use more in articles and presentations/screencasts, so I included some of those elements as well.

It didn't take long and I had 4 sheets of paper pretty well covered. Because I started by just sort of doodling, the elements didn't end up to scale, but I like the way some of the icons, etc. turned out. I'm thinking of fixing the tablet PC (battery won't hold even a slight charge and the stylus disappeared) and re-do them on a grid that will make them the right comparative size.

However, I did scan the pages to PNG's for sharing.

One of the other difficulties is that I've seen too many projects, particularly inside the firewall, receive horribly ugly and unusable web templates. At the same time, for my own projects, I'm not ready to pay a designer yet to do the final design (though I may be soon if you're a designer and interested), and need to get on with building things in the mean time.

So, I've been wanting a decent "generic" web application template that lets me focus on building functionality and, once that's nailed down, do the final design as more of a "redesign" from the generic template.

The thing is that, while there are thousands of web "site" templates and blog templates, there are very few that are oriented to web applications. There are so few, that something like 2/3 of the Google links for my searches led back to this 1 template. That theme is nice, but not the look I'm aiming at for my temporary template.

Among other requirements, like the sketchy mockups, I want the the ability to actually have a completely grayscale look to the initial prototypes to hammer home the temporary nature of the design. So, last night, after I tossed Revolver into the DVD player to finally watch it (incidentally, not the kind of movie to be working during as I'm going to have to re-watch it to entirely "get" it), I decided to take a shot at building such a template myself.

I grabbed a copy of the 960 CSS Framework (something else I've been wanting to mess with) and whipped up a batch of pages that covered the basic HTML elements, a bunch of form elements and a start on stuff for data display, like tables. I threw in a bit of jQuery for things like drop shadows and by the end of the movie, had some decent results. There are still plenty of little bugs (like the jQuery tabs didn't cooperate, so I threw some other ones in instead until I can figure that out), but I'm pretty happy with it so far.

It's grayscale by default, but all of the color definitions are in a single CSS file, so changing the colors is easy. All in all, it's pretty modular and I think it will turn into an ASP.NET master page quite easily as well as allow a complete swap-out for a different design later as nearly everything is pretty vanilla HTML tags.

I zipped up both the sketchy PNGs and the web application template together so you can download them if you'd like to take a look.

Download the Prototype/Design Toolset Package