Jeff Olson's bleg

Deployment with OpenWrap

2011-02-01T00:00:00-08:00

This post is a brief, hand-wavey outline of my efforts to integrate and abuse OpenWrap for the purposes of deployment automation in my shop. I began working on this post before starting the actual process of integrating OpenWrap and have updated as I’ve progressed. The goal should be that, when complete, it will provide an introductory outline of what a newcomer to OpenWrap (as I consider myself, prior to this post) will want to know/learn in order to competantly grapple with the topics contained herein.

I will begin by discussing some assumptions about the environment for this exercise (as they pertain to my particular situation), then talk about where my shop was before integrating with OpenWrap and what our current deployment process consists of. Next, I will outline where I want to be once fully integrated with OpenWrap. Finally (the important stuff), I will provide purposefully vague play-by-play of the conversion process (as, neccesarily, each shop’s needs and constraints will differ.. so excessive preciseness in the blog post will only serve to, ultimately, mislead you, the gentle reader) starting from a “vanilla non-OpenWrap’d” ASP.NET MVC 2.0 webapp.

It’s key to point out that, as far as I can surmise, using OpenWrap for:

Deployment automation
Packaging ASP.NET MVC projects

.. is a massive abuse of it’s intended purpose. I make no warranties as to the fitness of the process outlined herein and fully expect to inspire the Wrath of Seb as a result (have you ever seen how he browbeats the NuGet people?).

Some Assumptions

This post assumes that you’re working in a MS.NET environment deployed to some variant (most likely Server) of the Windows OS ecosystem. We’re still on Server 2003 (with a migration to 2008R2 in the next 12 months). It’s not-at-all clear to me what the Mono/Linux story is for OpenWrap, so I make no warranty that this process can be adapted for such an environment. My shop is still .NET 2.0/3.5SP1-based, so that is the point from which I’m starting in this post (.NET 4.0 is also on the 12 months plan, heh). Also, my actual production servers exist on a DMZ that is not domain joined and requires copious amounts of red-tape to be waded-through in order to open ports back into some of office backoffice machines in non-DMZ’d parts of our corporate intranet.

Our Pre-OpenWrap Deployment Process

Our current setup, roughly, looks like this:

Devs work and check-in to our mercurial repo, as needed
For every checkin, a jenkins build runs. The complete webapp project directory (bins, src, views, etc), post-build, is published as an artifact
Once we’re ready to deploy to prod, we have a special, manually triggered, jenkins build that will take the artifacts of the most recent, successful, build and do some config replacement for the prod environment (it’s configured for dev/test by default) and package the result into a zip. The resulting zip file can be decompressed on the root of the drive on our application servers (so it unpacks into Inetpub/wwwroot by default). This zip file is published as the artifact of the build
Whoever is doing the deployment (we’ll call them “Ops”) will go to the jenkins page and download the zip artifact from the last step to their local machine
Ops will then log into the application servers via RDP and copy the zip file onto the machine via the “Local Resources” exposure of their local hard drive

We’ve found that step 4 is the killer, as it appears to take ~15 minutes of manual time to copy the resource across the intranet. As noted above in the “Some Assumptions” section, the app servers are on a DMZ and all ports back into the intranet are closed by default (including network file shares) and the servers are not domain-joined. My suspicion is that this method of copying large files (the prod deployment zip is ~35MB, currently) across the Local Resources pipe (or, perhaps more aptly, straw) is sub-optimal. I’d like to test how long it’ll take when done via HTTP or other methods. I get a feeling that SMB fileshares will be a non-starter, here, as it appears to require the opening of 4-or-more ports and not to mention the whole Active Directory component that isn’t present on servers in the DMZ. I’d love to check out rsync, but I get the impression that it’s support is kind of flakey on Windows. So I’m interested in seeing what an HTTP transfer of an equivelant size, from the intranet, will look like time-wise.

But another component of what’s so horrible about Step 4 is not just the time, but the “disconnect” that it creates in what is otherwise a mostly fluid process. That is 15 minutes that an Ops person has to sit there and twiddle their thumbs waiting for the process to complete. It’s easy to get distracted and not check back until well after the copy has completed. I would like to get to an end-to-end automated process so that, regardless of how long it takes, we can click a button and know that the deployment will be finished, eventually, without dependence upon human intervention. An email notification of completion would be nice as well, but not neccesary.

Where we want to be

Still have the jenkins build(s) as detailed above
Chances are the manually-triggered “prod config/zip” jenkins build (detailed above in step 3) will be modified to also include publishing the resultant artifact to an intranet-based OpenWrap repository server. OpenWrap seems to support network fileshare-based repositories out-of-the-box, but I think this’ll be a non-starter for my shop based on the constraints of the DMZ outlined above. So I want a web-server based repository. The solution is quite simple (basically, we do both), and is detailed below (duh)
Open ip/ports from my production servers to the internal OpenWrap repository
From here, two options:
1. Setup scheduled tasks on the production servers that periodically checks our internal repo for a new package and updates, accordingly. A polling based approach.
2. Add support in the app, itself, to allow it to spin off a new process to check for updates via o.exe and update itself, as needed. We have a place in our app where this could live, already.
Finally, some kind of completion notification once the new site wrap has been installed.

Really the goal is to drastically reduce the amount of needed human intervention, in order to successfully deploy, to a couple of mouse clicks. Reducing the speed of the deployment would be nice, but is not essential (especially if we do discover that we are severely hamstrung by a bottleneck in throughput across the intranet between the DMZ and backoffice machines where the OpenWrap will live).

Step 1: Make The Web Application A Wrap

So, first things first, we have to set up an OpenWrap “skeleton” package into which we’ll place the source of the application that we’re trying to make into a deployable artifact (in OpenWrap parlance, this is called a “wrap”, although we’re misappropriate the framework a bit in order to have it do our evil bidding). In all places below, I use projectname as a standin for whatever you wanted to name your wrap, so please substitute appropriately.

I created a new “wraps” directory, inside of which I intend to place any packages I create. I think initialized my new project in there.

mkdir wraps
o init-wrap projectname

Since my environment is Windows Server 2003 based (and I’ve had problems dealing with the symbolic links/junctions that OpenWrap uses by default), I make sure to disable symbolic links as detailed here. Open up the projectname.wrapdesc in your newly created package directory and add:

use-symlinks: false

to the config.

After this, go ahead and copy in (or create) the project that you want to encapsulate in the wrap to the src/ directory in your package skeleton. After that, enter:

o init-wrap -all

from the CLI to have OpenWrap hook into your project file (it does this by changing the default compiler target in your .csproj). All of the intracacies of this process are beyond the scope of this blog post, so a good place to start, if you want to learn more, is here.

After this, you should now have your project set up as a wrap (in the default configuration). You should be able to do:

o build-wrap -quiet

from the CLI and get a .wrap file as output. This is actually just a zip, which you can rename/decompress to check out the contents of in order to get some understanding of how it packages your build artifacts by default.

For me, though, this wasn’t the end of the story…

Step 1.5: Shoehorning an ASP.NET MVC Webapp Structure Into OpenWrap

By default, it appears that OpenWrap is setup to work with projects that aren’t particularly finnicky about where the bins and resources are placed (or, perhaps more appropriately, are structured to lump everything together in a single build directory). Things like Class Library projects and CLI apps come to mind, here. This works pretty well for Class Libraries (and perhaps even typical CLI projects), but it doesn’t work (for me), out of the box, for ASP.NET apps (sadly). Of course, it should be noted that I’m not saying that OpenWrap is fundamentally incapable of dealing with these projects, but it seems that it doesn’t package them correctly out of the box (whether the remedy is education on how to properly configure OpenWrap or a feature implementation is beyond me at this time).

My solution, in this case, is to just hand-roll my own .wrap that abides the constraints required to have a running ASP.NET MVC site (all binaries in /bin under the root, expose Views, Content, Scripts, have a Default.aspx and Global.asax in the root, etc). For now, this is easier than bruising my brain trying to figure out how to make OpenWrap do this via configuration.

I’m not sure how much of the issue is solvable by configuration in the .csproj and how much is a current limitation of OpenWrap, but suffice to say that this is something I wanted to talk about so that it can either 1) be brought up as a possible feature/fix and use case for OpenWrap or 2) be solved by way of education. I’m also painfully aware of the fact that I’m taking OpenWrap way outside of it’s intended comfort zone as part of what I’m doing in this blog post.

Obviously, this situation is a big your-mileage-may-vary kind of thing, but to give you an idea of what I did to make an ASP.NET MVC package nicely into a wrap:

    # this was derived from:
    # http://snippets.dzone.com/posts/show/6409
    def get_newest_wrap(dir)
        matching = /.*wrap$/
        files = Dir.entries(dir).collect { |file| file }.sort { |file1,file2| File.mtime(file1) <=> File.mtime(file2) }
        files.reject! { |file| ((File.file?(file) and file =~ matching) ? false : true) }
        files.last
    end
    
    def get_version_from(path)
      f = File.open path
      lines = f.readlines
      f.close
      lines.last.sub(/\.\*/,'')
    end
    
    @siteDir = 'bin-net35'
    @toolsDir = '' # where I have some bin tools I use during builds
    @targetProjectName = 'My.Asp.Net.Project'
    desc "hand-roll a wrap for an ASP.NET MVC site"
    task :drybuildwrap do
      mkpath 'working'  
      version = get_version_from ".\\version"
      ts = Time.now.to_i.to_s # being lazy and using a unix timestamp
                              # as the build number. it works, i guess.
      pkg = "myaspnetproject"
      wrapFile = "#{pkg}-#{version}.#{ts}.wrap"
      Dir.chdir 'working'
      mkpath @siteDir
    
      # files needed in the root of the app, besides bin
      rootFiles = ['Default.aspx', 'Global.asax', 'robots.txt', 'Web.config', 'Content', 'Scripts', 'Views']
      rootFiles.each do |f|
        cp_r "..\\src\\#{@targetProjectName}\\#{f}", @siteDir
      end
      # /bin
      cp_r "..\\src\\#{@targetProjectName}\\bin", @siteDir
    
      # needed OpenWrap files
      cp_r "..\\version", "."
      cp_r "..\\#{pkg}.wrapdesc", "."
      mv @siteDir, 'bin-net35'
      
      # tools dir -- some stuff needed to bootstrap the self-updating
      # functionality .. covered in Step 4.5 below
      mkdir "tools"
      cp_r "..\\..\\..\\Tools\\7z\\7z.dll", "tools"
      cp_r "..\\..\\..\\Tools\\7z\\7z.exe", "tools"
      cp_r "..\\rakefile.rb", "tools"
    
      # Finally, let's create our wrap
      sh "..\\..\\..\\Tools\\7z\\7z.exe a -tzip -r ..\\#{wrapFile} *"
      Dir.chdir '..'
      rmtree 'working'
    end

Pretty wild, I know.

Step 2: Setup An OpenWrap Repository To Publish To

This was a real head-scratcher for me, at first, so I asked about it on the OpenWrap mailing list. I was looking for a full HTTP implementation of an OpenWrap server and, seeing only a stub on github, I went to harass Seb for the full source. The thread is here. His solution was pretty elegant. From the thread (where he’s speaking about what he does for the main OpenWraps repo):

As for wraps.openwrap.org, I use something much simpler. I use a file share repository to write packages from teamcity, and on the read side i simply share the same folder as an http folder, that works out of the box as is, so if you’re just looking at exposing the read side, you can do that now already.

So, it’s pretty simple. With this in mind, I set up two servers:

file://buildserver/openwraps/myaspnetsite – A fileshare based repository server that the build server can publish to after a successful build
http://buildserver – A read-only repository that is exposed via IIS 6 directory listing magic (And do remember to add the .wraplist and .wrap to your MIME-types if using IIS 6, as I am :/ )

You can find some documentation on setting up OpenWrap repositories here. The actual, esoteric details of your hosting needs are something that I can’t give specific guidance on.

For me, above setup gives the flexibility I need for my deployment environment (where I can open a single port on a machine in my backoffice intranet to the DMZ servers), while allowing me to easily publish packages in the backoffice environment (where our build machine lives and the rules are somewhat more lax).

Step 3: Integrate OpenWrap Publishing Into Build Process

This one I will leave as an exercise for the reader. Personally, I’m using Jenkins (née Hudson) for our builds (with most of the action in rake scripts). I’ve written some tasks to hand-roll the .wrap (as detailed in Step 1.5 above) and publish that to my fileshare repository.

Bottom line: You need to start producing and publishing wraps as part of your regular build process. Depending on how pervasive your deployment automation will be (just for prod? why not your continuously updated UAT/Staging server, too?) you could continuously push to one repository, while only “selectively” pushing to another, “production-only” repository when you’re ready to deploy via some manually triggered automation. Which approach is appropriate to your situation will depend, largely, on Step 5 below.

Step 4: Setup On The Application Servers

After doing the neccesary legwork on my DMZ servers (like opening the port to my read-only HTTP server in the backend and setting up the OpenWrap shell), I have to a few things.

First, I need to make OpenWrap aware of the repository from which it can fetch the wrap it needs. This was as simple as:

o add-remote myaspnetsite http://buildserver

As outlined in Step 2 above, this corresponds to the read-only server that I exposed for my DMZ servers to pull from.

Next, I will initialize a stub project that I’m going to use to “host” the application that I want to deploy via OpenWrap. Finding the place where I want it to be (like D:\Inetpub\wwwroot or what-have-you), I did:

o init-all myaspnetsite_prod

OpenWrap will give me some static about setting up the project structure. I can now go into there and set things up.

cd myaspnetsite_prod

As above in Step 1, my particular situation warrants that I use use-symlinks: false in my myaspnetsite_prod.wrapdesc file. With that out of the way, I can do:

o add-wrap myaspnetsite -Anchored

And, if everything’s wired up properly, OpenWrap will download and unpack your package within the myaspnetsite_prod/wraps/myaspnetsite directory. If any of the stuff from previous steps is messed up (like your publishing or hosting for the HTTP repository), you’ll have some problems here.

The -Anchored flag tells OpenWrap that the dependency you’re adding you should be unpacked in a “visible” location (ouside of the _cache dir). Normally this is so that you can check that dependency in, but it serves us here by exposing it in a fixed location (independant of the version). More info on package anchoring is available here. From here, you should be able to point IIS at the myaspnetsite_prod/wraps/myaspnetsite folder and it will work (provided your .csproj has its references, etc, working right and nothing squirrely happens with how you’re reworking the wrap.. this is some brittleness that I’ve brought on myself, here.. I hope in the course of discussing this process, a more elegant solution can be sussed out).

Step 4.5: A Brief Digression On The Topic Of Updating ASP.NET Sites

The whole point of this exercise is to get to a point where, once we have the OpenWrap infrastructure in place, updating to newer versions will be a snap. This is complicated by a major, blocking issue: OpenWrap won’t “re-anchor” updated packages if any of the files in the old package directory are locked. This is the case because OpenWrap updates anchored packages by removing the old directory and replacing with a new one (which is pretty sensible in most cases). This isn’t neccesarily what we want in ASP.NET land, though, we where can just unzip a new install over the old one and, provided there isn’t any weirdness with renamed assemblies, etc, ASP.NET will pick up the file changes on the next request and recycle the app automagically. And, while we can unzip over a running application without complaint from IIS, we cannot nuke the directory of a running without, at the least, taking downt he web site (which may cause false alarms with your application monitoring infrastructure).

I get the impression that there’s a lot of people who are used to stopping the web site in order to do updates. I’ve never been one of those people (unless there’s file locking issues that prevent a clean unzip, in which case you have no choice but to stop the site). With this in mind, I prefer a deployment process that involves unpacking a zip and letting it override an existing, live app. If this is a grossly irresponsible perspective to take, I’d like to hear about that (I mean, as much as anyone likes to hear “you’re an idiot!”).

Anyways, back to the issue at hand: How to work around OpenWrap’s design limitation in terms of updating a “live” site? More rake scripting, of course!

For me, the strategy is as follows:

I decided to modify the rake task outlined above in Step 1.5 to include an additional rakefile (containing the snippet below) and a CLI unzipping utility (7zip, if you must know) in the deployable artifacts for my wrap.
The rakefile that is now embedded in the .wrap contains logic to:
1. Find out what the version # is of the “newest”, locally held wrap for our package that we want to continuously update
2. Check the remote repository and, using o list-write and some output parsing magic, find out what its newest version of our target wrap is
3. If the remote’s newest is not equal to ours (unlikely that we’ll slidebackwards in revisions and maybe even desirable, at times, if so (like a rollback scenario)), then:
4. Run o update-wrap which will, of course, fail to anchor our package (but still download the .wrap for us)
5. Having downloaded a new version, we now find the filename of the newest version of our package (which should be the new one)
6. We copy out our CLI unzip’ing util and then use it to unzip our new package (you have to copy it out, otherwise it’ll explode when trying to overwrite itself)

I ended up with a script something like:

    # This is pretty lame and depends on the impl of o.exe's CLI output.. have to
    # keep an eye peeled for changes, here
    def newest_version_of_package_on_remote(pkg, repo)
      output = `o list-wrap -Query #{pkg} -Remote #{repo}`
      line = output.split("\n").last
      line = line.sub(/^.*available: /,'').sub(/\)$/, '')
      line.split(", ").last
    end

    task :hotupdate do # this is the task we'll run from a scheduled task on the app server
      pkg = "myaspnetproject"
      repo = "myaspnetprojectrepo"
      Dir.chdir "..\\.." # we're executing from somerepo\wraps\myaspnetproject\tools .. 
                         # need to move up the somerepo\wraps\ dir
      localWrap = get_newest_wrap(Dir.pwd).sub(/#{pkg}-/, '').sub(/\.wrap/, '')
      newestWrap = newest_version_of_package_on_remote pkg, repo
      if localWrap == newestWrap
        puts "at latest"
      else
        puts "updating..."
        sh 'o update-wrap'
        wrapFile = get_newest_wrap(Dir.pwd)
        Dir.chdir pkg
        toolsDir = "toolsTmp"
        mkdir toolsDir
        cp_r "tools\\7z.exe", toolsDir # have to copy out 7zip from the tools dir, so it doesn't
        cp_r "tools\\7z.dll", toolsDir # barf when trying to overwrite itself
        sh "#{toolsDir}\\7z.exe x -y -tzip ..\\#{wrapFile}"
        rmtree toolsDir
      end
    end

Step 5: Flip The Switch

If you made it here successfully, that means that you’re worked out the details for how to facilitate OpenWrap’s use as a deployment framework for your project. Now you’ll have to figure out how to put it on auto-pilot. The key will be to get to the point where you won’t have to log into your application servers at all to do a deployment.

My personal preference is set up a scheduled task on the deployment server that polls for updates on a regular, frequent basis. This way, my deployment choke point is now the act of pushing packages to that repository that is regularly polled (something that I can automate pretty easily as part of my existing build infrastructure). You can easily invert this approach by automating the pushing of packages to your repo and making the update poll be the manually triggered step (via added functionality in your app or some existing tool/server to kick off processes/scripts on your production server).

I use the :hotupdate rake task outlined above, executed every five minutes, to check for new versions of our application and handle the downloading and unzipping on an as-needed basis.

Conclusion

This post is a skeletal depiction of a process that I’ve worked through in order use OpenWrap for automating my deployment process (which is currently only automated up-through the point of package generation. Actual deployment on the application server is manual and a hassle). I kept things vague on purpose, as I feel like an overly detailed post would be less valuable (due to be the tendency to get bogged down in semantics and the fact that every environment is different).

There are a lot of details to work out, but I think the steps outlined above serve as a good place to start for getting things up and running. There’s a pretty good chance that I’m doing something Really Wrong, so I’m interested in having a discussion about what could be done to improve this process.

With the caveats aside, there are two key points in the OpenWrap workflow where I, pretty much, opt-out in favor doing things a bit differently to suit my own laziness and preferences:

When building a new wrap, I roll one by hand (Step 1.5) using a rake script, so that I don’t have to bother with wrapping my brain around the configuration (see what I did there?).
When trying to update the wrap (Step 4.5), I go through the motions (in order to have OpenWrap download the newest .wrap for me), but anticipate the re-anchoring of the new version failing (because IIS has locked the directory) and will go ahead and do a manual upgrade-in-place (by unzipping the .wrap over the existing anchor, which will work, most of the time, unlike a delete-and-replace). This may or may not be a bad thing and exposes you to certain Troubles as your application evolves naturally over time (assembly conflicts due to renames, to name just one of many.. not to mention the possibility for .dll or other files not being updated due to IIS’ rather eccentric and unpredictable file-locking practices). This whole thing stems from the desire to avoid taking down the site/IIS when I can help it.

I would love to have a discussion around these issues and see if there’s a possibility to evolve the OpenWrap framework to handle these scenarios/approaches more gracefully, if they’re deemed safe enough, valuable, etc.

I hope this post has been valuable for you in your quest to be a Lazier Developer, because that’s really what automation about, isn’t it?

An Introduction To machine.javascript Part 2

2010-02-18T00:00:00-08:00

This post is a continuation from Part 1, which introduced the ideas of machine-javascript, namely the script loader and a simple example, showing how a HelloWorldController was instantiated and attached to the page. What was not explored, however, was what the HelloWorldController actually looks like and how it works. That is the topic of this post.

Let’s Dive Right In!

Here is the content of HelloWorldController.js, loaded into our page in the example shown in Part 1 of the series:

    include('jquery.js');
    include('machine-controller.js');
    include(function() {
      var global = this;
      
      var viewLeft = '<div><a id="clicker" href="#">Hello world!</a> I have been clicked ';
      var viewRight= ' times.</div>';
      
      global.HelloWorldController = function() {
        this.init(); // kicks off Machine.Controller's internal setup
                     // .. always call this first.
        this.clickCount = 0;
        this.setView(viewLeft + this.clickCount + viewRight);
        this.addAction('click', '#clicker', this.onClickerDivClick);
      };

      global.HelloWorldController.prototype = new Machine.Controller();
      var hw = global.HelloWorldController.prototype;

      hw.onClickerDivClick = function(e) {
        // 'this' is always bound to the controller object
        this.clickCount += 1;
        this.setView(viewLeft + this.clickCount + viewRight);
        this.render(); // refresh the domRoot property
      };
    });

Okay! A lot to take in there. Let’s start with the include()s at the top of the file.

    include('machine-controller.js');
    include(function() {
      ...
    });

As noted in Part 1, calls to include() are never nested within the scope of a single file, but always occur serially. The first call is to load up machine-controller.js, upon which this file is dependant. If HelloWorldController were dependant upon a “ViewRenderer” (which is discussed further down), it would also be included at this point. Finally, the actual body of the script is contained within an include(). This is a convention that must be adherred to in order to leverage the utility of machine-includer.js. Strictly speaking, machine-controller.js takes no dependency upon machine-includer.js and you could easily use it with another script loader or no loader at all, but for the purposes of this example we are going to use it.

      var global = this;
      
      var viewLeft = '<div><a id="clicker" href="#">Hello world!</a> I have been clicked ';
      var viewRight= ' times.</div>';

At this point, we’re just setting up some variables that will be used in the course of defining HelloWorldController. I, by habit, typically assign the top-level this in a given script to global. The next two variables, viewLeft and viewRight are two chunks of text that will make up the “view” rendered by HelloWorldController (that is, the markup that it attaches to the DOM). This approach is not-at-all optimal for general use, but is merely meant to demonstrate the basic functionality of setView(), which will be covered in-depth below.

      global.HelloWorldController = function() {
        this.init(); // kicks off Machine.Controller's internal setup
                     // .. always call this first.
        this.clickCount = 0;
        this.setView(viewLeft + this.clickCount + viewRight);
        this.addAction('click', '#clicker', this.onClickerDivClick);
      };

This chunk of JavaScript is the declaration of HelloWorldController’s constructor/intializer and is where the majority of Machine.Controller-derived controller objects are configured. There’s a lot of important stuff here, so let’s go line by line.

        this.init(); // kicks off Machine.Controller's internal setup
                     // .. always call this first.

The init() function in a Machine.Controller-derived object is where internal initialization is housed. It should be the first thing you call in any controller you define. Obviously, overloading/replacing this would be a Bad Thing unless you really know what you’re doing. Thankfully, at least, much like Controller development on the server, you’re not terribly likely to have deeply nested inheritance hierarchies of Controller classes. But if you do need to do so, you can always take measures to Make It Work.

        this.clickCount = 0;

this.clickCount is a stateful counter that we’re going to use to track clicks to the DOM that this controller “owns”. This is an example of how client-side controllers are stateful (one thing that can be thought of differently from server-side controllers in many instances).

Another thing to note: it can generally be assumed that any call to this in the top-level scope of a function attached to the prototype for a Machine.Controller-derived object will reference the object itself. This is normal behavior typically but, interestingly, this also applies to functions bound to events using the addAction() function, which is shown below. Typically, at least with jQuery event binding, this is bound to some kind of context information for the event in question. Of course, callbacks passed into jQuery functions like $.each() and $.get() will still have their this variable re-bound, as is expected.

        this.setView(viewLeft + this.clickCount + viewRight);

This is the setView() function, mentioned above. It is used to tell a controller what mechanism it will use to get some markup that will represent its presence in the DOM. There are two valid signatures for this function:

        this.setView(aString);
        this.setView(aString, anotherString);

The first signature just takes a single string and is the format used in HelloWorldController. This treats the passed-in string as static markup text and will move to immediately convert it to a DOM and then attach it to the page’s DOM when needed. This is the simplest possible use case for setView() and isn’t often practical, but can be useful.

The second signature takes two strings: The first is an argument to pass in to a ViewRenderer. The second argument is the “key” for that ViewRenderer. In this case, the “key” refers to a string that the renderer uses to globally identify itself when it is registered with Machine.Controllers mechanism for tracking ViewRenderers. The details of how this works won’t be covered in this post, but just know that it is there.

If you’d like to see an example of this use of setView() and ViewRenderers right now, then look at the /example/example.html file in the machine-javascript github repo and check out the ‘ViewRenderers and Views’ example.

Back to the larger issue of the signifigance of setView(), after calling setView() you should know that any calls to the controller’s render() function will cause whatever “instructions” were passed in to setView() to be reevaluated and the results placed in the domRoot property of the object. You can also arbitrarily call setView() at your pleasure to change the “rendering strategy” for a given controller (but will of course want to call render() after that so the changes can be reflected in the domRoot).

The last line in our controller’s initializer function is:

        this.addAction('click', '#clicker', this.onClickerDivClick);

This is the previously mentioned addAction() function. It is a wrapper around jQuery’s event binding mechanism that provides a few advantages:

Events bound in this fashion don’t need to be manually rebound when the controller’s DOM changes (this uses a combination of jQuery Live Events and manual re-binding on DOM change).
As mentioned above, callbacks passed in to addAction() keep their this property bound to the controller object, instead of the context for the event that called it.
It provides a straightforward interface to pool your event bindings in a single location.

The syntax is straightforward: The first argument is the name of the event that should be listened for. This uses the jQuery convention for event names (onClick becomes click, onBlur becomes blur, etc). The second argument is the CSS selector for the element(s) you want a callback bound to. And the third argument is the function callback you want to pass in. For the sake of keeping things clean, I specify my event handlers on the prototype of the controller itself and pass those in. There’s nothing that says you can’t specify the function inline, if you so desire. Do note, though, that this will be bound to the enclosing controller regardless.

One last interesting (and important) detail: event callbacks bound using addAction() will only be triggered when the event occurs on elements in the subset of the DOM owned by the controller.

So if, for example, you have several controllers in a given page that each expose <a> elements that all have the foo class, then a call to addAction('click', 'a.foo', this.someCallback); will only trigger the this.someCallback function for clicks on those links in the view generated by the controller. Nifty, eh?

Looking at the next chunk of JavaScript in our HelloWorldController example, we have:

      global.HelloWorldController.prototype = new Machine.Controller();
      var hw = global.HelloWorldController.prototype;

Here, we see that Machine.Controller uses the usual prototypical inheritance found in JavaScript. It was originally based on John Resig’s “Simple JavaScript Inheritance” model but was subsequently converted to use the more common prototype model. After that, you can see that we assign HelloWorldController’s prototype to a simple, local variable called hw for convenience. This is to merely save keystrokes. It isn’t such a big time-saver for simple controllers like this one but, if your controller had more functions attached to its prototype, this sort of thing becomes more valuable.

Secondarily, it’s also useful to declare the “public” functions and properties for a Controller, while allowing you to create functions that aren’t attached to it and keep those as “private”, if that’s your thing.

And finally, we have:

      hw.onClickerDivClick = function(e) {
        // 'this' is always bound to the controller object
        this.clickCount += 1;
        this.setView(viewLeft + this.clickCount + viewRight);
        this.render(); // refresh the domRoot property
      };

This is the callback that was passed-in to addAction() in our constructor function. It is a typical event-handler callback, with the event information as the sole argument to the function. This may or may not be useful to you and you can omit it if you want to. As is indicated in the comment, this is bound to the enclosing controller. This means that we have access to any stateful information contained therein (clickCount, in this case).

The callback increments the clickCount property by one and then calls setView() with the same viewLeft and viewRight variables used in the constructor, but with the newly incremented clickCount. If we were using a more sophisticated rendering scheme, we would merely modify the controller’s model property and let that “trickle down” to the ViewRenderer. In this case, the call to setView() would go away and the subsequent call to render() will be all that would be needed to update the DOM. But, since we’re using the simplest possible approach to specifying a view in this example, we have to update the view’s markup ourselves via setView().

The finished product

You can see the complete example here! It is a pretty contrived example but, if you have dealt with complex client-side logic before, the advantages are obviously (in my opinion).

Conclusion

In this post, we covered:

The contents of the HelloWorldController.js file that is first mentioned in Part 1 of this series. A typical Machine.Controller-derived object contains:
A constructor function, with the first statement in it being a call to init(). This does the internal setup for Machine.Controller and should always be called first. Also don’t replace it in the prototype unless you know what you’re doing.
In additional to any per-controller setup, a constructor will usually contain:
A call to setView() to designate the view rendering scheme for this controller. You can either pass in static text (which you need to update yourself via calls to setView()) or designate a pre-registered ViewRenderer to handle the details for you. The details of how ViewRenderers work is for another post.
One-or-more calls to addAction() to bind event callbacks to elements in the subset of the DOM owned by the controller.
Each call t addAction() is going to need a corresponding callback. this in the callback will correspond to the enclosing controller, giving you access to state information.
Call render() when you want to update the markup in the domRoot.

Stayed tuned for Part 3, where I’ll go over one of the most useful architectural features of machine-controller.js: ViewRenderers. Until then, take it easy!

An Introduction To machine.javascript

2010-02-17T00:00:00-08:00

This is the first in a series of [hopefully many] blog posts on the topic of machine.javascript, an open source framework for helping developers create testable, deployable and well-factored UIs on the client in JavaScript. It covers some of my exposure to tools in this space and an overview of what machine.javascript consists of and how to use it. This post is tightly coupled to Part 2 in the series and each should not be read without considering the context of the other.

My experience with JavaScript MVC

As I’ve gotten involved in my professional and personal capacities with rich, client-based UIs in te browser, one of the most indispensible patterns I’ve had at my disposal would have to be “MVC”. I put MVC in scare-quotes because it is, in my opinion, one of the most overloaded terms in software development today. The most basic, valid definition for MVC in the context of this post, though, is to say that it is about creating patterns for application development that deliberately seek to segregate control and rendering logic which are optimized for the particular platform (in this case, JavaScript in the browser).

With this in mind, the Eleutian team (notably Daniel Tabuenca) produced a kick-ass library that I always felt really attacked the problem of building complex, client-side applications in JavaScript in an effective, sensical fashion. It could have to do with the fact that this framework was the first I ever used in this particular solution/problem space, but I’ve reviewed/experimented with several other frameworks since and never really liked any of them. Some reasons for this:

Many of them (JavaScript MVC in particular) are very heavy, requiring things like having Rhino installed to do command-line scaffolding (wtf?!)
Others are tightly coupled in the manner in which view/template rendering is handled. Modern JavaScript development in the browser is interesting because there are several different ways to “get there”, when it comes to how you programatically display content. Options include, but aren’t limited to: Direct DOM manipulation via jQuery/other frameworks ( $(‘#container’).append(’<div></div>’); ), JavaScript Templating (EJS, Pure, etc) and/or tools/plugins that take structured data and directly output markup (jQuery.datatables, jqplot, etc). Eleutian’s framework was originally tightly coupled to EJS for template rendering, but this has since changed (as I will explain shortly).
Some frameworks get the abstraction wrong, in my opinion. The scope of what I, the developer, want when doing client-side work is pretty narrow: aggregate events around some subset of the DOM and have a uniform way of manipulating/regenerating the markup in that subset in response to said events. The premise of “smart models” in an isolated browser environment (wiring up AJAX calls to the server) is way too much abstraction for me and, in the laissez-faire world of JavaScript, strikes me as very open to “entering a world of pain” scenarios.

From my time with the Eleutian team, one of the things I missed the most was the MVC framework I had used when I worked there. And while I can by no means take credit for creating or expanding machine.javascript, I can definitely say that I was able to harass Aaron Jensen and Dan into releasing the libraries in question as OSS (although they claim they were going to do it anyways). Hence, the github repo and this blog post.

What is machine-javascript?

At the heart of it, machine-javascript consists of two components, each an individual javascript file:

machine-includer.js – Specifies a “script loader” that works in a “nested” fashion (as opposed to “flat” includers like LABjs). It allows you to specify your dependencies in a manner similar to how you would in server-side code and then load it all (in a cascading fashion) during the page load. Like other script loaders, it is often not practical for performance-critical/production uses, but is great for dev/debug work and provides a critical advantage: The nested, per-file include() statements provide get hints for a recursive script parser to build a list of dependencies which you could use to create a single, bundled for your markup. The include() function that it creates can handle both external script loads (when passed a string) or evaluate functions passed to it. Regardless of which, both are evaluated in order of specification based upon the dependency tree created when looking at how invokations of include() are specified in files and the same external script is only loaded once.
machine-controller.js – Contains a prototype for the Machine.Controller object, which can be inherited in your “controllers” and provides a straightforward framework for specifying events and rendering logic on a piece of the DOM that the controller will be associated with.

Leveraging these two components, I will demonstrate how to create a simple controller with machine-javascript.

A Simple Example

Let’s consider the simplest possible hello world case, utilizing machine-javascript.

    <head>
      <title>Hello, world!</title>
      <script type="text/javascript" src="machine-includer.js"></script>
    </head>

In our page’s <head> tag section, we have a single script include. All we are going to load in this page in an explicit, static fashion is the machine-includer.js script.

Somewhere down in the <body>, we have a chunk of markup that looks something like:

    <script type="text/javascript">

      // this is the initial setup for machine.includer
      Machine.Includer.configure({scriptLocations : {'.*':''}});
      
      // include our external dependencies
      include('jquery.js');
      include('HelloWorldController.js');
      
      // and kick it all off once the page loads
      include(function() {
        $(function() {
          var hw = new HelloWorldController();
          hw.render();
          $('#container').append(hw.domRoot);
        });
      });

      include.load();

    </script>
    <div id="container"></div>

So this is interesting; We do some inital configuration for machine-includer.js, then load our external scripts that the page is dependant upon and then instantiate an object (defined in an external dependency) and attach one of its properties to the DOM via jquery’s append() function.

Let’s take each signifigant chunk on its own…

      // this is the initial setup for machine.includer
      Machine.Includer.configure({scriptLocations : {'.*':''}});

This is the only initialization call needed to set up machine.includer.js; after this, you can safely call include(). The hash that is passed into configure() has one notable parameter: scriptLocations. It is a series of pairs that say to the includer, “Hey, if you encounter the regex pattern in the left component in an includer URL, please prepend it with the contents of the right component.” This means that, if you had some hinting from the server based on environment (dev, production, etc), you could configure machine-includer.js to mangle the script loads done via include() so they actually called out to an external CDN or local folder, depending on the runtime environment.

For example, consider if your static content was delivered by a CDN like Cloudfront in your production setting, but was served from a local Scripts directory in the same web server when run in the dev/test environments.

In an ASP.NET MVC WebForms template with a strongly-typed view that had an Environment property hooked up to some mythical enum or the like (this applies just as easily to other server frameworks) the server-side template might look like:

      // this is the initial setup for machine.includer
      <% 
        var prefix = Model.Env == Env.Production ? 'http://s3.yourdomain.com/path/'
          : '/Scripts/';
      %>
      Machine.Includer.configure({scriptLocations : {'^static.*': '<%= prefix %>'}});

In this way, you can drive how your scripts are loaded at runtime with a simple, per-page check to the ambient environment.

      // include our external dependencies
      include('jquery.js');
      include('HelloWorldController.js');

In this chunk of code, we are declaring our dependencies for this page, utilizing calls to the include() function with strings for the paths to the files in question. As with <script> tags, you can pass in absolute or relative paths. But, as noted above with the scriptLocations configuration, you can also establish conventions (driven by server-side configuration) for mangling the path passed to include() to account for special cases.

      // and kick it all off once the page loads
      include(function() {
        $(function() {
          include.load(); // actually loads the scripts

          var hw = new HelloWorldController();
          hw.render();
          $('#container').append(hw.domRoot);
        });
      });

In this block we actually have some running javascript code, passed within a function to an invokation of include(). One thing worth pointing out, here, is that (when taken into context with the previous block where we used include() to load external scripts) calls to include() are never nested within the scope of a single file, they always appear serially. Now, when passing a string URL to include(), there will of course be calls to include() in the file indicated by the URL, but there are never calls to include() within the scope of a function passed to an existing call. Please remember this when writing your code or it’ll fail and you’ll be left scratching your head (speaking from experience, here).

Back to the function itself, you can see that it consists of a callback for jQuery’s “onReady” event binding using the $(function() { }); convention. When the document is ready (the DOM is completely loaded, mainly), we create an instance of a HelloWorldController, which we can assume was imported in a previous call to include() (the code for HelloWorldController will be shown shortly). There is some setup that happens in the course of HelloWorldController’s initialization, which allows us to then invoke its render() function. This is a function defined in the Machine.Controller prototype that leverages some configuration done during initialization and renders the controller’s desired markup in its domRoot property. Subsequent calls to render() will update the markup stored there.

Finally we, using jQuery’s append() function and attach our controller’s newly available domRoot to an element with an id of container (shown in the complete example above).

One of the fringe benefits of Machine.Controller is that subsequent calls to render() (outside of or within HelloWorldController) will automatically update the domRoot and be reflected in the DOM without having call to append() on your container element again. This also applies to events bound from within HelloWorldController using Machine.Controller’s syntax for doing so.

Going back to the complete example above, let’s examine, once again, the external scripts loaded as dependencies for the page.

      // include our external dependencies
      include('jquery.js');
      include('HelloWorldController.js');

You can guess pretty easily what’s within jquery.js, and since it was needed in order to append our instance of HelloWorldController to the container <div> we loaded it up via include().

But what about HelloWorldController.js? This contains the definition for our controller, the contents of which will be shown in Part 2 of this series.

Finally, we have:

      include.load();

This actually triggers the script load and should come after all of the previous calls to include().

Conclusion

In this post, we covered the following topics:

A brief overview of my anecdotal introduction to MVC in JavaScript and my personal opinions and nags about it.
A rough description of the two pieces that make up machine-javascript: machine-includer.js and machine-controller.js.
A “hello world” example was shown of the HTML markup for an example page that uses the includer and controller. The contents of the actual controller script have not yet been shown (and are saved for part 2 in this series).
Some exposure to the semantics of using machine-includer.js.
Shows how a finished controller is instantiated and attached to the DOM.

I hope this post has been enlightening for you, the reader. Parts 1 and 2 really belong together (as each is uselss without the other), but if combined it’d be a classic tl;dr article. I thought that was a good place to split them up. In any case, please continue on to Part 2 for the rest of the story!

I am ALT.NET Pursefight!

2010-01-15T00:00:00-08:00

This post is only of use to those who were monitoring/involved in the ALT.NET “movement” circa 2007 and 2008. It was an age of high adventure where a desire, predicated on the assumption that the status quo in terms of Microsoft-driven guidance on Patterns & Practices and tooling was not leading to the creation of Good Software in the short and long terms, had given rise to a set of “alternative” orthadoxies meant to improve the lot of developers.

If this situation doesn’t mean anything to you, or if you arrived “after the party”, then this post has little value.

Recently, I decided to write the following email to one Scott Bellware:

I’ve been wanting to write you to get something off of my chest: I am the sole person behind the ALT.NET Pursefight blog (since discontinued), twitter account etc. It’s something that’s been sticking in my craw for some time and I’m only now getting around to rectifying it. My apologies for my laziness on this. We’ve only met once (at the “Night of BDD” event that you attended in Seattle where you sat there and got verbally abused by Jim Newkirk and Charlie Poole for two hours). We don’t have any kind of relationship, professional or otherwise, but it’s something that I’ve wanted to tell you, nonetheless.

I’ve grown to respect you over time (during and after my period operating under the ALT.NET Pursefight monicker). Not just for your community work, but for striking out as an independent and pretty much putting your money where your mouth is. I figure that I would, in fact, have a lot to learn from you (by virtue of your longevity in the industry alone) and were I ever to actually contact you to try and dig out some of that knowledge for my own benefit or the benefit of the community at-large, I’d be misrepresenting myself if I didn’t make this disclosure. As far as my respect goes, all I can say is: you were pretty much proven right on every count with regards to your insights as to the trajectory of the “ALT.NET movement”, as well as on numerous technical issues: EF is indeed a travesty, amongst other things. I also agree that education is the way forward to “raise the water line” for the community as a whole in terms of bringing the average level of competency up for the aggregate of developers but, having experience with the “minister to the heathens” approach to outreach/education, I still think the process needs some work (sadly, most of the issues I observed came from trying to motivate the “knowledgable” people to give up their time and put aside profit/renown motives to engage in this public service). I am also, perhaps unlike you, quite dubious about the whole Software Craftsmanship thing, even though its proponents seemed to have ( perhaps from observing ALT.NET’s fate) hedged their bets a bit and reduced it to a series of bland “we care more” truisms.

As for why I did the whole ALT.NET Pursefight thing in the first place, I thought it was needed at the time. And perhaps there was some capacity where it could have been useful or performed some function of value for the community; If that was the case, I feel like I utterly failed in it and, as such, just gave up on the whole thing. I apologize for the ad hominem attacks against you. If you still want to kick my ass, I guess you now know with whom you can make arrangements to try and do so next time you’re in Seattle. In all honesty, you’re the only person that I’ve disclosed this information to (besides one personal friend who knew from the start), as I feel like you’re the only person who bore any real amount of non-substantive criticism from me. What you do with it (wreck my professional persona, firebomb my house, etc) is up to you, I guess. That being said, I don’t really feel like I have any real “need” to have a public coming out where I can announce to the world that I was ALT.NET Pursefight; it wouldn’t achieve anything. I don’t need any boost from the pseudonym to properly heckle and ridicule any of the current crop of self-aggrandizing idiots and their enablers in the community. And I doubt that I really fall into line, in retrospective, with any of the opinions expressed (I don’t think I ever really took any sides, just threw tomatoes at others from safe cover) as ALT.NET Pursefight and would have to defend were I do to claim the identity. Chalk it up to youth and stupidity.

Anyways, I just wanted to share this with you so that, if I ever do meet you, I can at least buy you a beer and not feel like a douchebag while doing so. I can’t expect you to want to “forgive” (is that even the right word?) me or what-have-you, but at least you’ll know that the person who said stupid things eventually came clean.

To which he replied.

The value in making this disclosure public would be in giving the community one more example of public integrity. Every exemplary action that goes unseen doesn’t contribute as much as it can. Although I appreciate starting my week with your example of courage.

Why did I take up the ALT.NET Pursefight! name? I can’t say for sure, but I think it boils down to something like: I lacked the technical chops, at the time, to enter into and interact with the community as peer, so I assumed an anonymous personality and begin a meta-commentary on the “community” and its personalities. In a way, I was catering to a specific sub-group that wanted to heckle and redicule the “prevailing personalities” of the ALT.NET community (who, warts aside, were providing value for said community). But, regardless of whether or not I was just “giving them what they wanted”, I have to take personality responsibility for my actions.

You can gather from the email that I was reluctant to make the public disclosure that I am making now. But Scott makes a good point on the value of doing so, the merits of which I can add nothing to.

I don’t feel like I owe the “community” as an abstract any apology; It has soldiered onwards, marching blandly towards its own irrelevancy without any help/hinderance from me personally. Some good things have come out of it (although it’s perhaps fallacious and premature to assume the correlation of Microsoft’s changing stance towards OSS in the dev community was caused by ALT.NET the movement– Although I’m sure many of the prominent OSS people that they took “out of circulation” has helepd in changing minds internally). That being said, I do owe apologies to many individuals within the community, about whom I said stupid and unjustified things. To them: I am sorry.

The future of “progressive” techniques in the .NET developer community still hangs in the balance, as far as I’m concerned. Microsoft continues to crank out vendor lock-in focused patterns and APIs, especially if you take into context the possibility of the “Silverlight for every platform” mutterings and the whole EF/One-data-binding-to-rule-them-all talk coming from some bloggers. I personally feel like their open source overtures are a short/mid-term strategy to win ethical boons while still slipping in their typically sub-standard solutions and betting that the mediocre developers and managers will pick the easy wrong, rather than the hard right. To me, that’s what their Open Source support is: A hedge. Whether I’m correct in that assessment or not is something that history will have decide.

We have to recognize this for what it is and address it as such in the sunlight of public discourse and intellectual honesty; Anonymous sniping and heckling isn’t the solution. This is where I was wrong in the past and this blog post is a step towards correcting that mistake.

Saber - a Newspeak web framework

2009-11-06T00:00:00-08:00

I’m very pleased to announce the creation of yet another web framework, although it is the first (that I know of) for the Newspeak platform. It started out as an intellectual exercise to explore Newspeak and some of it’s reflective capabilities and idiosyncracies (specifically the pervasive class nesting). It was a very interesting and enjoyable development experience, to be sure (although it wasn’t without its own gotcha moments and difficulties). Prior to this project, I played with Newspeak a bit at the time of the first release in late February. Besides that, my only experience with this style of development is some recreational programming with Squeak (I worked through about two-thirds of the laser-game tutorial). Newspeak is a very young, yet capable and robust platform and I am more than impressed with what I’ve encountered so far, although a lot of its much-heralded functionality has yet to be implemented. All in good time, I suppose.

Saber and web frameworks in a historical context

Saber is a web application framework that takes a somewhat different approach to web application development and modelling from other, more popular frameworks that you may have heard of. In a way, it may be a sort of counter-evolution in terms of conceptualization of how web frameworks have been structured since Rails came on the scene.

In the Bad Old Days, HTTP was used exclusively as a static content presentation protocol, where URL requests typically mapped to documents in a folder under the wwwroot of some site. When server-side technology solutions (ASP, Java Servlets, CGI, etc) became more common, it ended up being some script/code handling requests, which may or may not have been routed to a script corresponding to the URL somehow. But, with the rise of Rails and the slew of other HTTP-based “MVC” frameworks that have followed in its wake, we now have a popular paradigm of Controllers and Actions and the basic request-handling conventions that they bring to mind. When the need for more taxonomy beyond Controller/Action became perceived as necessary, we got things like Areas or custom routing schemes to match a request for some URL to a given action, perhaps with some parameter mapping to give the whole thing a more RESTful feel.

The pitfalls of the Controller/Action modelling pattern for web frameworks is something that isn’t within the scope of the topic of this post, but it suffices to say that it has become enough of a nuisance after a while for any non-trivial example application that several people have come up with alternative approaches. Foremost of these in my mind is Sinatra, an excellent framework that models endpoints as “routes” (that is, some HTTP Method/URL pair that can be matched by a request) and a block of code to handle that request when the route is matched. I have experimented a bit with porting this approach to the ASP.NET world (the results of which can be found here). This approach is present in several frameworks as a ‘first-class’ alternative to the Controller/Action structure so popular elsewhere (from the direct url->handler WSGI mapping in Google AppEngine’s webapp framework to OpenRasta).

Saber subscribes to the above request/routing-centric viewpoint, but with a twist. From the Saber README:

Essentially: Instead of artificially aggregating endpoints into a Controller/Action hierarchy, Saber lets the structure of the routing handlers becoming the organizing taxonomy for the application.

To further entertain this MVC straw-man, most stereotypical MVC web frameworks reveal the intent and function of their endpoint handlers by aggregating them as actions within controllers. In some static frameworks (in Java or .NET, mostly), the aggregation is also an administrative concern for convenience when using an Inversion-of-Control scheme, leading to kludgey groupings of functionality that don’t scale well, in my experience.

So, how does Saber acheive this taxonomy? By using a fundamental feature of Newspeak: pervasive class nesting. In Newspeak, there is no global namespace. Practically, this means that every “top-level” class itself acts as a module. Within this top-level class, you can nest other classes. It seemed natural, to me, to use this approach to model a web site’s request handlers, with a class called foo handling requests to /foo, and a class nested within that one called bar handling requests to /foo/bar, etc. Each of these “Handler” classes implements one or more methods that each correspond to an HTTP method (onGet: for GET, onPost: for POST, etc). These Handler methods are themselves the actual endpoints for a given request.

Saber also includes a “toy” view rendering system, creatively named SaberView.

So what does this look like?

At the highest level, you have a “site” class that acts as a model/container for your site’s definition. This site contains a few administrative things (like the name of the site for the httpd, the port, etc). Besides that, it must include a nested class called SiteRoot that inherits from SaberHandler (which is the basic route Handler class in Saber). That class itself will match on requests to the root of a site. From there, you can nest classes that match to “nodes” in a route request. For example:

+ SiteRoot
  - foo
  - baz
    - bar

This would be an example of a taxonomy of classes that would have handlers that could potentially match requests for /, /foo, /baz or /baz/bar. Once again, a handler will only be called if it implements a method corresponding to the HTTP method of a request. So if /baz/bar only implements onPost:, then a GET request to /baz/bar will return a 404.

Specialized handlers

But vanilla route handling isn’t all that Saber does. It handles a number of “must have” features for routing-centric web frameworks, including: a NotFound handler scheme, ‘parameter’ handlers and static file serving.

Not found handlers

By default, Saber will return a generic 404 response if a request’s url cannot be matched to a handler. Optionally, you can specify a special handler that inherits from SaberNotFoundHandler that will be processed when a request cannot be matched to an existing Handler. Otherwise, it works just like any other handler with the onGet:, onPost:, etc.

Parameter handlers

Saber allows specifying a Handler nested within another one, inheriting from SaberParameterHandler to be used as a special-case parameter matching handler. That is, given the following taxonomy:

+ SiteRoot
  - foo
  - baz
    - bar
    - nameParam

In this case, the nameParam class is a class that inherits from SaberParameterHandler. What this means is that, at request time, if the request url doesn’t match to any of the other ‘sibling’ handlers at the same level as the parameter handler, then the parameter handler itself will be matched. This way, it could be used as a catch-all and won’t interfere with handlers at the same level. This means that requests for /baz/bar would match to the bar class, but requests for /baz/john or somesuch would match to the nameParam handler, since there is no other handler that would match the request, otherwise. Also, the value of the request (in the case of the previous example ‘john’) will be stored within the requests query fields with the name of the handler as the key.

Naturally, this means that only a single SaberParameterHandler-inheriting class can be nested within a group of sibling handlers. Besides the above issues, the handler behaves exactly like a normal handler, including allowing other handlers to be nested within it.

Serving static content

And what web framework would be complete without a means with which to serve up CSS, JavaScript, etc? Saber allows for classes that inherit from SaberStaticFileHandler that are nested with the SiteRoot to be mapped to physical locations on the filesystem (exposed by overriding the documentRoot method on the class) and mapped to the class’ name. So a class inheriting from SaberStaticFileHandler named static and pointing to a location on the filesystem would be matched. A request for /static/style.css would be matched to whatever path was specified in the documentRoot with style.css appended to the end.

What’s not-so-great about Saber, right now?

First and foremost: I’m not terribly happy with the rather “minimalistic” view engine. It has two very useful features (value substition and block inheritance), but that’s it. I, personally, am a very big fan of pushing as much work onto the client javascript as possible, using server-side views to deliver JSON and that’s it, if possible. But there are always some situations where it makes sense to just do the rendering on the server (like some non-interactive reporting that won’t feature any AJAX callbacks after the initial request). With this in mind, a slighlty more robust solution with conditionals, looping and more sophisticated value substitution (that allows passing in actual objects instead of just strings) would be nice, longterm.

Also, Saber is currently tightly coupled to the underlying platform for the Newspeak environment: Squeak. This isn’t a bad thing necessarily, but eventually Newspeak will move beyond this and Saber currently uses HttpService, which is used in Seaside (not to mention several collection primitives, HttpRequest/Response, etc). I’m glad that the Squeak libraries are there, as they allowed me to cut some corners and concentrate on features instead of low-level plumbing, but these things will have to be replaced longterm. Whether or not Saber survives long enough to see that day is another question.

Another nagging issue is that Saber doesn’t support multiple sites in the same instance using the HTTP 1.1 host header. This is supported in the underlying httpd from Squeak, but isn’t something that I’ve gotten around to implementing.

Overall, I tried to keep the scope of the project somewhat narrow (I’ve only been working on it for a few weeks and I’m quite pleased with the progress I’ve made). I mostly wanted to get the project out “into the wild” in order to get some feedback on the viability of this approach. I wouldn’t really recommend the nested class approach other platforms like Java or .NET, even though they support this feature. Alot of what makes nested classes appealing in Newspeak is that it’s a first-class concern and is built into the IDE, so there’s some tooling to reduce the friction. I wouldn’t say that this same tooling/mindset exists for other platforms.

How to see Saber in action

The README.md in the Saber repository has some pointers on how to get started, should you be so inclined.

Conclusion

I’d like to give a big thanks to the motivating people behind this project: The Newspeak team, for conceiving of this awesome vision and delivering a working prototype for people like me to hack around with. I wish this project the best and hope to find a way to get involved with helping move things forward, longterm. Also, all of the people who aren’t satisfied the state of things in terms of software abstractions and patterns in general. Whether you’re a pathological tinkerer like me or a more pragmatically-minded person, it’s the unreasonable nerd who isn’t satisfied with the reigning paradigm that actually moves the ball forward.

Please note that I don’t actually consider Saber to be a part of the canon of earth-shattering software innovations or theory; I just like to speak in broad, sweeping terms.

Thinking about a "Step 2" for Context/Specification-style testing

2009-10-11T00:00:00-07:00

This is a post about the evolution of my thinking and practice regarding Context/Specification-style test composition. I’d like to review my understanding and approach to it over time and some of the frictions that I encountered after the initial re-alignment of thinking that occurs when changing gears to this style of TDD, as opposed to more orthodox varieties. There may be nothing new here for some practitioners of TDD (if that’s you: sorry for wasting your time); but for others who have made/are making the same mistakes as I or have some kind of “itch” in the back of their mind about how their tests just don’t quite “work” with how they’re currently writing them, perhaps this will help.

I by no means claim to be an expert of any sort on BDD or test-first development in general. That being said, I consider myself a pretty active practitioner of these techniques (I fall into the ”Context/Specification aka “Bellware-Driven-Development” camp).

Some time back, I received an email, asking me a question about some missing code in a pastebin I linked to in a mailing list discussion (here, if you’re interested). Sadly, the code is lost to the ages, but going back and reading that thread got me thinking about what the discussion was about and what I had learned since then.

The thread itself was about the value of DRY when composing tests in a Context/Specification style, specifically how adhering to DRY might lead one to create deep, nested class hierarchies of Contexts which serve the purpose of making test composition more convenient for the programmer, but having the consequence of obfuscating the intention of said Contexts and Specifications. Ayende has written a bit on this topic recently here. Specifically:

One of the main problems with unit testing is that we are torn between competing forces. One is the usual drive for abstraction and eradication of duplication, the second is clarity of the test itself.

An example domain

Before we look at some code to illustrate this, let’s talk a bit about the domain used for this exercise. For now, we’ll rely on a very generic, very shallow Bank application. Briefly, it consists of:

A Customer, who has a collection of Accounts.
The Accounts are broken down into two types:
- CheckingAccounts
- SavingsAccounts

Additionally, here are some business rules:

When a new Customer is created, they must have at least one Account.
Don’t allow debits from Accounts that send the balance into negative territory.

Granted, in the real world, business rules aren’t this black and white. But they’ll do.

Based on this, we have enough to get started with some specifications. For the purposes of this post, I’ll be using C# tests written with MSpec. If you’re not familiar with MSpec, I can’t help you.

Reasons to use inheritance when composing tests

Broadly speaking, there are two reasons to use parent/child relationships when composing tests:

You have “administrative” concerns, such as configuring dependencies for some service under test (maybe setting up a DB session for an Integration test or generating mocks/stubs for something tested in isolation).
You want to aggregate context, such as “any context that inherits from this class is testing an entity/sccenario that is configured/modeled in some specific fashion”.

At the end of the day, though, both of these concerns are about code reuse and not having to repeat yourself with cut’n’pastes of mundane, boiler plate code.

Some code to illustrate the first item:

    public class CustomerRepositoryThatIsStubbed
    {
      Establish context = () =>
        customerRepository = Mocks.GenerateStub<ICustomerRepository>();

      protected static ICustomerRepository customerRepository;
    }

In the above example, the class would serve up a stubbed out ICustomerRepository that we could use for other services that may depend on it. This class could be used in any number of ‘child’ Contexts that actually provide Specifications in order to get some predefined functionality.

You can more generally use these kinds of classes to aggregate a group of Specs, perhaps by domain concern (e.g. CustomerWithdrawalSpecs, CustomerRepositoryQueryingSpecs, etc).

Which leads into the second point noted above:

    public class new_customer_with_a_checking_account_with_a_hundred_dollar_balance
    {
      Establish context = () =>
      {
        var account = new CheckingAccount(100.0m);
        customerWithHundredDollarsInCheckingAccount = Customer.NewWithAccounts(new [] {account});
      };

      protected static Customer customerWithHundredDollarsInCheckingAccount;
    }
   
    public class when_making_a_withdrawl_from_an_account_that_does_not_exceed_the_accounts_balance :
      new_customer_with_a_checking_account_with_a_hundred_dollar_balance
    {
      Establish context = () =>
        debitThatDoesNotExceedCustomersBalance = new Debit(-60.0m);

      Because of = () =>
        customerWithHundredDollarsInCheckingAccount
          .Accounts.Where(x=>x.IsChecking).Single() // this could go away
          .ProcessTransaction(debitThatDoesNotExceedCustomersBalance);
   
      It should_withdraw_the_amount_from_the_customers_account = () =>
        customerWithHundredDollarsInCheckingAccount
          .Accounts.Where(x=>x.IsChecking).Single()
          .Balance.ShouldEqual(40.0m);
   
      static Debit debitThatDoesNotExceedCustomersBalance;
    }
   
    public class when_making_a_withdrawl_from_an_account_that_does_not_exceed_the_accounts_balance :
      new_customer_with_a_checking_account_with_a_hundred_dollar_balance
    {
      Establish context = () =>
        debitThatDoesNotExceedCustomersBalance = new Debit(-60.0m);

      Because of = () =>
        exception = Catch.Exception(() =>
          customerWithHundredDollarsInCheckingAccount
            .Accounts.Where(x=>x.IsChecking).Single() // this could go away
            .ProcessTransaction(debitThatExceedsCustomersBalance);
     
      It should_cause_an_error = () =>
        exception.ShouldNotBeNull();

      It should_not_withdraw_the_amount_from_the_customers_account = () =>
        customerWithHundredDollarsInCheckingAccount
          .Accounts.Where(x=>x.IsChecking).Single()
          .Balance.ShouldEqual(100.0m);
   
      static Debit debitThatExceedsCustomersBalance;
      static Exception exception
    }

These three classes all deal with building out actual Context for use in defining our Specifications. The first class sets up a general Use Case or ‘shape’ for the behavior we want to verify and the latter classes make use of it for their own ends. That being said, there is some friction.

Keep the good and lose the bad

At first glance, there doesn’t seem to be too much wrong with that approach, from an intention-revealing perspective (which ought to be foremost in your mind when writing these tests). The base class for the latter tests is right there, if you want to see what it does and its name gives a pretty good indicator of what it does, already.

Soon or later, though, you may run up against a wall where this approach doesn’t scale. What if you deeply nest your Contexts?

    public class new_customer_with_a_checking_account_with_a_hundred_dollar_balance
    {
      Establish context = () =>
      {
        var account = new CheckingAccount(100.0m);
        customerWithHundredDollarsInCheckingAccount = Customer.NewWithAccounts(new [] {account});
      };

      protected static Customer customerWithHundredDollarsInCheckingAccount = Customer.NewWithAccounts(new [] {account});
      };

      protected static Customer customerWithHundredDollarsInCheckingAccount;
    }
   
    public class and_another_customer_with_a_zero_dollar_balance :
      new_customer_with_a_checking_account_with_a_hundred_dollar_balance
    {
      Establish context = () =>
      {
        var account = new CheckingAccount(0.0m);
        customerWithZeroBalance = Customer.NewWithAccounts(new [] {account});
      };

      protected static Customer customerWithZeroBalance;
    }
   
    public class when_transfering_funds_between_two_accounts :
      and_another_customer_with_a_zero_dollar_balance
    {
      Because of = () =>
        customerWithZeroBalance.TransferFundsFrom(
          customerWithHundredDollarsInCheckingAccount.Accounts.Where(x=>x.IsChecking).Single()
          , 60.0m);
     
      It should_add_the_funds_to_the_destination_account = () =>
        customerWithZeroBalance.Accounts.Where(x=>x.IsChecking).Single().Balance.ShouldEqual(60.0m);
   
      It should_remove_the_funds_from_the_source_account = () =>
        customerWithHundredDollarsInCheckingAccount.Accounts.Where(x=>x.IsChecking).Single()
          .Balance.ShouldEqual(40.0m);
    }

In this case, the third class (the one that actually has Specifications) only shows its relationship to its immediate parent. If these classes were split up, for whatever reason, amongst different source code files then it would become problematic to be able to easilly demonstrate what the intention was. For me, this approach was appealing because it seemed clever, at the time. But, after going down the path a ways, it didn’t work out.

Additionally, having several base classes that set up specific edge cases for how scenarios are represented, data-wise, is kind of putting the cart before the horse. In Context/Specification testing, you want the Contexts themselves to be just as important, if not moreso, than the Specs. It should plainly and clearly spell out “Because stuff is configured in such a fashion and we did something just so..”, we get a result that we expect in our Specifications.

    public class when_transfering_funds_between_two_accounts
    {
      
      Establish context = () =>
      {
        var zeroBalanceAccount = new CheckingAccount(0.0m);
        customerWithZeroBalance = Customer.NewWithAccount(zeroBalanceAccount);
        var hundredDollarsAccount = new CheckingAccount(100.0m);
        customerWithHundredDollarsInCheckingAccount = Customer.NewWithAccount(hundredDollarsAccount);
      };
    
      Because of = () =>
        customerWithZeroBalance.TransferFundsFrom(
          customerWithHundredDollarsInCheckingAccount.Accounts.Where(x=>x.IsChecking).Single()
          , 60.0m);
     
      It should_add_the_funds_to_the_destination_account = () =>
        customerWithZeroBalance.Accounts.Where(x=>x.IsChecking).Single().Balance.ShouldEqual(60.0m);
   
      It should_remove_the_funds_from_the_source_account = () =>
        customerWithHundredDollarsInCheckingAccount.Accounts.Where(x=>x.IsChecking).Single()
          .Balance.ShouldEqual(40.0m);
    }

Here, we’ve just taken the stuff that went in the superclass Contexts and put them into the class that has the Specifications, which pretty much puts you back at square one. The reason things like using inheritance to contain Context is appealing is because, in the above example, the test composer has to repetitively do things like create objects for the System/Behavior Under Test, wasting precious keystrokes.

So how to do we mitigate this annoyance, thus acheiving the following:

Foremost, the Contexts reveal the intention and provide meaningful information on what behavior, exactly, is being verified.
Reduce repetition as much as possible while still satisfying the previous point.

Sharing information about lessons learned at this “stage” of competency for Context/Specification testing is where the community is, right now. Aaron Jensen blogged recently on the topic here (disclosure: he used to be my boss). Much of the post is the body of an email that he wrote replying to someone’s question about Context/Specification testing issues. Speaking about base classes that aggregate context, he said:

”…I don’t like it now. I much more prefer to just have a base class that contains any utility/meaningless cruft my specs have. You also seem to have taken this to a bit of an extreme. Would you make a regular base class just to create a single instance variable and set it to null in the constructor? Probably not. Same stuff applies here. There’s no value in creating base classes unless they provide value. There’s no naming or understanding benefit. As a matter of fact, they hinder understanding more than anything.”

The bit about pushing utility methods into base classes is a pretty valid approach to reducing keystrokes in tests while retaining an intention-revealing value to them.

Consider the above example where we do all the setup in a single Context, but with the “uglyness” of the repatition. What if it were reworked to look more like:

    public class CustomerAccountTransferSpecs
    {
      public static Customer CustomerWithStartingBalanceOf(decimal startingBalance)
      {
        var account = new CheckingAccount(startingBalance);
        return Customer.NewWithAccount(account);
      }
    }
    
    public class when_transfering_funds_between_two_accounts : CustomerAccountTransferSpecs
    {
      Establish context = () =>
      {
        customerWithZeroBalance = CustomerWithStartingBalanceOf(0.0m);
        customerWithHundredDollarsInCheckingAccount = CustomerWithStartingBalanceOf(100.0m);
      };
      ....
    }

Aside from some (possible) nitpicks regarding what kind of account a customer gets when using the helper method, this approach provides a net benefit in terms of revealing intent and keystroke reduction. From here, the direction to push is towards stream-lining this process. Once again, Aaron Jensen has done some blogging on the topic. Also, Greg Young chimes in with a similar approach for an albeit different set of circumstances.

Conclusion

A few things to remember:

Using inheritance to DRY your way out of having to re-specify context is the wrong approach. “Context explosion” is something that you have to learn to deal with, but this approach will only serve to obfuscate your context, longterm. If you want to trim your keystrokes, instead consider the following:
Having to re-specify context over and over is a code smell. So think about the best, most intention-revealing way to present that context-building for you and push it into some helper methods/fluent fixtures/etc instead of using rigid class hierarchies to represent that information.

Good luck and happy testing!

Hello world GitHub blog post

2009-08-17T00:00:00-07:00

Wow, five days short of a whole year since my last post on my blogger.com blog. I’ve kind of had the idea in my head for some time to move my blog to GitHub. I may lose some bells and whistles, but I gain an easier way (in my opinion) to create and publish posts (and plus, markdown > *).

Anywho, hopefully I’ll post more, since I don’t have any shortage of ideas kicking around that I’d like to share.

And now, some code to see how highlighting looks:

    import System
    import System.Collections.Generic
    
    public class Foo(Bar):
      public def constructor():
        pass
      
      public def Baz(input as string) as string:
        return "Foobie "+input

Here’s to many more!