Notes to self

Deployment woes.

I code in Java at work, and all of my projects can be divided into two camps:

1. Those that build to .jar files and deploy to our maven2 repo.
   
2. And those that build to .war files and are deployed to some application server.
   

Once configured, maven itself is pretty good at making case #1 run smoothly - "mvn deploy" is all it really takes. But do I have anything similar for case #2?

Well, not quite. I tried banging something together with Capistrano, but that didn't work. For one thing, Capistrano expects you to be deploying a Rails application and these are typically all deployed in the same way, which is very different from deploying a .war file. Secondly, I had a build/compile step in my process that I needed to do locally (because compiling on the server is troublesome and a bad idea), this meant that I had to use the put function to upload my .war file. Put worked nicely for small text files (ie. my project.properties file), but failed miserably when the payload war a near 20 MB heavy .war file.
Googling and tweaking the flags on the File object didn't work. It simply refused to upload that damn .war file.

Itch-scratchers mentality.

I suppose I could have written a shell script and leveraged the existing scp and ssh tool chain. And I did throw a pebble down this route to see how it felt, but eventually rejected the idea. The reason is that I'm deploying to multiple hosts simultaneously. I tried looking at clusterssh to see if it would help me in this regard, but it turned out to be odd, klonky and basicly not designed for such a task. I also took a look at Dancer's Shell, dsh, which was a lot closer to home, but my username contains a backslash and dsh meticulously stripped it out prior to logging in, foiling any and all of my attempts at tricking it into taking it at face value.

What was I suppose to do? I had pretty much given up on automating the process and started to accept defeat, when I happened upon the paramiko module for python. Paramiko is a pure python implementation of the SSH 2 protocol, and lets you log into servers, execute commands and upload files. It prompted me to the idea to write my own pythonic version of a Capistrano like tool, and then use that for deployment. It could be made open source and a perfect oppotunity try out Git on a real project.

Birth of Fabric.

So I swiftly went to work. In two days, I had written the first prototype. It supported the most basic operations such as running remote shell commands, sudo'ing and uploading files. Then I registered my project on nongnu.org (chosen because of their Git support). And ba-da-bim, Fabric became an open source project: https://savannah.nongnu.org/projects/fab/

Fabric looks, on the surface at least (or for those who've only spent a short while with either), a lot like Capistrano. You have a fabfile (as oppose to a capfile) in you project directory, and that file describes all of your deployment tasks (or commands, in Fabric speak).

Commands are really just regular python functions (and you're allowed to call it fabfile.py if you like) that simply makes calls to some other, more magical, functions called operations. Operations are magical because they just sort of exist; you don't import them from a module and don't find them in an object - you just call them.

So, to get the feet wet, here's a hello-world'ish simple fabfile:

set(
    fab_user = 'joe.shmoe',
    fab_mode = 'rolling', # run stuff on one host at a time
    fab_hosts = ['node1.servers.com', 'node2.servers.com'],
)

def deploy():
    "Build and deploy a war file to our app. servers."
    local("mvn clean package")
    put("target/myapp.war", "myapp.war")
    run("mkdir /rollback/$(fab_timestamp)")
    run("cp /$(fab_host)/deply/myapp.war /rollback/$(fab_timestamp)/myapp.war")
    sudo("cp myapp.war /$(fab_host)/deploy/myapp.war")
    sudo("$(fab_host) restart")

Then, all it takes to deploy is a "fab deploy" in the directory where the above file is found. It's still pretty basic, low-level and imparative, but it's a good start.

Fabric also has a built in help system that is powered by your doc-strings; try typing "fab help:deploy" for instance. If you want to see what other commands are availble to you, then simply type "fab list". You can also get a list of operations (the local(), put(), run(), sudo() kind of things) by typing "fab help:ops" and get more details about the individual operation with, for instance, "fab help:put".

And that's basically it for today.

Code generation, automatic or not, have emerged in numerous places as a productivety enhancing technique or a powerful tool of abstraction, or, most often, both.

It come in quite a number of incarnations, from the obvious to the hidden. Some frameworks, like Rails and CGLIB even have code generation as a core feature.

With such broad applicability, and so many different kinds of code generators, they're bound to not all act the same. This difference in behavior is an entrance to mistakes, and is at odds with the useability principle of least surprise.

I have seen, worked with and written numerous code generators - I can hardly imagine a developer who have not. And looking back on this experience, I'll try to jot down a few simple principles that, when a code generator follows them, will make the experience of working with the generator much more pleasent.

However, with the diversity in application, the're bound to be exceptions to the rules. I'll do my best to also dig up these exceptions, and explain why they are a good idea in the given situation.
So, here follows the Principles of the Well Behaved Code Generator:

1: No destructive update.
A code generator should not make any destructive updates. Generated code, like schaffolds in Rails, are intended to releaf the user of burden of writing bioler plate code and leave a fill-in-the-blanks template. Once this is done, the users will often go ahead and... fill in the blanks.

But development is iterative. The foundation from which the code was generated might change over time. Two good examples are generating O/R-mapping code from a database schema, and generating stub-classes from a WSDL in interface-first web service development: both the database schema and the WSDL might change over time as development move through iterations - and when the foundation changes, a good way to keep the generated code up to date it to regenerate it.

But therein lies a catch; what becomes of the filled-in blanks when the code is regenerated? If the generator overwrites the developers careful changes, then it would mean good work is lost into the thin air. Source code management systems and various backup techniques are not withstanding because the generator cannot rely on those features to be available.

Loosing good work is something that we cannot affort; code generators is meant to enhance productivety, not destroy existing productions. There really is only one way around this: code generators must avoid destructive updates at all costs.

If a file we would like to write already exists, give the user a warning. A code generator is not in the business of data backup nor in the business of data shredding. If the file exists, skip it. Then notify the user and let him decide how to handle the situation. If he want's to back the file up, let him do it. If he want's to delete it, let him delete it. But don't go over his head and make arbitry decisions.

Note that this dosn't mean that you cannot touch his files. If you can make the update to his existing code without breaking his work, then by all means go right ahead! I know the visual GUI editor in JDeveloper has the ability to read back the code it generates after it has been altered by hand, and then make changes to the code without breaking the manual changes.

Exception: Not his files
The most obvious exception is when the user isn't the logical "owner" of the generated files. Examples include the Java code a JSP compiler would generate on an application server - the user is by no means suppose to fiddle with these files, so destructive updates to these files are OK as nothing is lost by it. Nothing is suppose to be lost by it anyway. A user making changes to those files had it coming.

Another example is the Java code generated on the fly by Jasper Reports - it is pretty obvious from their hidious style that you aren't suppose to touch those.

2: Exemplary correctness
As a code generator, there can be no doubt about it: your users are a bunch of unshaved, half-asleep coffee slurpers who a too lazy to write their own code, so we might aswell set the standard and show'em How It's Done(TM).

It's easy to make good code bad, but hard to make bad code good. So since a code generator generally don't have a clue about the quality of the existing code, the generator should produce code of the highest quality possible.

This means that, when applicable, generated code should ...

• Be properly indented.
  
• Have meaningful variable, method and class names.
  
• Be consistent with its naming convention, and in line the naming convention of the language/technology/environment.
  
• Have a good and consistent commenting policy.
  
• Be readable.
  
• Make correct use of exception handling.
  
• Make sane use of logging features.
  
• Be debuggable.
  
• Be amenable to testing.
  

Not all of these make sense in every situation as is depends heavily on the domain in which the code generator operates, but the fundamental point remains: code generators should produce high quality code.

Exeption: It's only run once
I have on several occasions written programs that generated (for the most part) SQL insert scripts. These scripts were only intended to be executed once, and then thrown away.

I don't see much point in going out of your way to care for every little detail in the code when you know it's headed for the bin in a few minutes anyway.

This kind of code just have to be free of any bugs (that will manifest themselves the single time the code is run) and get the job done. Therefor, this is an exeption to the principle of exemplary correctness.

3: Blend in
If there's one thing generated should not look like, then it's generated code!

This principle is much along the same vain of the previous principle of exemplary correctness, but I still think it bears mention of its own. Especially becouse I have a nice little story to go with it.

I once wrote a hidious hack code generator for a Java web framework called Rife. This framework came with its own pojo-based ORM framework, and the mapping was (is) written in pure Java in some special Metadata classes. What my nifty little tool did, was to read the source files for a bunch of pojo-beans and then produce and equivelent amount of stub-metadata classes with all the blanks filled with some default values (based on heuristics). In order to make these generated classes blend in with the existing code base, my tool went to great lengths to copy the indentation policy of the pojo-files in the generated metadata-files.

If you used tabs, then it would use tabs, and if you used four spaces then it would use four spaces, and so forth. It didn't copy the placing of curly braces or any other fancy stuff like that, but it did go that extra mile to make the code look a little bit more as if I had written it myself.

More general examples are visual GUI editors and code templates in IDEs. A respectable IDE knows how you like to indent your code and where you like your curly braces, and you'd also think that code generated by an IDE is expected stay around in the code base for a long time and be available for tinkering, so there's simply no excuse for an IDE to not try and replicate your coding style when it generates code.

Exception: Nobody cares
I think the exeptions for the first an the second principles also applies to this principle. If you know that nobody's ever going to deal with the code on an permanent basis, then there's no point in polish like this. Besides, it may not even be possible to deduce these policies or heuristics in the given situation.

4: Let the user decide
User: "Hey! I want a class that extends this other class and implements these interfaces."
Eclipse: "Do you want comments with that?"

Put briefly; if a user wants a generator to override these principles, let him do it. If he wants to overwrite existing files, give him a --force flag. If he wants to turn off comments in generated code, give him a checkbox to click.

You get the idea.

Users want different output in different situations, so the ability to tweak the generators behavior could easily come in handy.

Exception: YAGNI
You ain't gonna need it.

There's seldomly a need to allow tweaking of every thinkably incarnation of these four principles. This is especially true for ad hoc generators that themselves are going to be thrown away shortly after their inception.

I suppose the general rule is that tweakability is good, but there's no need to overdo it, and that basicaly depends on the situation; who's the generator for? what domain is it operating in? what's its life expectancy?

So there we have it
Four principles that, when implemented in a code generator, should make working with it slightly (or considerably, depending on how it is without) more pleasant.

I came upon this article by Pat Eyler of Linux Journal:
http://www.linuxjournal.com/node/1000217/print

A nice little read that suggests that my prediction about Haskell getting a come back ain't entirely off, rather, this guy is seeing pretty much the same patterns - he only did it before I did, plus he's putting his money on Erlang whereas I have a gut feeling about Haskell.

And while I'm at it, here's a bonus link to an article that goes beyond static/dynamic and strong/loose in discussing how to discuss types: http://cdsmith.twu.net/types.html

I have a gut feeling, that the functional language paradigme is on the return.

We went from procedural C to an object-oriented paradigme with C++. This paradigme was refined and improved in Java, and later, in C#.

Now we see the rise of the dynamic languages; Ruby and Python. They augment the object-oriented way by loosening up the type system and give us developers a glipse and a sip of the taste of functional.

Function objects? Lambda expressions? List comprehensions? Closures? We kinda like these new old toys - even Java looks to reinvent itself with closures (at least some wishes it that way, but I've not noticed any JSR yet).

The net result is that we tasted functional and we want more. For the time being, we'll be making due with Ruby, Python and, eventually, Java-with-closures. But these technologies will only take us so far, and wont last forever.

I believe that in two or three years time, Haskell will enjoy much the same popularity that Ruby has now. On the five-year horison, it might top at one-third of the popularity that Java has now, and then start to stagnate and slowly decline again.

Since Haskell is purely functional, I don't think it'll ever reach the popularity of Java and C#. I don't think we'll be able to abandon the object-oriented paradigme the same way we abandoned procedural programming.
We want the power of both worlds, plus an excellent console/intepreted mode and dynamicy in the type system.

I think Haskell will eventually fall short of all these requirements, but in its fall, it will provide the seed to the next generation language - a language that probably haven't been invented yet (or is in the works on some garage-geeks laptop ).
I think it is this next-gen language that will put Java and C# in their respective graves (which is to catch the hearts and minds of developers world wide) - so both of these camps have nothing to worry about until then (that's right, Java people, Ruby is not going to kill you and neither is .NET ).

And while I'm at it, this next language may not be one but several languages. We've been headed for a diversity in programming languages; right tool for the right problem - and I don't see this development go away anytime soon, if ever.

So, why do I think that Haskell, of all choices, will be the functional that gets a come-back? Well, first of all, Lisp is too crude and is pretty much the functional Fortran. Other languages are, well, unkown and don't have the same maturity and wealth in existing libraries to offer.
As far as I can see, the closest contender will be Erlang, but I know very little about this language so it's hard to say how it'll fare - it just seems to be the only other option on the horison.
Haskell, on the other hand, is mature, well tested and has the minimum of libraries to be useful in modern projects. And finally, this book will be the spark that lights the flame.

So, clear as mud, this is my prediction for the near to midterm future of programming languages. Feel free to leave thoughts in the comments

This is an interesting talk by Josh Block on how to get API design right: http://www.infoq.com/presentations/effective-api-design

Some key points from the talk:

• Have a good power to weight ratio. Get a lot of power without excessive bloat, but not too much.
  
• When ever in doubt about a feature, omit it. You can always add it later if people need it, but you can't take features out.
  
• Fail fast. As soon as someone makes a mistake, blow up so they'll know what went wrong.
  
• Avoid returning information in Strings. People will end up parsing the String, so when ever you add information to this String, or change the way it's structured, you will break these parsers.
  
• Make the simple things easy, and to complex things posible.
  
• Design for inheritence, or prevent it.
  
• Be consistent in naming, metaphors and ordering of parameters.