Cymen's Blog

Finish up the setup of the internal site on production server

Ben and I completed setting up the internal rails project on a production server. We were mimicking an existing production site that used vlad for production. That was a bit confusing as “rails -T” didn’t show all the tasks plus one of the rake files seemed to be in the wrong location. There is likely some room for improvement there. There were a number of little system administration things  and programming tasks that needed to be done so it took most of the day. We also changed the instance of our application running on Heroku to be a staging server which was interesting without having direct shell access.

More Backbone

I spent some of the morning looking into Backbone and jQuery UI Sortable. I wanted to use both but now Backbone would be responsible for updating the server and so forth. I figured out how to bind a Backbone model to an event triggered by jQuery UI Sortable however I wasn’t too sure of my approach. Mark (another apprentice) and I looked it over. The event is happening on an individual item within a list. Each item is rendered by a Backbone view and the list is also a Backbone view. Mark and I wondered if we could bind to events on the list at a higher level instead of at the item level. We figured out an approach that worked but we are really interested in inter-list events (moving items from one list to another).

I discussed this with Mike (a craftsmen) to get his view on things. He mentioned that his approach is to make Backbone in charge of everything and one thing I might consider is having a Backbone view being responsible for rendering the list and item views. So adding another view layer to the approach. And then having that view be responsible for binding/configuring jQuery UI. We have another internal tool that is doing something similar so I’m going to look at it and see how this problem was solved before. One of the problems with separating responsibilities with events is that how things work can get confusing. One has to make that leap from the triggered event to the listener to trace how the system works.

I really wanted to do a story this iteration on a project but it wasn’t selected by the client. My scheme was to get the current stories done and then pluck it from the backlog but I then learned it doesn’t work that way: in the event more stories are needed for the current iteration it is likely the client will pick from the backlog by their priority. Thankfully, I discussed my interest in a specific story with the client and happily received the offer of being free to work on that story if the other stories are finished. However I’m still left wondering how developers sell the client on plumbing that they really think is important but doesn’t result in a clearly visible change as a new feature would.

How do you sell necessary but unexciting stories?

Deploying the internal application

I paired with Ben for a good part of the day in a big push to get our internal rails application deployed to a production server. The server is running a specific version of ruby and while it does have RVM so in theory one could use any version in practice Passenger running under Apache is going to use a specific version of ruby with global gems for everything. You can install the passenger gem and compile an apache module and then use that in your virtual host configuration in apache but it’s really ugly and a total hack. So we took the easy path and changed our development environment to match production. With the low cost of virtual machines, I think it makes sense to either have a couple of big servers with one running the most recent stable version of ruby or having one for each application if there is only a handful to begin with. On the other hand, maybe we can get the other applications on that server up to a more recent version of ruby and everyone benefits (if I recall correctly, 1.9.3 had some performance improvements over 1.9.2).

This deployment is complicated by the fact we also want to use SSL and authenticate with our google accounts (using OAuth2). We finished the day with the stack working except for redirect issues between the google authentication services and our host preventing any actually usage. Hopefully, it’s just the DNS update for the new server name taking time to propagate.

Trying out Backbone.js

I spent some time today learning more about Backbone.js. 8th Light craftsman Mike Jansen has a project on github named Childermass that has been quite useful:

Childermass is a way to browse GitHub. It dives into the depths of repos and users and plucks out the most interesting information for you to read.

Childermass is largely an experiment with Backbone.js, sitting on top of Rails. The goal is to have a sufficiently complex example of structured, test driven Javascript for people to emulate and use in their next project.

Using Childermass as an example, I recreated the “three list” view of our internal application in Backbone.js. My reason for doing this is that currently the JavaScript code for the view has to manually update the items in the lists after they are edited in a “popup” (really an in page jQuery UI Dialog). There is also a lot of parsing of the DOM going on to figure out what to send to the server. In short, it is trying to do what Backbone.js is doing but it’s painful and it’s more lines of code that have to be tested. It’s also inflexible compared to Backbone.js. For example, say in the list you want to show another field from the item. With Backbone.js, this is a simple change of the template that renders the item. With the DIY approach you’ll likely have a couple of places to make changes and your testing is likely going to need to change.

It’s simple: the client tells you what they want and you make it.

It isn’t!

The hardest hurdle is actually listening to what the client says and analyzing it.

You need to convert their ideas into code.

That is why you’re being paid, right? The client isn’t typically a developer and they are not going to specify their ideas to the degree that the idea writes the code. The more you understand the client’s intent, the more you can focus development on achieving what the client wants without over engineering.

You need to do it fast.

If you’re using agile methods, you probably have a week to complete all your stories. Most projects are more complex than a week of work. So figuring out their intent will let you get them what they want now over some plumbing that you want but do not need yet.

You need to test it.

Testing is documentation. Testing allows refactoring when you do make write or integrate that piece of plumbing that satisfies a number of demands in a wide horizontal swath of your project. Testing ensures you don’t experience the same bug twice. Testing means your clients aren’t beta testers. Testing means your team doesn’t have maintenance programmers that hate their jobs. Testing means a new developer can come up to speed quickly and make changes to a complex system without worrying that they will break things (if they do, your tests are wrong). Testing means you can deliver a project that you can stand behind professionally with a guarantee that to the best of your ability it is free of bugs and you can afford to guarantee that you will fix any bugs at your cost.

You need to set expectations.

The client has a lot of ideas and wants everything at once. The agile approach using iterations and stories is one way to set expectations as it forces the client to prioritize the development efforts. The client begins to understand what goes into the development effort. They don’t receive a bill for 40 hours as one line item but instead those 40 hours are broken down into pieces of work that where requested by the client.

You need to turn it around and ask them how it is supposed to work.

When purposing stories it is critical to play with the ideas and go back and forth to understand exactly what the story is for. If you don’t understand it now it is going to cost more time later. So turn around things in your mind and ask pointed questions that help you target intent.

And you have to do it in a way that doesn’t piss them off.

We aren’t playing 20 questions. There are usually important questions that determine bigger paths to the actual intent. Avoid being distracted by anything unrelated to the intent of the client. It is easy to distract developers by bringing up technical topics but if they are not related to the intent of the client why bring it up? This time is valuable. You need to focus.

And you need to get every single last little detail and analyze all these details to find the contradictions.

But you only have to do it for a single iteration/story! Imagine if you had to do it for a whole project that was quoted at fixed bid!

There is a fine line here between being a pain in the ass and being valuable. It is also tricky to manage the long term intent of the client with the intent within the scope of the iteration/story. This is the area I hope to learn more about as I don’t quite understand yet how the agile approach works for the scope of a project. I suspect it’s something like this: both the client and the developer have an understanding of how the application is going to work. Both have a lower definition view of the overall project. The work of iterating the project defines it and the design emerges more strongly.

And you need to not get too far ahead of the iteration.

It is daunting to try to get a view of the whole project right now. If the agile approach works as I suspect, you’re being silly. You don’t need to see the whole project. It is okay if it’s murky or unclear. It’s okay if there are dragons out there breathing fire. As long as they are not in this iteration, it is not your concern. Focus on the clarity of the current iteration. The stories should be clear, sharp and well defined with strong intent. And you need to take those stories and deliver and that is all you need to worry about right now.

Last night I realized one of the stories for our internal project was not only about design and in fact had a huge amount of work in it that we had not adequately estimated. The story was about enhancing the user interface so that it would be an “all in one page” application. When clicking on an item, it would be displayed in a window within the main view. I think we estimated only the design part of the story. So I faced the decision: do I stay up late trying to get this done or do we present the story as incomplete? In the end, both happened and I think that is an important lesson.

So I stayed up working on the feature. I used jQuery UI dialogs to make the “windows within the main view” work as I’ve done before. However it was complete work as I needed to change all the links to work within the dialog instead of opening new pages. The server side was fairly straight forward: instead of returning a view with the layout (the layout having the site wrapper so logo, etc), return a partial which is only the contents of the actual view. Then I added some classes to all the links and had jQuery intercept the click action so that the content was fetched via AJAX and then put into the dialog window. This worked however I also needed to handle form posts.

Then after all of that was working, I realized we needed to update the main view if any edits had been saved in the dialog! This turned into a ball of wax that was solvable but quickly looked to be going the way of code that I personally would rather not be working on because changing it would be difficult. It was also untested as I knew from my experience during the day that writing it test-driven would take more time than I had that evening…

What was really wrong with this approach?

If one clicks on an item and edits say the name and saves it then goes back to the view by closing the pop-up dialog the main view now has to fetch the item again from the server as JSON and update itself. So if we edit A and change the name to BA and save it the record is PUT/POSTed to the server. Then we go back to our main view and we have another GET to retrieve the same data we just posted! This is not helping the speed of our UI although it is fast in practice. Our code has to manually decide what items in the JSON response need to be put into the DOM to update it. It’s a very manual process.

What would be better?

Moving to a framework like Backbone.js. The way this would work is:

• Main view: has three empty lists for our item. We may be able to optimize by having the server populate these for the initial request.
• Main view load: Backbone.js makes a request to the server most likely using it’s idea of collections. The server (Rails) responds in JSON with say an array of items. Backbone.js takes these items and puts them into a collection as objects using the models we have defined.
• On edit: Backbone.js sends the update to the server but most critically also triggers an event that the item has been updated. The main view receives the trigger and can update itself using the data that Backbone.js holds client side. It does not need to make a request to the server.

There are a couple big things going on here however from a testing standpoint the big idea is don’t parse DOM but instead have JavaScript responsible for updating the DOM (without having to parse it to figure out what to update) and the state on the server side. A big advantage of this is that we can leverage the power of convention as used in a framework to make possible things like client-side storage (using HTML5 local storage). We may not use that capability in this project but it’s there. We also have a more responsive user interface as we do not have to ask the server what we just did in order to update a related view. We also in theory have less code to test as we do not need to test Backbone.js — we only want to test our own code which should be fewer lines as framework should be responsible for some of the plumbing that was causing ugly code when we tried to do it ourselves.

As mentioned previously, some of the apprentices began working on an internal project. Our iteration meetings are on Tuesday mornings so today we were working more intensely on the iteration stories. The story I worked on was making items drag and droppable between three lists on a page. I’d done something similar with jQuery UI sortable before but not test driven. I wrote the code last week to know what the direction was and today I left that code literally at home. Sue and I paired and wrote the same code using a test-driven approach with Jasmine. At the end of the day, we had completed the task including mocking the jQuery AJAX call that went to the backend. The best part was writing a test for the case where the server/internet disappears. It is possible to simulate this without testing but it’s a pain to do over and over again. However with testing, it is easy to mock the AJAX call so it fails (calls the error parameter of $.ajax()). So now when the server goes away and one drag and drops between lists the item goes back to the original list if it fails to save.

It was an interesting experience but it made it even clearer that the typical mash up of putting some data in the DOM for your jQuery script to use has a lot of flaws. For example, we put data-val-id and data-val-modified attributes on our list items. When the view is loaded, the server is rendering it server side. Then our jQuery script can use these attributes to do sorting and make AJAX calls client side. However, sometimes it has to update the DOM depending on what the server returns. And it has to do some conversions between JSON response data and DOM attributes as strings. And sorting the lists again requires conversion if sorting by numbers (well, technically it appeared to sort integer strings as expected but it seemed like a bad thing to rely on).

The hardest part of testing this was when we were close to the DOM. Jasmine has fixtures so we can populate the DOM with some HTML content and then run our tests on that however doing comparisons of jQuery objects and DOM objects is problematic in some cases. There are some weird work tricks that are ugly. If one is practicing test-driven development, the impetus to move on to a more structured approach to asynchronous user interfaces comes faster and harder because there is now testing pain on top of development pain.

What is an asynchronous user interface?

Simply put, it is a user interface in which the state seen by the user does not always match the state on the server. An example of this would be dragging and dropping an item from one list to another. The drop of the item onto the new list at that moment in time happens on the client without knowledge of the server. The client then sends an update request in the background to change state on the server. The whole purpose of doing this is to make the user interface fast. In our case, we indicate that we are doing something in the background by putting a small image next to the item that animates indicating action. Then when the request to the server completes, we remove the image and that conveys to the client that server state now matches client state. In the case of an error, we move the item back to the original list. This works however it really needs a bit more information as I would want to know why the item suddenly went back to the original list.

The speaker at today’s 8th Light University (on 3/23/2012) was Stephanie Briones. Stephanie completed a design apprenticeship under Billy Whited. Her presentation on bridging the gap between developers and designers was the final challenge in her apprenticeship. She is the first design apprentice to complete a residency at 8th Light!

The presentation was fun. Stephanie started by presenting some of the issues that cause the gaps between the two camps and then focused on what can connect the two gaps. The final part of the presentation was designing a web application that had an overview screen, a login screen and screen to manage a subscriptions (possibly a large number). It was an interesting challenge. The audience divided up into groups based on where they were sitting with at least one designer in each group (8th Light is currently developer-heavy if that wasn’t clear already). Then each team presented one or two designs.

From listening around the office, I came to the conclusion that having Stephanie complete her apprenticeship and move onto client work is a big deal. 8th Light has had a number of developer/craftsmen apprenticeships complete and accept positions with 8th Light but we need more designers. I am very encouraged to see that Paul and Micah really do believe in growing the company in a careful way instead of trying to get big fast.

The program recordMyDesktop works well for screencasting but the process of recording, converting and uploading to a Vimeo or Youtube is not straight forward. The versions of ffmpeg and mencoder (from mplayer) in even Ubuntu 11.10 cannot transcode the Ogg Vorbis output of recordMyDesktop to a suitable format for the video hosting providers. Fortunately, there is a way to transcode that is fast and straight forward.

1. Change recordMyDesktop settings: open recordMyDesktop and click on the Advanced button. Go to the Performance tab and uncheck “Encode On the Fly”. You can try leaving it enabled but some people report the audio gets out of sync with the video.
2. Verify sound settings: if you use an external USB headset (highly recommend), you likely will need to open the system mixer and set it as the default input as recordMyDesktop requires the system mixer to take care of the audio settings. Under Ubuntu 11.10, you can open the Sound application and go to the Input tab. There you can select the correct input, verify it is not muted and also verify it is working.
3. Test recording: make a very short recording while speaking and play it back to verify it is working correctly.
4. Record screencast: note that recordMyDesktop puts a small red circle on your system status bar. You can click that to pause or stop the recording. Because we disabled “Encode On the Fly” when you stop encoding recordMyDesktop will need to perform the encoding. This takes a little while and it grows in time with the length of the recording. You can cancel the encoding if you want to trash it. The output file will be in your home directory as out.ogv.
5. Use HandBrakeCLI to transcode to MP4: install HandBrake if you do not already have it. I used these directions from AskUbuntu. The transcoding process is extremely simple with HandBrakeCLI:
   
HandBrakeCLI -i out.ogv -o out.mp4

You can now upload the MP4 file to Youtube and/or Vimeo and it will work as expected.

I’m going to do the wordwrap kata at an 8th Light University soon. Part of the process is now recording a run through and having it reviewed to ensure the kata is useful. I meant to get this done earlier this week but a perfect storm of nervousness, project deadlines on another projects, mulling over good ideas of how to show rspec output in vim (to avoid jumping back and forth) and figuring out screen recording in Linux meant I didn’t get it done until today.

One of the criteria for presenting a kata is doing something new. I am doing this kata in a way that we end up with a two path branch (if … else …). Uncle Bob’s example ends with a three path branch (if … else if … else …). I am also presenting one of Uncle Bob’s variants in which the wrong path is taken. Katas don’t typically have a moral but this one does and it is that when writing test driven code the tests should indeed drive the code however one has to think carefully about the tests. If one jumps too far ahead and tests complex functionality that doesn’t build on a solid base one runs the risk of having to backtrack. A sign of this is getting stuck when writing the code. If you’re guessing, your tests are probably wrong and you should focus on fixing the tests not the code itself!

One of the facets of presenting a kata is that there is a limited amount of time to do so. Typically, a kata ends with a clean refactored code. This one does not due to the time limit. However, if one has more time it is indeed possible to refactor the two branches (if … else …) to no branches. I won’t present the steps here to do so but one solution that I thought was relatively clean is this:

I’m coming to the Rails world via the work of 8th Light on the Clean Architecture approached advocated by Robert Martin and 8th Light. My prior experience with web frameworks is Zend (PHP) and ASP.NET MVC 2 and 3 (using C#). My initial thoughts on this is that Ruby on Rails has a lot of “magic”. Way more than ASP.NET MVC. I think this is because of the collaborative nature of Rails. There are tons of gems at rubygems.org and Microsoft is playing catch up on this with NuGet. However my opinion is that Microsoft would do well not to fully match the Rails experience. At first, gems seem like an excellent idea. But the fast pace of change with Rails, the locking down to specific gem versions in Gemfiles and the perverse extent of “magic” sometimes cause quite a few problems that aren’t present with ASP.NET MVC.

It’s as if all the work to improve Rails knocked off all the sharp edges (some still present in ASP.NET MVC) however it kept going. Now your JavaScript files are not static resources but are instead parsed files on the server side for doing inclusions. The same thing is happening with Stylesheets! These two things are examples of “magic” which I think goes too far. Rails is overly complex when you attempt to go against the grain. At least if you attempt to start from vanilla and withdraw (as Clean Architecture advocates away from the tight integration of business logic into models that tie into ActiveRecord database logic). I have seen some blog posts arguing that Rails 3 makes things much more modular and that this fact is not currently well known or popular in the Rails community. I would like to learn more about this…

I see something similar to Clean Architecture advocated in the ASP.NET MVC in Action (Manning) book by Jeffrey Palermo, Ben Scheirman, and Jimmy Bogard. It is not presented as abstractly as Robert Martin has done in the 8th Light blog post but instead wrapped into the presentation of how the authors use ASP.NET MVC and AutoMapper to separate the business logic from the framework in a complex web application. The book actually feels somewhat like two stories in one: how to use ASP.NET MVC and how authors use it in a specific context. However it works well as it is an “in Action” book and it deliveries more than expected (dependency injection, testing, etc).

Similarly, I’ve split this post into two streams:

1. Rails versus ASP.NET MVC
2. Clean Architecture

I don’t have anything more to currently say about Rails as it is early days. No doubt the “magic” will soon seem inconsequential. However I wonder if a better approach to Clean Architecture (which still conforms to Robert Martin’s blog post) is to let the web framework be the web framework. Don’t fight it. Let it have it’s models but use those models for presentation and use Interactors to populate those models in the controller actions from a completely separated code base which can be tested in isolation. The way the ASP.NET MVC in Action authors have accomplished this is to use AutoMapper to map between the framework models and the business logic models. I think that approach makes a lot of sense and results in a clean separation of concerns.