Cymen Vig

It’s simple: the client tells you what they want and you make it.

It isn’t!

The hardest hurdle is actually listening to what the client says and analyzing it.

You need to convert their ideas into code.

That is why you’re being paid, right? The client isn’t typically a developer and they are not going to specify their ideas to the degree that the idea writes the code. The more you understand the client’s intent, the more you can focus development on achieving what the client wants without over engineering.

You need to do it fast.

If you’re using agile methods, you probably have a week to complete all your stories. Most projects are more complex than a week of work. So figuring out their intent will let you get them what they want now over some plumbing that you want but do not need yet.

You need to test it.

Testing is documentation. Testing allows refactoring when you do make write or integrate that piece of plumbing that satisfies a number of demands in a wide horizontal swath of your project. Testing ensures you don’t experience the same bug twice. Testing means your clients aren’t beta testers. Testing means your team doesn’t have maintenance programmers that hate their jobs. Testing means a new developer can come up to speed quickly and make changes to a complex system without worrying that they will break things (if they do, your tests are wrong). Testing means you can deliver a project that you can stand behind professionally with a guarantee that to the best of your ability it is free of bugs and you can afford to guarantee that you will fix any bugs at your cost.

You need to set expectations.

The client has a lot of ideas and wants everything at once. The agile approach using iterations and stories is one way to set expectations as it forces the client to prioritize the development efforts. The client begins to understand what goes into the development effort. They don’t receive a bill for 40 hours as one line item but instead those 40 hours are broken down into pieces of work that where requested by the client.

You need to turn it around and ask them how it is supposed to work.

When purposing stories it is critical to play with the ideas and go back and forth to understand exactly what the story is for. If you don’t understand it now it is going to cost more time later. So turn around things in your mind and ask pointed questions that help you target intent.

And you have to do it in a way that doesn’t piss them off.

We aren’t playing 20 questions. There are usually important questions that determine bigger paths to the actual intent. Avoid being distracted by anything unrelated to the intent of the client. It is easy to distract developers by bringing up technical topics but if they are not related to the intent of the client why bring it up? This time is valuable. You need to focus.

And you need to get every single last little detail and analyze all these details to find the contradictions.

But you only have to do it for a single iteration/story! Imagine if you had to do it for a whole project that was quoted at fixed bid!

There is a fine line here between being a pain in the ass and being valuable. It is also tricky to manage the long term intent of the client with the intent within the scope of the iteration/story. This is the area I hope to learn more about as I don’t quite understand yet how the agile approach works for the scope of a project. I suspect it’s something like this: both the client and the developer have an understanding of how the application is going to work. Both have a lower definition view of the overall project. The work of iterating the project defines it and the design emerges more strongly.

And you need to not get too far ahead of the iteration.

It is daunting to try to get a view of the whole project right now. If the agile approach works as I suspect, you’re being silly. You don’t need to see the whole project. It is okay if it’s murky or unclear. It’s okay if there are dragons out there breathing fire. As long as they are not in this iteration, it is not your concern. Focus on the clarity of the current iteration. The stories should be clear, sharp and well defined with strong intent. And you need to take those stories and deliver and that is all you need to worry about right now.

Last night I realized one of the stories for our internal project was not only about design and in fact had a huge amount of work in it that we had not adequately estimated. The story was about enhancing the user interface so that it would be an “all in one page” application. When clicking on an item, it would be displayed in a window within the main view. I think we estimated only the design part of the story. So I faced the decision: do I stay up late trying to get this done or do we present the story as incomplete? In the end, both happened and I think that is an important lesson.

So I stayed up working on the feature. I used jQuery UI dialogs to make the “windows within the main view” work as I’ve done before. However it was complete work as I needed to change all the links to work within the dialog instead of opening new pages. The server side was fairly straight forward: instead of returning a view with the layout (the layout having the site wrapper so logo, etc), return a partial which is only the contents of the actual view. Then I added some classes to all the links and had jQuery intercept the click action so that the content was fetched via AJAX and then put into the dialog window. This worked however I also needed to handle form posts.

Then after all of that was working, I realized we needed to update the main view if any edits had been saved in the dialog! This turned into a ball of wax that was solvable but quickly looked to be going the way of code that I personally would rather not be working on because changing it would be difficult. It was also untested as I knew from my experience during the day that writing it test-driven would take more time than I had that evening…

What was really wrong with this approach?

If one clicks on an item and edits say the name and saves it then goes back to the view by closing the pop-up dialog the main view now has to fetch the item again from the server as JSON and update itself. So if we edit A and change the name to BA and save it the record is PUT/POSTed to the server. Then we go back to our main view and we have another GET to retrieve the same data we just posted! This is not helping the speed of our UI although it is fast in practice. Our code has to manually decide what items in the JSON response need to be put into the DOM to update it. It’s a very manual process.

What would be better?

Moving to a framework like Backbone.js. The way this would work is:

• Main view: has three empty lists for our item. We may be able to optimize by having the server populate these for the initial request.

• Main view load: Backbone.js makes a request to the server most likely using it’s idea of collections. The server (Rails) responds in JSON with say an array of items. Backbone.js takes these items and puts them into a collection as objects using the models we have defined.

• On edit: Backbone.js sends the update to the server but most critically also triggers an event that the item has been updated. The main view receives the trigger and can update itself using the data that Backbone.js holds client side. It does not need to make a request to the server.

There are a couple big things going on here however from a testing standpoint the big idea is don’t parse DOM but instead have JavaScript responsible for updating the DOM (without having to parse it to figure out what to update) and the state on the server side. A big advantage of this is that we can leverage the power of convention as used in a framework to make possible things like client-side storage (using HTML5 local storage). We may not use that capability in this project but it’s there. We also have a more responsive user interface as we do not have to ask the server what we just did in order to update a related view. We also in theory have less code to test as we do not need to test Backbone.js – we only want to test our own code which should be fewer lines as framework should be responsible for some of the plumbing that was causing ugly code when we tried to do it ourselves.

As mentioned previously, some of the apprentices began working on an internal project. Our iteration meetings are on Tuesday mornings so today we were working more intensely on the iteration stories. The story I worked on was making items drag and droppable between three lists on a page. I’d done something similar with jQuery UI sortable before but not test driven. I wrote the code last week to know what the direction was and today I left that code literally at home. Sue and I paired and wrote the same code using a test-driven approach with Jasmine. At the end of the day, we had completed the task including mocking the jQuery AJAX call that went to the backend. The best part was writing a test for the case where the server/internet disappears. It is possible to simulate this without testing but it’s a pain to do over and over again. However with testing, it is easy to mock the AJAX call so it fails (calls the error parameter of $.ajax()). So now when the server goes away and one drag and drops between lists the item goes back to the original list if it fails to save.

It was an interesting experience but it made it even clearer that the typical mash up of putting some data in the DOM for your jQuery script to use has a lot of flaws. For example, we put data-val-id and data-val-modified attributes on our list items. When the view is loaded, the server is rendering it server side. Then our jQuery script can use these attributes to do sorting and make AJAX calls client side. However, sometimes it has to update the DOM depending on what the server returns. And it has to do some conversions between JSON response data and DOM attributes as strings. And sorting the lists again requires conversion if sorting by numbers (well, technically it appeared to sort integer strings as expected but it seemed like a bad thing to rely on).

The hardest part of testing this was when we were close to the DOM. Jasmine has fixtures so we can populate the DOM with some HTML content and then run our tests on that however doing comparisons of jQuery objects and DOM objects is problematic in some cases. There are some weird work tricks that are ugly. If one is practicing test-driven development, the impetus to move on to a more structured approach to asynchronous user interfaces comes faster and harder because there is now testing pain on top of development pain.

What is an asynchronous user interface?

Simply put, it is a user interface in which the state seen by the user does not always match the state on the server. An example of this would be dragging and dropping an item from one list to another. The drop of the item onto the new list at that moment in time happens on the client without knowledge of the server. The client then sends an update request in the background to change state on the server. The whole purpose of doing this is to make the user interface fast. In our case, we indicate that we are doing something in the background by putting a small image next to the item that animates indicating action. Then when the request to the server completes, we remove the image and that conveys to the client that server state now matches client state. In the case of an error, we move the item back to the original list. This works however it really needs a bit more information as I would want to know why the item suddenly went back to the original list.

The speaker at today’s 8th Light University (on 3/23/2012) was Stephanie Briones. Stephanie completed a design apprenticeship under Billy Whited. Her presentation on bridging the gap between developers and designers was the final challenge in her apprenticeship. She is the first design apprentice to complete a residency at 8th Light!

The presentation was fun. Stephanie started by presenting some of the issues that cause the gaps between the two camps and then focused on what can connect the two gaps. The final part of the presentation was designing a web application that had an overview screen, a login screen and screen to manage a subscriptions (possibly a large number). It was an interesting challenge. The audience divided up into groups based on where they were sitting with at least one designer in each group (8th Light is currently developer-heavy if that wasn’t clear already). Then each team presented one or two designs.

From listening around the office, I came to the conclusion that having Stephanie complete her apprenticeship and move onto client work is a big deal. 8th Light has had a number of developer/craftsmen apprenticeships complete and accept positions with 8th Light but we need more designers. I am very encouraged to see that Paul and Micah really do believe in growing the company in a careful way instead of trying to get big fast.

The program recordMyDesktop works well for screencasting but the process of recording, converting and uploading to a Vimeo or Youtube is not straight forward. The versions of ffmpeg and mencoder (from mplayer) in even Ubuntu 11.10 cannot transcode the Ogg Vorbis output of recordMyDesktop to a suitable format for the video hosting providers. Fortunately, there is a way to transcode that is fast and straight forward.

1. Change recordMyDesktop settings: open recordMyDesktop and click on the Advanced button. Go to the Performance tab and uncheck “Encode On the Fly”. You can try leaving it enabled but some people report the audio gets out of sync with the video.

2. Verify sound settings: if you use an external USB headset (highly recommend), you likely will need to open the system mixer and set it as the default input as recordMyDesktop requires the system mixer to take care of the audio settings. Under Ubuntu 11.10, you can open the Sound application and go to the Input tab. There you can select the correct input, verify it is not muted and also verify it is working.

3. Test recording: make a very short recording while speaking and play it back to verify it is working correctly.

4. Record screencast: note that recordMyDesktop puts a small red circle on your system status bar. You can click that to pause or stop the recording. Because we disabled “Encode On the Fly” when you stop encoding recordMyDesktop will need to perform the encoding. This takes a little while and it grows in time with the length of the recording. You can cancel the encoding if you want to trash it. The output file will be in your home directory as out.ogv.

5. Use HandBrakeCLI to transcode to MP4: install HandBrake if you do not already have it. I used these directions from AskUbuntu. The transcoding process is extremely simple with HandBrakeCLI: ` HandBrakeCLI -i out.ogv -o out.mp4`

You can now upload the MP4 file to Youtube and/or Vimeo and it will work as expected.

I’m going to do the wordwrap kata at an 8th Light University soon. Part of the process is now recording a run through and having it reviewed to ensure the kata is useful. I meant to get this done earlier this week but a perfect storm of nervousness, project deadlines on another projects, mulling over good ideas of how to show rspec output in vim (to avoid jumping back and forth) and figuring out screen recording in Linux meant I didn’t get it done until today.

One of the criteria for presenting a kata is doing something new. I am doing this kata in a way that we end up with a two path branch (if … else …). Uncle Bob’s example ends with a three path branch (if … else if … else …). I am also presenting one of Uncle Bob’s variants in which the wrong path is taken. Katas don’t typically have a moral but this one does and it is that when writing test driven code the tests should indeed drive the code however one has to think carefully about the tests. If one jumps too far ahead and tests complex functionality that doesn’t build on a solid base one runs the risk of having to backtrack. A sign of this is getting stuck when writing the code. If you’re guessing, your tests are probably wrong and you should focus on fixing the tests not the code itself!

One of the facets of presenting a kata is that there is a limited amount of time to do so. Typically, a kata ends with a clean refactored code. This one does not due to the time limit. However, if one has more time it is indeed possible to refactor the two branches (if … else …) to no branches. I won’t present the steps here to do so but one solution that I thought was relatively clean is this:

I’m coming to the Rails world via the work of 8th Light on the Clean Architecture approached advocated by Robert Martin and 8th Light. My prior experience with web frameworks is Zend (PHP) and ASP.NET MVC 2 and 3 (using C#). My initial thoughts on this is that Ruby on Rails has a lot of “magic”. Way more than ASP.NET MVC. I think this is because of the collaborative nature of Rails. There are tons of gems at rubygems.org and Microsoft is playing catch up on this with NuGet. However my opinion is that Microsoft would do well not to fully match the Rails experience. At first, gems seem like an excellent idea. But the fast pace of change with Rails, the locking down to specific gem versions in Gemfiles and the perverse extent of “magic” sometimes cause quite a few problems that aren’t present with ASP.NET MVC.

It’s as if all the work to improve Rails knocked off all the sharp edges (some still present in ASP.NET MVC) however it kept going. Now your JavaScript files are not static resources but are instead parsed files on the server side for doing inclusions. The same thing is happening with Stylesheets! These two things are examples of “magic” which I think goes too far. Rails is overly complex when you attempt to go against the grain. At least if you attempt to start from vanilla and withdraw (as Clean Architecture advocates away from the tight integration of business logic into models that tie into ActiveRecord database logic). I have seen some blog posts arguing that Rails 3 makes things much more modular and that this fact is not currently well known or popular in the Rails community. I would like to learn more about this…

I see something similar to Clean Architecture advocated in the ASP.NET MVC in Action (Manning) book by Jeffrey Palermo, Ben Scheirman, and Jimmy Bogard. It is not presented as abstractly as Robert Martin has done in the 8th Light blog post but instead wrapped into the presentation of how the authors use ASP.NET MVC and AutoMapper to separate the business logic from the framework in a complex web application. The book actually feels somewhat like two stories in one: how to use ASP.NET MVC and how authors use it in a specific context. However it works well as it is an “in Action” book and it deliveries more than expected (dependency injection, testing, etc).

Similarly, I’ve split this post into two streams:

1. Rails versus ASP.NET MVC

2. Clean Architecture

I don’t have anything more to currently say about Rails as it is early days. No doubt the “magic” will soon seem inconsequential. However I wonder if a better approach to Clean Architecture (which still conforms to Robert Martin’s blog post) is to let the web framework be the web framework. Don’t fight it. Let it have it’s models but use those models for presentation and use Interactors to populate those models in the controller actions from a completely separated code base which can be tested in isolation. The way the ASP.NET MVC in Action authors have accomplished this is to use AutoMapper to map between the framework models and the business logic models. I think that approach makes a lot of sense and results in a clean separation of concerns.

Internal Project

I’m currently working on a feature in which a page has three lists and items can be dragged between the lists. This is quite easy to do with jQuery UI however I’ve never tried to test something like that. The current approach I’m looking at is using jasmine (with spies) and that is promising.

Kata

It is a struggle to juggle things. I need to finish a video of a practice run of my kata so that it can be reviewed. I am happy with the kata however I’m uncertain about the narration. The other issue is that it is always a struggle to share the status of the tests passing (green) or failing (red) with the audience. Typically, people flip back and forth between terminal sessions or have a small output window in their IDE. These options don’t work particularly well and I’m thinking about how to update the status bar in the vim editor to reflect the pass/fail of the specs in the current buffer. For the kata, we really only need red/green not

Today the team of apprentices working on the internal project presented our first iteration to Paul playing the role of the client. We completed the iteration successfully which was a relief however it was not without a lot of last minute effort. Thankfully, the deadline was extended from 10am to 2pm which gave us enough time to get the testing up to snuff. Paul as the client accepted the iteration then we collaborated to create the next iteration. After the purposed stories were created, the team without the client estimated the stories. Later on Paul returned and had to move one story to the backlog as our total hours were exceeded.

Then we looked at the code base with Paul and he pointed out some areas in which we were lacking “screaming architecture”. We also discussed some of the abstractions present in the code base. The main questions were why were we using the abstraction? What was the value of having a layer of abstraction between the interactor and active record which we’ve called a database adapter? We also looked at some of the tests and Paul questioned why some testing was done as it was. Did we really need a singleton pattern in some cases? Or was that only present in order to make testing easier yet there were other ways to achieve the same thing?

These were all interesting questions. Our architecture has definitely become more complex and I think the reasons for this are:

• using the interactor approach

• inheriting an existing project and going with it instead of questioning the abstractions as a team

• complicated testing

If we continue with the “controller actions pass a callback to the interactor” approach it seems clear to actually benefit from this we need to pass in a failure path callback which could do something like return an HTTP 500 response or some other response with error text. It would also be useful to see what exactly using a callback is giving us. Paul mentioned at one point we could simply call the interactor and get a response which in other interactor approaches would be a presenter object. Perhaps when Doug and I go over my work (as suggested by Paul) we can discuss this.

However I don’t mean to be overly pessimistic. The project is going well. The next interaction has some exciting stories!

So having 4 make that 3 programmers with a designer and a low number of hours is kind of tricky. We’re also using the Interactor approach that is still not solidified at 8th Light. We found that out after we put in our estimates on stories. I spent the day working on that project with Wai Lee and Ben. I spent the evening working on it some more as we did not have enough time to complete it the next day before the 10am deadline.

It’s coming along but it is frustrating trying to work with a large group on a small project with a low number of hours. It seems like half the team is waiting for a leader and the other half is just doing stuff (which sometimes overlaps and results in duplicate work). I don’t know what to do about waiting for a leader but for duplicating work it would make sense to break down the big stories into smaller stories and go by that. For a new project though with beginners even that can be difficult so working in pairs and communicating seems to be very important.

Today at 8LU Jeff Cohen from Code Academy presented “Beginner-focused Learning” which was both interesting and useful. Part of working at 8th Light is teaching others no matter what you’re position is so it was very relevant.

Doug had some more audio-visual work for me as we need to put in a new VGA run to the projector and Doug redid the audio in the room. We also relocated the video camera that is used to share 8LU with those at remote sites. The changes seemed to work well.

I came to a realization today: spec tests in Clojure are likely going to be somewhat different than spec tests in other languages. Clojure is a functional language and this means my program is comprised of a large number of functions. The organization of the project is in namespaces instead of classes. An HTTP request is going to call a large number of functions in a many name spaces before a final value is returned that represents the response. My testing is also organized by namespace and for complicated namespaces the testing gets more complex as one reads down the file because simpler functions are tested first. I’ve extracted number of simple functions from complex functions in order to reduce complexity and improve how the code reads.

The part that is strange to me compared to say testing a ruby program is that at some point one seems to have to have a fairly large number of inputs (or a small number of complex inputs) in the tests… Of course, it may also be the nature of testing something like an HTTP server. In any case, I’ve asked a Craftsman to review my project and I am looking forward to discussing this with them.