Cymen's Blog

At my prior position one of my ongoing tasks was to create a CMS on top of ASP.NET MVC. If you’re not familiar with it, ASP.NET MVC is very similar to Ruby on Rails without the magic and without any prescribed data access method (so no equivalent to Active Record in the box).

Now an important point is that this was not the first CMS my employer had gone through the process of creating. They already had one that was created in ASP/ASP.NET. It had some performance issues. One of these was that the performance of the data access method via an ORM was not well understood or monitored by the developers. One day I dove into this issue and discovered two facts:

1. Performance increases by adding indexes where appropriate based on analysis of the queries in the SQL Server engine were easy to obtain.
2. There was a fatal performance flaw: the CMS concatenated keys and then did lookups or joins by parts of the concatenated key. So like a composite or compound key yet the key was actually stored as a key and queried upon as a whole targeting subsets of the key value!

It is almost impossible to optimize for the second issue. It didn’t look like it would cause the websites to fall over but it did increase the load on the SQL server and increase latency in response. This was typically a negligible amount but once you had a handful or more of these sites hitting the same database and occasionally getting indexed by a search engine all bets were off.

Based on this experience, one of the requirements of my CMS was to be very very simple. So no ORM and no more complexity than necessary. These restrictions, particularly the ORM one, where often annoying however in the end I think the CMS we created accomplished these goals. It certainly had some issues with how it worked that required some ingenuity to resolve however the performance was great and the code base was small. It certainly had less features than the old CMS however as those of us using ASP.NET MVC became more familiar with the MVC approach it became clear that making the CMS implement complex requirements may not be the best idea. Of course selling developers still on the old CMS on that idea was difficult. They would need to see this for themselves by working more with ASP.NET MVC. In truth, I suspect it was a bit of both the idea that maybe the CMS shouldn’t provide everything and that it could provide a bit more than it currently did.

I will continue this series on the CMS with some background on the data access method , using the nested set model in SQL to find related pages at the same depth and branch in navigation and other interesting tidbits that came up while working on the project.

Recent experiences pairing

This week I’ve paired with Angeleah, Michael, Mike and Ben. We’re all working on the internal project. Angeleah and I worked on a story that was not completed in the prior iteration. The reason it wasn’t completed was that the interface wasn’t done. We had some minimal HTML with input fields while the mock up shown to the client had an edit link next to each field that could be edited and looked much nicer. When we tried to actually implement this we discovered it was very annoying to have an edit link. The work flow would be:

• Click edit
• Make change
• Click save
• Repeat for each field that needs to be edited

We discussed how we felt it should work and what are options were. We decided that it would be much nicer to have an edit view that toggled all the fields to editable status and then one would click save and return to the non-edit view (which could then be submitted or edited yet again). We checked with our customer via email for approval to this change and then went forward with it. The end result was a much nicer work flow and we brought over some style changes that resulted in a much improved appearance.

Michael and I worked on a couple of stories. One of the more interesting ones was an export feature. Michael had written the backend already with Ben but we needed a front end for it. It was a new feature on the site and starting something new even when it doesn’t seem that big always has the potential for taking more time than expected. It was a pleasant surprise to test-drive it to completion in much less time than estimated.

Mike and I worked on an export feature that required an oddly formatted CSV file for import into a proprietary application. This was frustrating as the import would do odd things like sum up two lines and create one import item instead of two import items. We had to try many variations of the file format to come up with one that did what we wanted. Mike spent a lot more time than I did on this story (along with Michael) and I was very thankful for that.

Using Amazon S3 with aws-s3 gem

Ben and I paired on a story that required we make some small test scripts to upload and then retrieve files from Amazon S3 securely. We used my Amazon Web Services account and the aws-s3 gem with Ruby and it was extremely straight forward. The file upload was very easy. Instead of making a download script, we made one that would print out a URL to the file on S3 with an access token that granted 30 seconds from the time the token was generated. This was also very easy and straight forward.

I spent some time considering how we would use this and realized that if we have a list of resources that can be clicked on we don’t want to have to generate the S3 URL with the access token when the list is made. The reason for this is the aws-s3 gem requires a connection to generate the URL so apparently some kind of interaction is required with the S3 API and generating a lot of URLs for resources may be costly in time. A simple solution to this is to have the resource point to our server which then creates the S3 URL on demand and redirects to it.

Another Kata

I’m working on another kata. Doug had an interesting idea. I’ve been trying to figure out the right tools. It is taking longer than expected for that part and I know I should already be much further along. I really need to figure this out by the end of the day tomorrow.

The work on the internal project continues. We’re almost at a point where it is usable. This last iteration was the first one in which a story was not completed. It was disappointing but ultimately understandable. During the iteration we accomplished a few things that didn’t have a specific story but really needed to be done. One thing in specific was modifying how Jasmine was testing all of our Backbone.js code.

Jasmine can load HTML fixtures into a specific DOM area for use in tests. The default path for these fixtures is relative to the Jasmine installation. In our case, we use a lot of mustache templates. So we’d copy over the templates but changes in the “real” templates would have to be manually copied over. This was really bad because our tests were not useful. We modified Jasmine to point to the rails view directory and updated all the fixture inclusions to use the new paths and it worked out great!

In our project, Jasmine is installed at ./spec/javascript/ which contains the helpers and suport directory along with any other directories one might want to create. We added a file called backbone-views.js to the helpers directory:

Then a call to loadFixtures would look like:

It is a small thing but clearly became essential over time.

About a year ago I came across wkhtmltopdf and thought it sounded wonderful. There are many ways of creating PDFs however almost all are very painful. The idea with wkhtmltopdf is to make a PDF based on HTML including CSS and JavaScript. The program includes the full webkit rendering engine and has some tweaks to force page pagination.

When I came across it last, we already had a solution in place. It wasn’t pretty but it worked. For the internal project at 8th Light I looked at another alternative that was already in use: Prawn. The problem with Prawn is the same as the problem with most PDF generation techniques: you have to learn a new language or custom language and while it may allow you to have exact pixel output it also requires a substantial investment in time. Our designers already know CSS and HTML and we didn’t need pixel perfect output so I picked the PDFKit gem (another choice is wicked_pdf which I’d also like to try) and got to work with a sample mock up of HTML and CSS along with web fonts from our designer Chris.

The biggest problem I ran into was that wkhtmltopdf requires absolute paths for all the external resources. PDFKit has a method to load stylesheets that uses a relative path but for all the images, CSS and web fonts I’d need to use a view helper to write absolute URLs. With that problem solved I had a working PDF. Well, sort of… The version of wkhtmltopdf I’m using is an older release and it didn’t seem to like SVG web fonts. Thankfully, the SVG variant wasn’t needed so I was able to remove it from the HTML presented to wkhmtltopdf.

The generated PDF looks great! It includes the web fonts and the background images as desired. The sample logo definitely looks a bit fuzzier however it is low resolution so once we substitute a higher resolution one or use web fonts to create the logo we’ll have a great looking PDF in a minimal amount of time. There certainly are trade-offs with this approach but I have a hard time imagining doing anything else until how the PDF should look is completely nailed down.

Another iteration meeting on the internal project

We had another iteration meeting for the internal project. We completed all the stories successful for the iteration. There were one or two design changes related to stories but the client and the designer made a side deal to take care of the minor changes for 0 points in this iteration.

Our simple syncing approach is working well. There are some obvious ways to optimize it: update the API to have an option to only return data modified since a certain timestamp. All of the records we’re getting have Active Record timestamps which means they have created_at and updated_at on every record. It is easy however we’d need to make sure the API was time zone agnostic so perhaps require date times in GMT. For now though I think we made the right choice: there isn’t that much data to sync so brute force is fine.

Patrick presents

Patrick presented “Distributing Jobs And Data For System Scalability” as the final part of his apprentice challenge. It was a great presentation and a lot of fun. He incorporated audience participation as part of his presentation (as did the most recent two apprentice presentations by Stephanie and Wai Lee).

Patrick started a simple Rails site on his laptop that presented each person with a single character input box and a submit button. Then we all tried to spell out “helloworld” without a typo. On typo, the attempt restarted. It was very hard to do and even in later rounds when we were allowed to communicate with each other it still took a while. The optimal approach at that time would have been to give 10 people one letter each and have them go in order. That way they could pre-populate the input form and submit in order quickly. In any case, it was a lot of fun and an interesting presentation.

Today I paired with Mark to finish up the work on syncing (really a one way pull) of some data from one system to another system. It was mostly straight forward however we ran into some odd bugs and learned that .to_json takes a :method argument that specifies what methods should be run on the object being converted to JSON. Otherwise it ignores those methods and you get nil values which isn’t fun at all. Thanks to Kevin for pointing that out to us!

On Wednesday, I worked with Darius on the syncing between some internal projects. For our main project we want to retrieve some data from another system. That system has a wrapper API on it. The API needed to be expanded to return more types of data for the project we were working. So we needed to modify the remote system, modify the code behind the API on the remote system and then modify the gem that can be used to abstract connecting to the API via HTTP to retrieve records. Then we needed to use the gem to actually pull remote data.

There are a lot of steps here and a lot of places to go wrong. The gem that sits between the systems was creating it’s own Ruby classes. The initial version was very simple. A class would look similar to this:

That was fine but more complex records required more accessors and more line of boiler plate code to populate the object attributes. And that kind of code unless it is generated is a hassle. Did you forget one of 13 fields? Are you going to test every single field? It becomes a lot of work and it’s just something one is going to get wrong eventually and it is likely going to cause odd errors.

I refactored this code by creating a base class:

Now the Person class looks like this:

And the much larger objects are similar — just a list of attributes that are expected to come from the JSON request on one side and be populated into our object that is passed off to the gem consumer. And now it is easy for the gem consumer to call .attributes on the object and use that for populating its own class instance that needs this data.

Here is our example class:

And here is how we can mock the convert method and return one or more the arguments to convert:

Working on a Rails-based client site

Today I worked with Doug on a client site written in Rails. It was interesting to see a larger code base that makes more use of Rails features (helpers, etc) and testing those features. I wrote a script to import a bunch of data from a CSV file and wrote a migration to update the database schema to accommodate storage of the new data. I modifid the seed data generation script to be cleaner and generate more data so there is data present where additions where made in the schema.

Thoughts on ideal backend for Backbone.js

The problem with using an MVC backend for Backbone.js is that typically the controllers are DRY but also do things that are un-DRY in order to make things easier. For example, say you have an orders controller that pulls up order history. You’ll probably need more than just order data — you’ll likely need to join it a customer, product, address or some other table. The temptation which makes complete sense is to have that controller retrieve the related data even though it is out of the scope of that controller. There are of course ways of working around this in MVC frameworks but you’re still trying to shim another responsibility onto a controller action that is unrelated to it’s primary responsibility.

What is interesting about Backbone.js is that you can have it handle requesting completely unrelated data in order to build views without compromising the primary responsibility of your controllers in MVC. The most simplistic example of this that I can think of is needing to display multiple choices in a dropdown for a field on a form. The choices might be in one table and they are related to the model behind the form via a foreign key. This is a pain point in the typical MVC framework. The controller needs to load the model for the form but also the related choices from somewhere else…

The odd thing though is that if you take this approach with Backbone.js and have it be responsible for combining the model that is being edited along with the choices in the select list in a client-side view everything becomes very DRY. However it also makes one think that perhaps an MVC framework is immense overkill as a backend for Backbone.js.

What does Backbone.js really need on the backend?

Bootstrap: When you first hit a Backbone.js page the recommended approach is to serve an HTML page with the JSON-encoded data (collections of models) within that HTML. In other words the server takes all the data you initially need for your view and encodes it into a string that is put within the HTTP response to your browser. When Backbone.js loads on the client it converts that string into collections of models. The point is to decrease startup time by avoiding having the page load then having Backbone.js fire off requests for additional data. That way the application is fast and responsive.

Running: When a Backbone.js application is running in the browser it can be very economical in terms of communication with the server. It can also communicate in a way that is very DRY: each model/collection has a configured URL that the backend server can respond to and so each update can be posted separately keeping clean separation of concerns on the backend.

So what?

Well when using MVC it feels like the controllers are functional in the running stage but the bootstrap is painful. It also feels like during the running stage the controllers are immense overkill. During the bootstrap, all that work to remain DRY gets shot in the foot in the name of performance. That performance is critical but it also makes MVC feel like the wrong backend.

Going forward…

I’m still thinking about what the ideal backend would look like. I think it could be much simpler than an MVC framework. The controllers are much simpler as they are REST endpoints. Security is an important concern that has to remain in the forefront. But what would work well for both the bootstrap and the running phases? Hrm…

I spent the day expanding the API on an internal tool in order to support additional functionality that is required for another internal tool. This was interesting because the API is wrapped by a Ruby gem created with Jeweler. Ruby gems are small packages of code that have version numbers. Ruby on Rails is itself a gem and the typical Rails project relies on additional gems. I’d never created a gem before so it was an interesting experience.