Tuesday, March 31, 2009

Accessing Sinatra Helper Methods in View Specs

‹prev | My Chain | next›

I left off last night pondering how to make Sinatra helpers available to Haml view specs. I tried digging into how helpers works in Sinatra, but was unable to discern a way to put any of it to use.

My ultimate solution is to break out my helper methods into a file named helpers.rb:
module Eee
module Helpers
def hours(minutes)
h = minutes.to_i / 60
m = minutes.to_i % 60
h > 0 ? "#{h} hours" : "#{m} minutes"
end
end
end
Requiring that file and then mixing it in inside the Sinatra helpers block gets me back to where I was last night:
require 'helpers'

helpers do
include Eee::Helpers
end
In order to make the helper methods available to my view specs, I configure Spec::Runner to include the helper module:
Spec::Runner.configure do |config|
config.include Webrat::Matchers, :type => :views
config.include Eee::Helpers
end
Finally, I change the default scope of the Haml rendering from a vanilla Object.new to self:
def render(template_path)
template = File.read("./#{template_path.sub(/^\//, '')}")
engine = Haml::Engine.new(template)
@response = engine.render(self, assigns_for_template)
end
Binding the render call to self (instances of Test::Unit sub-classes) gives the Haml engine an object that defines the helper methods (because they were included in the configuration block).

With those changes, I have all of my views specs passing, including displaying 5 hours instead of 300 minutes:
cstrom@jaynestown:~/repos/eee-code$ ruby ./spec/views/recipe.haml_spec.rb -cfs 

recipe.haml
- should display the recipe's title

recipe.haml a recipe with no ingredient preparations
- should not render an ingredient preparations

recipe.haml a recipe with 1 egg
- should render ingredient names
- should render ingredient quantities
- should not render a brand

recipe.haml a recipe with 1 cup of all-purpose, unbleached flour
- should include the measurement unit
- should include the specific kind of ingredient
- should read conversationally, with the ingredient kind before the name

recipe.haml a recipe with 1 12 ounce bag of Nestle Tollhouse chocolate chips
- should include the ingredient brand
- should note the brand parenthetically after the name

recipe.haml a recipe with an active and inactive preparation time
- should include preparation time
- should include inactive time

recipe.haml a recipe with no inactive preparation time
- should not include inactive time

recipe.haml a recipe with 300 minutes of inactive time
- should display 5 hours of Inactive Time

Finished in 0.063692 seconds

14 examples, 0 failures
(commit)

I may want to simply stub my helper method calls inside my view specs and test my helper methods independently of my views. Something to decide another day.

Monday, March 30, 2009

Implementing Recipe Details, Part 2

‹prev | My Chain | next›

Thanks to the Recipe Details feature, I do no lack for next steps of things to do in my chain. Next up, "Viewing a recipe with non-active prep time":
cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_details.feature -n \
-s "Viewing a recipe with non-active prep time"
Feature: Recipe Details

So that I can accurately reproduce a recipe at home
As a web user
I want to be able to easily recognize important details
Scenario: Viewing a recipe with non-active prep time
Given a recipe for Crockpot Lentil Andouille Soup
When I view the recipe
Then I should see 15 minutes of prep time
And I should see that it requires 5 hours of non-active cook time


1 scenario
1 step skipped
3 steps pending (3 with no step definition)

You can use these snippets to implement pending steps which have no step definition:

Given /^a recipe for Crockpot Lentil Andouille Soup$/ do
end

Then /^I should see 15 minutes of prep time$/ do
end

Then /^I should see that it requires 5 hours of non\-active cook time$/ do
end
To implement the Given step, I cut & paste the previous scenario's Given.

Yeah, that's right. I cut&paste. I will almost certainly DRY this up in a little bit, but I prefer repetition and wordiness over indirection in my specs. The easier it is to readily see what is going on in the spec, the better.

In the new Given step, the recipe will need preparation and inactive time:
  recipe = {
:title => @title,
:date => @date,
:inactive_time => 300,
:prep_time => 15
}
(commit)

With the Given and When steps passing (the latter re-using the same When step from the previous scenario), it is time to move back inside to work with view specs.

Implementation of the first several specs goes smoothly:
recipe.haml a recipe with an active and inactive preparation time
- should include preparation time
- should include inactive time

recipe.haml a recipe with no inactive preparation time
- should not include inactive time
I do run into a problem trying to implement the inner workings needed to satisfy the last step, "I should see that it requires 5 hours of non-active cook time". The view specification that I use is:
  context "a recipe with 300 minutes of inactive time" do
before(:each) do
@recipe['inactive_time'] = 300
render("views/recipe.haml")
end
it "should display 5 hours of Inactive Time" do
response.should contain(/Inactive Time: 5 hours/)
end
To convert from 300 minutes to 5 hours, I defined the following helper in the main Sinatra application:
helpers do 
def hours(minutes)
h = minutes.to_i / 60
m = minutes.to_i % 60
h > 0 ? "#{h} hours" : "#{m} minutes"
end
end
I then use that helper in the Haml template as:
.eee-recipe-meta
%div= "Preparation Time: " + hours(@recipe['prep_time'])
- if @recipe['inactive_time']
%div= "Inactive Time: " + hours(@recipe['inactive_time'])
But when I run the view specs, every spec fails with an undefined method `hours' for #<Object:0xb7295de4>.

I am able to get everything working by moving the hours method outside of the helpers block:
cstrom@jaynestown:~/repos/eee-code$ ruby ./spec/views/recipe.haml_spec.rb -cfs 

recipe.haml
- should display the recipe's title

recipe.haml a recipe with no ingredient preparations
- should not render an ingredient preparations

recipe.haml a recipe with 1 egg
- should render ingredient names
- should render ingredient quantities
- should not render a brand

recipe.haml a recipe with 1 cup of all-purpose, unbleached flour
- should include the measurement unit
- should include the specific kind of ingredient
- should read conversationally, with the ingredient kind before the name

recipe.haml a recipe with 1 12 ounce bag of Nestle Tollhouse chocolate chips
- should include the ingredient brand
- should note the brand parenthetically after the name

recipe.haml a recipe with an active and inactive preparation time
- should include preparation time
- should include inactive time

recipe.haml a recipe with no inactive preparation time
- should not include inactive time

recipe.haml a recipe with 300 minutes of inactive time
- should display 5 hours of Inactive Time

Finished in 0.060968 seconds

14 examples, 0 failures
It does not sit well with me to put the helper outside the helper block, so tomorrow, I will likely try to figure out how to make Sinatra helpers available to Haml view specs. But this is a good stopping point for tonight.
(commit)

Sunday, March 29, 2009

Finishing Working Inside the Recipe Details, Moving Back Out

‹prev | My Chain | next›

Yesterday, I got in the flow of outside-in Behavior Driven Development. I reached a decidedly inside stopping point driving the development of my Haml views—even finding and addressing a boundary condition.

Work today is smoother as I add four more examples describing various display attributes. At this point, my specification describes the recipe.haml template as:
cstrom@jaynestown:~/repos/eee-code$ ruby ./spec/views/recipe.haml_spec.rb -cfs 

recipe.haml
- should display the recipe's title

recipe.haml a recipe with no ingredient preparations
- should not render an ingredient preparations

recipe.haml a recipe with 1 egg
- should render ingredient names
- should render ingredient quantities

recipe.haml a recipe with 1 cup of all-purpose, unbleached flour
- should include the measurement unit
- should include the specific kind of ingredient

recipe.haml a recipe with 1 12 ounce bag of Nestle Tollhouse chocolate chips
- should include the ingredient brand

Finished in 0.07205 seconds

7 examples, 0 failures
(commit)

I believe that I have completed all of the necessary implementation for the "Viewing a recipe with several ingredients" scenario. To remind myself of where I left off in the "outside" scenarios:
cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_details.feature -n -s "Viewing a recipe with several ingredients"
Feature: Recipe Details

So that I can accurately reproduce a recipe at home
As a web user
I want to be able to easily recognize important details
Scenario: Viewing a recipe with several ingredients
Given a recipe for Buttermilk Chocolate Chip Pancakes
When I view the recipe
Then I should see an ingredient of "1 cup of all-purpose, unbleached flour"
expected the following element's content to include "1 cup of all-purpose, unbleached flour":

Buttermilk Chocolate Chip Pancakes (Spec::Expectations::ExpectationNotMetError)
./features/step_definitions/recipe_details.rb:17:in `Then /^I should see an ingredient of "(.+)"$/'
features/recipe_details.feature:11:in `Then I should see an ingredient of "1 cup of all-purpose, unbleached flour"'
And I should see an ingredient of "¼ teaspoons salt"
And I should see an ingredient of "chocolate chips (Nestle Tollhouse)"

1 scenario
2 steps passed
1 step failed
2 steps skipped
The scenario is failing because it does not build any ingredient preparations. Adding them, and then training the "I should seen an ingredient" step to ignore whitespace, I still get a failing scenario:
cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_details.feature -n -s "Viewing a recipe with several ingredients"
Feature: Recipe Details

So that I can accurately reproduce a recipe at home
As a web user
I want to be able to easily recognize important details
Scenario: Viewing a recipe with several ingredients
Given a recipe for Buttermilk Chocolate Chip Pancakes
When I view the recipe
Then I should see an ingredient of "1 cup all-purpose, unbleached flour"
expected the following element's content to match /1\s+cup\s+all-purpose,\s+unbleached\s+flour/:

Buttermilk Chocolate Chip Pancakes
1
cup
flour
all-purpose, unbleached (Spec::Expectations::ExpectationNotMetError)
./features/step_definitions/recipe_details.rb:33:in `Then /^I should see an ingredient of "(.+)"$/'
features/recipe_details.feature:11:in `Then I should see an ingredient of "1 cup all-purpose, unbleached flour"'
And I should see an ingredient of "¼ teaspoons salt"
And I should see an ingredient of "chocolate chips (Nestle Tollhouse)"


1 scenario
2 steps passed
1 step failed
2 steps skipped
This is failing because the conversationally written spec (an ingredient of "1 cup all-purpose, unbleached flour") is not matching up with the order of implementation. As implemented, the ingredient name ("flour") comes before the kind of ingredient ("all-purpose, unbleached"), but the scenario expects them to be reversed. For ease of reading, conversationally written should win.

Back into the view specs...

Although it is not quite xpath, I need to rely on some gnarly CSS3 to specify that ingredient "kind" comes before the "name": .ingredient > .kind + .name. The whole example is:
    it "should read conversationally, with the ingredient kind before the name" do
response.should have_selector(".preparations") do |preparations|
preparations.
should have_selector(".ingredient > .kind + .name",
:content => 'flour')
end
end
Making the spec pass is a simple matter of re-arranging the Haml template.
(commit)

While I was in the view, I also took some time to arrange the brand (e.g. Nestle Tollhouse) in the ingredient listing.
(commit)

Moving back out to the Cucumber scenario (and adding the requisite ingredient preparations), I now have the entire scenario passing:
cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_details.feature -n -s "Viewing a recipe with several ingredients"
Feature: Recipe Details

So that I can accurately reproduce a recipe at home
As a web user
I want to be able to easily recognize important details
Scenario: Viewing a recipe with several ingredients
Given a recipe for Buttermilk Chocolate Chip Pancakes
When I view the recipe
Then I should see an ingredient of "1 cup all-purpose, unbleached flour"
And I should see an ingredient of "¼ teaspoons salt"
And I should see an ingredient of "chocolate chips (Nestle Tollhouse)"


1 scenario
5 steps passed
(commit)

Saturday, March 28, 2009

Inside with View Specs

‹prev | My Chain | next›

Cleanup for Cross-Project Consistency

Reading through the RSpec Beta book, I noticed that view specs are rendered without the initial slash (e.g. render("views/recipe.haml")). To prevent confusion when switching between projects, I modify the my Sinatra spec helper to work with or without the initial slash:
def render(template_path)
template = File.read("./#{template_path.sub(/^\//, '')}")
engine = Haml::Engine.new(template)
@response = engine.render(Object.new, assigns_for_template)
end
(commit)

So now, it's back to view spec'ing...

Specifying the Ingredients Display

We already have the recipe title showing, next up is displaying ingredients. On the recipe page, there should be a section listing the ingredients, with simple preparation instructions (e.g. diced, minced, 1 cup measured, etc). This can be expressed in RSpec as:
  it "should render ingredient names" do
render("views/recipe.haml")
response.should have_selector(".preparations") do |preparations|
prepartions.
should have_selector(".ingredient > .name", :content => 'egg')
end
end
Running this spec fails because the needed HTML document structure is not in place:
strom@jaynestown:~/repos/eee-code$ ruby ./spec/views/recipe.haml_spec.rb
.F

1)
'recipe.haml should render ingredient names' FAILED
expected following output to contain a <.preparations/> tag:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body><h1>
Recipe Title
</h1></body></html>
./spec/views/recipe.haml_spec.rb:27:
./spec/views/recipe.haml_spec.rb:3:

Finished in 0.010398 seconds

2 examples, 1 failure
Implementing that document structure in Haml is pretty darn easy:
%h1
= @recipe['title']

%ul.preparations
%li.ingredient
%span.name
egg
Running the spec now passes:
cstrom@jaynestown:~/repos/eee-code$ ruby ./spec/views/recipe.haml_spec.rb
..

Finished in 0.01163 seconds

2 examples, 0 failures
A template that only includes a single ingredient of "egg" is not going to be all that useful. To get the ingredient name into the HTML output, we need to build up the @recipe data structure in the example. The ingredient preparation data structures in our legacy system are somewhat complex. For example, 10 ounces of frozen spinach defrosted in the microwave is represented as:
       {
"quantity": 10,
"ingredient": {
"name": "spinach",
"kind": "frozen"
},
"unit": "ounces",
"description": "cooked in microwave"
}
That level of complexity is not needed to specify that an ingredient name exists. Instead use an egg preparation of:
  @recipe['preparations'] =
[ 'quantity' => 1, 'ingredient' => { 'name' => 'egg' } ]
Updating the Haml template to display a recipe's ingredients is easy:
%h1
= @recipe['title']

%ul.preparations
- @recipe['preparations'].each do |preparation|
%li.ingredient
%span.name
= preparation['ingredient']['name']
(commit)

Uncover a Bug? Don't Fix It... Write a Spec!

Running the whole spec, however, causes a failure in the first example!
cstrom@jaynestown:~/repos/eee-code$ ruby ./spec/views/recipe.haml_spec.rb
F.

1)
NoMethodError in 'recipe.haml should display the recipe's title'
undefined method `each' for nil:NilClass
(haml):5:in `render'
/home/cstrom/.gem/ruby/1.8/gems/haml-2.0.9/lib/haml/engine.rb:149:in `render'
/home/cstrom/.gem/ruby/1.8/gems/haml-2.0.9/lib/haml/engine.rb:149:in `instance_eval'
/home/cstrom/.gem/ruby/1.8/gems/haml-2.0.9/lib/haml/engine.rb:149:in `render'
/home/cstrom/repos/eee-code/spec/spec_helper.rb:20:in `render'
./spec/views/recipe.haml_spec.rb:11:
./spec/views/recipe.haml_spec.rb:3:

Finished in 0.011262 seconds

2 examples, 1 failure
Ah, there are no preparation instructions in the first example. When a recipe includes no preparation / ingredients, nothing should be displayed:
  context "no ingredient preparations" do
before(:each) do
@recipe[:preparations] = nil
end

it "should not render an ingredient preparations" do
render("views/recipe.haml")
response.should_not have_selector(".preparations")
end
end
I like the use of context here. In describing the example above, I used the word "When", which suggests a specific context in which the spec runs. It also aids in readability. Even in an age in which monitors are 1600 pixels wide, the less horizontal scanning needed, the easier it is to read the core concept of the code / spec.

I can make this pass with a conditional in the Haml template:
%h1
= @recipe['title']

- if @recipe['preparations']
%ul.preparations
- @recipe['preparations'].each do |preparation|
%li.ingredient
%span.name
= preparation['ingredient']['name']
Making this spec page also resolves the problem in the original spec:
cstrom@jaynestown:~/repos/eee-code$ ruby ./spec/views/recipe.haml_spec.rb -cfs 

recipe.haml
- should display the recipe's title
- should render ingredient names

recipe.haml a recipe with no ingredient preparations
- should not render an ingredient preparations

Finished in 0.015988 seconds

3 examples, 0 failures
It would have been a mistake to attempt to fix the first spec failure directly by adding the conditional to the Haml template. The first spec was meant to test something very specific, if very simple. It happened to uncover a boundary condition. Fixing the boundary condition issue in that spec would have left the boundary condition uncovered. It also would not have given me the opportunity to codify my thinking in resolving the issue.

As it is, I am in a much better place now. I still have my original, simple spec which may yet uncover other boundary condition defects. I also have a specification of how I handle this specific boundary condition.
(commit)

Friday, March 27, 2009

RSpec with Sinatra & HAML

‹prev | My Chain | next›

Last night, I was able to get HAML working together with Sinatra and CouchDB. Unfortunately, I got stuck trying to spec HAML views independently of the Sinatra application. I have grown accustomed to the cushy life afforded by rspec-on-rails and need to be able to do similar things with HAML.

The assigns[] and render methods are nowhere to be found—so how do I spec the views? The answer is to instantiate a Haml::Engine object and manually invoke its render method:
require File.expand_path(File.dirname(__FILE__) + '/../spec_helper' )
require 'haml'

describe "recipe.haml" do
before(:each) do
@title = "Recipe Title"
@recipe = { 'title' => @title }

template = File.read("./views/recipe.haml")
@engine = Haml::Engine.new(template)
end

it "should display the recipe's title" do
response = @engine.render(Object.new, :@recipe => @recipe)
response.should have_selector("h1", :content => @title)
end
end
The call to render requires two arguments as documented in the Haml API. The first is the scope in which the the template is evaluated. It is useful if you want to bind an object whose methods can be evaluated by the HAML template. Since I have no need for this, I simply pass in a new, top-level Object instance (which is the default).

The second argument to render is a hash assignment of local variables. In this case, I want the template to see an instance variable @recipe and assign it the value of the @recipe variable defined in the before(:each). To do this, pass in a single-record hash with a key of :@recipe and value of the before(:each)'s @recipe instance variable.

With that in place, the spec passes:
cstrom@jaynestown:~/repos/eee-code$ ruby ./spec/views/recipe.haml_spec.rb 
.

Finished in 0.008689 seconds

1 example, 0 failures
(commit, commit)

It is working, but I would much prefer to keep the Haml::Engine overhead out of my view specs. In other words, I would like to have these looking more like normal view specs:
require File.expand_path(File.dirname(__FILE__) + '/../spec_helper' )

describe "recipe.haml" do
before(:each) do
@title = "Recipe Title"
@recipe = { 'title' => @title }

assigns[:recipe] = @recipe
end

it "should display the recipe's title" do
render("/views/recipe.haml")
response.should have_selector("h1", :content => @title)
end
end
That turns out to be not nearly as difficult as I expected it would be. Here is the updated spec_helper.rb that allows the above spec to pass (comments inline):
ENV['RACK_ENV'] = 'test'

require 'eee'
require 'spec'
require 'spec/interop/test'
require 'sinatra/test'

require 'webrat'
require 'haml'

Spec::Runner.configure do |config|
config.include Webrat::Matchers, :type => :views
end

# Renders the supplied template with Haml::Engine and assigns the
# @response instance variable
def render(template)
template = File.read(".#{template}")
engine = Haml::Engine.new(template)
@response = engine.render(Object.new, assigns_for_template)
end

# Convenience method to access the @response instance variable set in
# the render call
def response
@response
end

# Sets the local variables that will be accessible in the HAML
# template
def assigns
@assigns ||= { }
end

# Prepends the assigns keywords with an "@" so that they will be
# instance variables when the template is rendered.
def assigns_for_template
assigns.inject({}) do |memo, kv|
memo["@#{kv[0].to_s}".to_sym] = kv[1]
memo
end
end
I am fairly pleased with my poor-man's rspec-for-haml-views implementation—mostly in that it actually works. It is not as robust as it would need to be for a gem release—when you neglect to set the proper assigns, you tend to get errors along the lines of:
undefined method `[]' for nil:NilClass
(haml):2:in `render'
Still, it is good enough for now. More importantly it gave me a chance to explore how these things are typically implemented—good knowledge to have in my toolbelt.
(commit)

Thursday, March 26, 2009

Oragnization for RSpec, HAML, and Sinatra

‹prev | My Chain | next›

Since I am learning during my chain, I may as well give a new templating system a try—specifically HAML. While doing do, I am going to follow the advice of the RSpec book and spec my views.

First up, though I need to do a little house cleaning from last night. I had added my RSpec configuration into my main Sinatra spec file. To keep my specs organized, I need a spec/spec_helper.rb file:
ENV['RACK_ENV'] = 'test'

require 'eee'
require 'spec'
require 'spec/interop/test'
require 'sinatra/test'

require 'webrat'

Spec::Runner.configure do |config|
config.include Webrat::Matchers, :type => :views
end
(commit, commit)

Next up, I replace the raw HTML in the get '/recipes/:permalink' Sinatra block with HAML:
get '/recipes/:permalink' do
data = RestClient.get "#{@@db}/#{params[:permalink]}"
result = JSON.parse(data)

haml "%h1 #{result['title']}"
end
Re-running my specs verifies that all is OK.

Lastly, it is time to move the inline HAML into a standalone template. Since this application is read-only, I will name the view views/recipe.haml (as opposed to something with "show" in the filename) with the following contents:
%h1
= @recipe['title']
To set the @recipe instance variable and call the new template, update the get '/recipes/:permalink' Sinatra block to read:
get '/recipes/:permalink' do
data = RestClient.get "#{@@db}/#{params[:permalink]}"
@recipe = JSON.parse(data)

haml :recipe
end
Re-running last night's spec verifies that the changes have not broken the limited functionality that we have so far (good time to commit).

At this point, I realize that I was specifying view behavior in my previous RSpec test:
  it "should include a title" do
get "/recipes/#{@permalink}"
response.should be_ok
response.should have_selector("h1", :content => @title)
end
So I will pull the have_selector matcher into a view spec instead, something like:
require File.expand_path(File.dirname(__FILE__) + '/../spec_helper' )

describe "recipe.haml" do
before(:each) do
@title = "Recipe Title"
@recipe = { :title => @title }
assigns[:recipe] = @recipe
end

it "should display the recipe's title" do
render "recipe.haml"
response.should have_selector("h1", :content => @title)
end
end
Unfortunately, this does not work:
cstrom@jaynestown:~/repos/eee-code$ ruby ./spec/views/recipe.haml_spec.rb
F

1)
NameError in 'recipe.haml should display the recipe's title'
undefined local variable or method `assigns' for #
./spec/views/recipe.haml_spec.rb:7:
./spec/views/recipe.haml_spec.rb:3:

Finished in 0.023361 seconds

1 example, 1 failure
I believe that the assigns method (and the render method) are part of rspec-on-rails rather than rspec proper. So I will have to figure out some way to render HAML templates directly inside RSpec tests. Tomorrow.

Wednesday, March 25, 2009

RSpec with Sinatra & CouchDB

‹prev | My Chain | next›

I left off last night moving into the guts of the application. The plan was to start BDDing with RSpec. It occurred to me, however, that I had no idea how to do it. Happily, Sinatra's testing documentation includes RSpec information.

Based on that documentation, I create spec/eee_spec.rb:
require 'eee'
require 'spec'
require 'spec/interop/test'
require 'sinatra/test'

require 'webrat'
Spec::Runner.configure do |config|
config.include Webrat::Matchers, :type => :views
end

describe 'GET /recipes' do
include Sinatra::Test

before(:all) do
RestClient.put @@db, { }
end

after(:all) do
RestClient.delete @@db
end

before (:each) do
@title = "Recipe Title"
end

it "should include a title" do
get '/recipes/test-recipe'
response.should be_ok
response.should have_selector("h1", :content => @title)
end
end
One addition in there is the inclusion of webrat matchers (because they're so darn nice). I have also included CouchDB build & teardown before(:all) and after(:all) blocks (based on the Cucumber Before and After blocks). Finally, I add a simple test for a title tag (using a Webrat have_selector matcher).

Since I have yet to add anything to the recipes action, I fully expect this example to fail when run:
cstrom@jaynestown:~/repos/eee-code$ ruby ./spec/eee_spec.rb 
F

1)
RestClient::RequestFailed in 'GET /recipes before(:all)'
HTTP status code 412
/home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:144:in `process_result'
/home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:106:in `transmit'
/home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:103:in `transmit'
/home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:36:in `execute_inner'
/home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:28:in `execute'
/home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:12:in `execute'
/home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient.rb:65:in `put'
./spec/eee_spec.rb:15:
./spec/eee_spec.rb:11:

Finished in 0.028201 seconds

0 examples, 1 failure
Indeed it does fail—just not the failure I had expected. Instead of a non-matching text failure, I get a RestClient failure. Experience has taught me to look in the couch.log almost immediately after seeing these and I am not disappointed this time:
[info] [<0.10798.5>] 127.0.0.1 - - 'PUT' /eee 412
Ah, just as with Cucumber, the Sinatra configure :test blocks are not being honored when running RSpec tests. Adding ENV['RACK_ENV'] = 'test' to the top of the eee_spec.rb file (before the require 'eee' statement) resolves that problem:
cstrom@jaynestown:~/repos/eee-code$ ruby ./spec/eee_spec.rb
F

1)
'GET /recipes should include a title' FAILED
expected following output to contain a <h1>Recipe Title</h1> tag:


./spec/eee_spec.rb:31:
./spec/eee_spec.rb:13:

Finished in 0.039395 seconds

1 example, 1 failure
Changing the get '/recipes/:permalink' block to:
get '/recipes/:permalink' do
"<h1>Recipe Title</h1>"
end
resolves the failure.

After some more red-green-refactoring, I end up with the following spec:
  before (:each) do
@date = Date.today
@title = "Recipe Title"
@permalink = @date.to_s + "-" + @title.downcase.gsub(/\W/, '-')

RestClient.put "#{@@db}/#{@permalink}",
{ :title => @title,
:date => @date }.to_json,
:content_type => 'application/json'
end

it "should include a title" do
get "/recipes/#{@permalink}"
response.should be_ok
response.should have_selector("h1", :content => @title)
end
This passes when run against the following get '/recipes/:permalink' block:
get '/recipes/:permalink' do
data = RestClient.get "#{@@db}/#{params[:permalink]}"
result = JSON.parse(data)

"<h1>#{result['title']}</h1>"
end
(tonight's commit of this code)

Tomorrow night: HAML (I think) and more progress on building the innards needed to implement the first Recipe Details scenario.

Tuesday, March 24, 2009

Implementing Recipe Details, Part 1

‹prev | My Chain | next›

Having cleared out the first spec failures last night, I get started implementing the first scenario in the Recipe Details Cucumber story:
Feature: Recipe Details

So that I can accurately reproduce a recipe at home
As a web user
I want to be able to easily recognize important details

Scenario: Viewing a recipe with several ingredients

Given a recipe for Buttermilk Chocolate Chip Pancakes
When I view the recipe
Then I should see an ingredient of "1 cup of all-purpose, unbleached flour"
And I should see an ingredient of "¼ teaspoons salt"
And I should see an ingredient of "chocolate chips (Nestle Tollhouse)"
Mostly following along with what I discovered in my Cucumber / Sinatra spike, I implement the Given and When steps as:
Given /^a recipe for Buttermilk Chocolate Chip Pancakes$/ do
@date = Date.new(2009, 03, 24)
@title = "Buttermilk Chocolate Chip Pancakes"
@permalink = @date.to_s + "-" + @title.downcase.gsub(/\W/, '-')

RestClient.put "#{@@db}/#{@permalink}",
{ :title => @title,
:date => @date }.to_json,
:content_type => 'application/json'
end

When /^I view the recipe$/ do
visit("/recipes/#{@permalink}")
end
The next pending step is reported as:
You can use these snippets to implement pending steps which have no step definition:

Then /^I should see an ingredient of "1 cup of all\-purpose, unbleached flour"$/ do
end
Implementing this as a simple response-should-contain:
Then /^I should see an ingredient of "(.+)"$/ do |ingredient|
response.should contain(ingredient)
end
When this is run, I get the following error:
    Then I should see an ingredient of "1 cup of all-purpose, unbleached flour"
expected the following element's content to include "1 cup of all-purpose, unbleached flour":
Not Found (Spec::Expectations::ExpectationNotMetError)
./features/step_definitions/recipe_details.rb:17:in `Then /^I should see an ingredient of "(.+)"$/'
features/recipe_details.feature:11:in `Then I should see an ingredient of "1 cup of all-purpose, unbleached flour"'
This simply means that the requested resource has not been defined in the Sinatra app. To get rid of the warning, define an empty resource:
get '/recipes/:permalink' do
end
Re-running the spec changes the error to:
    Then I should see an ingredient of "1 cup of all-purpose, unbleached flour"
expected the following element's content to include "1 cup of all-purpose, unbleached flour": (Spec::Expectations::ExpectationNotMetError)
./features/step_definitions/recipe_details.rb:17:in `Then /^I should see an ingredient of "(.+)"$/'
features/recipe_details.feature:11:in `Then I should see an ingredient of "1 cup of all-purpose, unbleached flour"'
We have gone from a Not Found error to an expectation not met error. Progress!

At this point, I need to move from the outside in by dropping down into RSpec. Tomorrow...

Monday, March 23, 2009

Proper Cucumber Sinatra Driving

‹prev | My Chain | next›

During my Cucumber / Sinatra spike, I ended up driving the development / test / production database URLs with a method defined right in the main body of the Sinatra app:
#####
# Use a different CouchDB instance in test mode
def db
Sinatra::Application.environment == :test ?
"http://localhost:5984/eee-test" :
"http://localhost:5984/eee-meals"
end
While reading through the Sinatra documentation, I came across the configure :environment method, which seems to be the Sinatra idiomatic way to do this. So I add the following to the empty eee.rb file from yesterday:
require 'rubygems'
require 'sinatra'
require 'rest_client'
require 'json'

configure :test do
@@db = "http://localhost:5984/eee-test"
end

configure :development, :production do
@@db = "http://localhost:5984/eee"
end
Following along with the spike, I add the following to features/support/env.rb:
Before do
RestClient.put @@db, { }
end

After do
RestClient.delete @@db
end
Running cucumber features/recipe_details.feature, however, gives this:
Feature: Recipe Details  # features/recipe_details.feature

So that I can accurately reproduce a recipe at home
As a web user
I want to be able to easily recognize important details
Scenario: Viewing a recipe with several ingredients # features/recipe_details.feature:7
/home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:144:in `process_result': HTTP status code 412 (RestClient::RequestFailed)
from /home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:106:in `transmit'
from /usr/lib/ruby/1.8/net/http.rb:543:in `start'
from /home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:103:in `transmit'
from /home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:36:in `execute_inner'
from /home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:28:in `execute'
from /home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient/request.rb:12:in `execute'
from /home/cstrom/.gem/ruby/1.8/gems/rest-client-0.9.2/lib/restclient.rb:65:in `put'
from ./features/support/env.rb:22:in `__cucumber_-610192688'
...
If RestClient is failing, it is time to check couch.log, which shows:
[info] [<0.2824.0>] 127.0.0.1 - - 'PUT' /eee 412
Hunh? PUTing /eee? But that is the development DB (which already exists, hence the 412). So what's up?

It turns out the configure blocks are evaluated during the require at the beginning of features/support/env.rb. Adding ENV['RACK_ENV'] = 'test' to the top of that file:
ENV['RACK_ENV'] = 'test'

# NOTE: This must come before the require 'webrat', otherwise
# sinatra will look in the wrong place for its views.
require File.dirname(__FILE__) + '/../../eee'
resolves the issue:
cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_details.feature -n
Feature: Recipe Details

So that I can accurately reproduce a recipe at home
As a web user
I want to be able to easily recognize important details
Scenario: Viewing a recipe with several ingredients
Given a recipe for Buttermilk Chocolate Chip Pancakes
When I view the recipe
Then I should see an ingredient of "1 cup of all-purpose, unbleached flour"
And I should see an ingredient of "¼ teaspoons salt"
And I should see an ingredient of "chocolate chips (Nestle Tollhouse)"

Scenario: Viewing a recipe with non-active prep time
Given a recipe for Crockpot Lentil Andouille Soup
When I view the recipe
Then I should see 15 minutes of prep time
And I should see that it requires 5 hours of non-active cook time

Scenario: Viewing a list of tools used to prepare the recipe
Given a recipe for Chicken Noodle Soup
When I view the recipe
Then I should see that it requires a bowl, a colander, a cutting board, a pot and a skimmer to prepare

Scenario: Main site categories
Given a recipe for Mango and Tomato Salad
And site-wide categories of Italian, Asian, Latin, Breakfast, Chicken, Fish, Meat, Salad, and Vegetarian
When I view the recipe
Then the Salad and Vegetarian categories should be active


4 scenarios
16 steps pending (16 with no step definition)

You can use these snippets to implement pending steps which have no step definition:
...
I am not certain that is much better than original inline method, but I will stick with idiomatic Sinatra wherever possible.

Time to get started on implementing those steps...

Sunday, March 22, 2009

First Steps Toward Implementation

‹prev | My Chain | next›

Having spiked both Merb and Sinatra, I prefer Sinatra for reimplementing EEE Cooks on CouchDB. It keeps me close to the metal. I am not planning on adding any editing features that might benefit from an ORM like DataMapper. Getting the spike running for Sinatra was at least as easy as it was for Merb. All of which makes Sinatra the right choice for me.

I have three features drafted, so I will get started with the recipe details feature. First up, I need to add the env.rb Cucumber support file. The standard Cucumber file works just fine now (I had to muck with it some during the spike):
# NOTE: This must come before the require 'webrat', otherwise
# sinatra will look in the wrong place for its views.
require File.dirname(__FILE__) + '/../../eee'

# RSpec matchers
require 'spec/expectations'

# Webrat
require 'webrat'
Webrat.configure do |config|
config.mode = :sinatra
end

World do
session = Webrat::SinatraSession.new
session.extend(Webrat::Matchers)
session.extend(Webrat::HaveTagMatcher)
session
end
Running the story with cucumber features/recipe_details.feature, I get:
/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- ./features/support/../../eee (LoadError)
Failed to load features/support/env.rb
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from ./features/support/env.rb:3
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from /home/cstrom/.gem/ruby/1.8/gems/cucumber-0.1.16/bin/../lib/cucumber/cli.rb:227:in `require_files'
from /home/cstrom/.gem/ruby/1.8/gems/cucumber-0.1.16/bin/../lib/cucumber/cli.rb:225:in `each'
from /home/cstrom/.gem/ruby/1.8/gems/cucumber-0.1.16/bin/../lib/cucumber/cli.rb:225:in `require_files'
from /home/cstrom/.gem/ruby/1.8/gems/cucumber-0.1.16/bin/../lib/cucumber/cli.rb:148:in `execute!'
from /home/cstrom/.gem/ruby/1.8/gems/cucumber-0.1.16/bin/../lib/cucumber/cli.rb:13:in `execute'
from /home/cstrom/.gem/ruby/1.8/gems/cucumber-0.1.16/bin/cucumber:6
from /home/cstrom/.gem/ruby/1.8/bin/cucumber:19:in `load'
from /home/cstrom/.gem/ruby/1.8/bin/cucumber:19
To address the error message, I simply need to create my sinatra code file
cstrom@jaynestown:~/repos/eee-code$ touch eee.rb
Re-running the story, I find that I need to implement some steps:
cstrom@jaynestown:~/repos/eee-code$ cucumber features/recipe_details.feature 
Feature: Recipe Details # features/recipe_details.feature

So that I can accurately reproduce a recipe at home
As a web user
I want to be able to easily recognize important details
Scenario: Viewing a recipe with several ingredients # features/recipe_details.feature:7
Given a recipe for Buttermilk Chocolate Chip Pancakes # features/recipe_details.feature:9
When I view the recipe # features/recipe_details.feature:10
Then I should see an ingredient of "1 cup of all-purpose, unbleached flour" # features/recipe_details.feature:11
And I should see an ingredient of "¼ teaspoons salt" # features/recipe_details.feature:12
And I should see an ingredient of "chocolate chips (Nestle Tollhouse)" # features/recipe_details.feature:13
Next up, I need to create a test DB into which I can dump test data. I did that with a ternary operator in my spike, but would like to get it into Sinatra configuration this time around. Something to figure out tomorrow.

Saturday, March 21, 2009

Recipe Search Feature

‹prev | My Chain | next›

A very full day of real life intruding, so I keep my chain simple tonight by adding a first draft of a recipe searching feature. Again, I am not describing a new feature, but putting an existing feature (advanced searching) into story terms.

The story:
Feature: Search for recipes

So that I can find one recipe among many
As a web user
I want to be able search recipes

Scenario: Matching a word in the ingredient list in full recipe search

Given a "pancake" recipe with "chocolate chips" in it
And a "french toast" recipe with "eggs" in it
When I search for "chocolate"
Then I should see the "pancake" recipe in the search results
And I should not see the "french toast" recipe in the search results

Scenario: Matching a word in the recipe summary

Given a "pancake" recipe with a "Yummy!" summary
And a "french toast" recipe with a "Delicious" summary
When I search for "yummy"
Then I should see the "pancake" recipe in the search results
And I should not see the "french toast" recipe in the search results

Scenario: Matching a word stem in the recipe instructions

Given a "pancake" recipe with instructions "mixing together dry ingredients"
And a "french toast" recipe with instructions "whisking the eggs"
When I search for "whisk"
Then I should not see the "pancake" recipe in the search results
And I should see the "french toast" recipe in the search results

Scenario: Searching titles

Given a "pancake" recipe
And a "french toast" recipe with a summary of "not a pancake"
When I search titles for "pancake"
Then I should see the "pancake" recipe in the search results
And I should not see the "french toast" recipe in the search results

Scenario: Searching ingredients

Given a "pancake" recipe with "chocolate chips" in it
And a "french toast" recipe with eggs in it and a summary of "does not go well with chocolate"
When I search ingredients for "chocolate"
Then I should see the "pancake" recipe in the search results
And I should not see the "french toast" recipe in the search results
The actual commit.

Friday, March 20, 2009

VPS Notes

‹prev | My Chain | next›

I think I have all of my exploratory code / prototyping done at this point.

Rather than jump directly in real code tonight, I will gather some VPS hosting data for the ultimate deployment.

I am currently running EEE Cooks on a shared hosting site with Hosting Rails. I have had absolutely no problems with them and their service, but they are not going to allow me to run a CouchDB instance in their shared environment.

Hosting Rails does offer a VPS solution, so I will certainly consider them here. Along with them, I will also look at Slicehost and Linode.

Hosting Rails VM256Linode 360Slicehost 256 slice
Cost$17.92$19.95$20
Memory256360256
Disk Space (GB)201210
Bandwidth200200100


Aside from those variables, the three options are pretty much the same. All offer Debian & Ubuntu (the only distributions that I would manage). All offer root access. All offer a dedicated IP address.

So all I have to do is decide which of these variables (if any) really matters and maybe add one or two intangibles to the list.

Thursday, March 19, 2009

Full Text Indexing of CouchDB with Lucene

‹prev | My Chain | next›

Having gotten couchdb-lucene and edge CouchDB installed and running, I'll keep my chain going by trying to get indexing and searching to work.

I am running in local development environment (./utils/run), so I need to edit the etc/couchdb/local_dev.ini to include:
[couchdb]
os_process_timeout=60000 ; increase the timeout from 5 seconds.

[external]
fti=/usr/bin/java -jar /home/cstrom/repos/couchdb-lucene/target/couchdb-lucene-SNAPSHOT-jar-with-dependencies.jar -search

[update_notification]
indexer=/usr/bin/java -jar /home/cstrom/repos/couchdb-lucene/target/couchdb-lucene-SNAPSHOT-jar-with-dependencies.jar -index

[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}
The next step is to start up the CouchDB server:
cstrom@jaynestown:~/repos/couchdb$ ./utils/run 
Apache CouchDB 0.9.0a756286 (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
[info] [<0.58.0>] 127.0.0.1 - - 'GET' /_all_dbs 200
[info] [<0.58.0>] 127.0.0.1 - - 'GET' /eee/_design/lucene 404
[info] [<0.58.0>] 127.0.0.1 - - 'GET' /eee 200
To verify that the index is working, you can access the _fti resource of the database:
cstrom@jaynestown:~/repos/couchdb-lucene/target$ curl http://localhost:5984/eee/_fti
{"doc_count":7,"doc_del_count":2,"last_modified":1237514082000,"current":true,"optimized":false,"disk_size":13669}
Nice! I do have 7 documents in there, so we look to be in good shape.

To search, append a q query parameter to the request with a value in the form attribute_name:search term. We like our greens, so, to search for all recipes (in our limited sample) that include a word starting with "green" in the summary, you would supply the search term: q=summary:green*.

Giving it a try, I find that we do indeed have 2 recipes mentioning "greens":
cstrom@jaynestown:~/repos/couchdb-lucene/target$ curl http://localhost:5984/eee/_fti?q=summary:green*
{"q":"+_db:eee+summary:green*","etag":"1202191c377","skip":0,"limit":25,"total_rows":2,"search_duration":1,"fetch_duration":1,
"rows":[{"_id":"2006-10-08-dressing", "score":0.9224791526794434},
{"_id":"2006-08-01-beansgreens","score":0.8661506175994873}]}
Aside from the yak shaving needed to get edge CouchDB running, this was by far the easiest experience I have ever had in getting Lucene indexing running.

I would ultimately like to be able to search an entire document, not just individual fields, but this will do for now.

Yak Shaving is the new Dependency Hell

‹prev | My Chain | next›

Important Note: to run CouchDB 0.9, you must have the erlang emulator 5.6 or higher installed (erlang 12.b.3 that comes with Ubuntu 8.10 / Intrepid will work). To see which version of the emulator you are running, issue the following at the command prompt:erl +V

One of the requirements for EEE Cooks is full text searching. So, for my chain tonight, I wanted to give couchdb-lucene a try. Since it only works on 0.9 (latest trunk), I need to do a bit of yak shaving.

The install of couchdb-lucene is straight-forward. On my Ubuntu system, all that was needed was:
sudo apt-get install maven2
git clone git://github.com/rnewson/couchdb-lucene.git
cd couchdb-lucene/
mvn
Installing couchdb from subversion incurred a thin slice of dependency hell. After checking it out and bootstrapping:
svn co http://svn.apache.org/repos/asf/couchdb/trunk couchdb
cd couchdb
./bootstrap
I give configuration a try only to hit this:
checking for icu-config... no
*** The icu-config script could not be found. Make sure it is
*** in your path, and that taglib is properly installed.
*** Or see http://ibm.com/software/globalization/icu/
configure: error: Library requirements (ICU) not met.
On Ubuntu, this is resolved with
cstrom@jaynestown:~/repos/couchdb$ sudo apt-get install libicu-dev
But then I get:
checking for curl-config... no
*** The curl-config script could not be found. Make sure it is
*** in your path, and that curl is properly installed.
*** Or see http://curl.haxx.se/
configure: error: Library requirements (curl) not met.
Man, I hope this doesn't take too long. Resolve this dependency with:
cstrom@jaynestown:~/repos/couchdb$ sudo apt-get install libcurl-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package libcurl-dev is a virtual package provided by:
libcurl4-openssl-dev 7.18.2-1ubuntu4.3
libcurl4-gnutls-dev 7.18.2-1ubuntu4.3
You should explicitly select one to install.
Great. Not looking good. Well, I'll pick one. let's say the gnutls one:
cstrom@jaynestown:~/repos/couchdb$ sudo apt-get install libcurl4-gnutls-dev
Have I reached my yak yet?
./configure
...
checking for erl... /usr/bin/erl
checking for erlc... /usr/bin/erlc
checking erl_driver.h usability... no
checking erl_driver.h presence... no
checking for erl_driver.h... no
configure: error: Could not find the `erl_driver.h' header.

Are the Erlang headers installed? Use the `--with-erlang' option to specify the
path to the Erlang include directory.
Nope, have not quite reached the yak.
cstrom@jaynestown:~/repos/couchdb$ sudo apt-get install erlang-dev
Please let that be it:
./configure
...
You have configured Apache CouchDB, time to relax.

Run `make && make install' to install.
Woo hoo!

I am not one for running downloaded, unsigned code as root, so after making the code, run the local development mode to verify that it is functional:
make && ./utils/run 
Apache CouchDB 0.9.0a756286 (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
Next up: full text indexing.

Wednesday, March 18, 2009

Benchmarking

‹prev | My Chain | next›

At this point in my chain, I have a good understanding of the work involved in making the switch to CouchDB.

I am favoring the idea of running Sinatra based on the idea that it will keep me close to the metal, but is it enough? One of the nice things about running on Rails is the awesome built-in caching. In our old code, we cache the entire page. After it has been served up once, it is always served directly from cache for any subsequent requests. Can Sinatra compete?

I just want ballpark answers here, so ApacheBench ought to be more than sufficient. So, start up the old Rails app in production mode:
cstrom@jaynestown:~/repos/eee.old$ ./script/server -e production -d
=> Booting Mongrel (use 'script/server webrick' to force WEBrick)
=> Rails 2.1.2 application starting on http://0.0.0.0:3000
Then run ApacheBench against for 100 or so requests:
cstrom@jaynestown:~/repos/eee.old$ ab -n 100 http://127.0.0.1:3000/recipes/show/568
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done


Server Software: Mongrel
Server Hostname: 127.0.0.1
Server Port: 3000

Document Path: /recipes/show/568
Document Length: 10793 bytes

Concurrency Level: 1
Time taken for tests: 1.221 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 1125500 bytes
HTML transferred: 1079300 bytes
Requests per second: 81.89 [#/sec] (mean)
Time per request: 12.212 [ms] (mean)
Time per request: 12.212 [ms] (mean, across all concurrent requests)
Transfer rate: 900.07 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 2 12 92.6 2 927
Waiting: 2 12 92.6 2 926
Total: 2 12 92.6 2 927

Percentage of the requests served within a certain time (ms)
50% 2
66% 2
75% 2
80% 2
90% 2
95% 3
98% 52
99% 927
100% 927 (longest request)
The first request takes 927ms, subsequent cache hits were almost always of the order of 2ms. The most important number in the results is "Requests per second", which comes in at 81.89 [#/sec]. That first 927ms is really throwing off the statistics. Running the benchmark again yields a number of requests per second on the order of 300 per second.

To compare to a Sinatra/RestClient application, I resurrect the code from my CouchDB spike and start it up in production mode as well:
cstrom@jaynestown:~/repos/tmp$ ruby ./eee_sinatra.rb -e production
== Sinatra/0.9.1 has taken the stage on 4567 for production with backup from Mongrel
Running the benchmark suite returns:
cstrom@jaynestown:~/repos/eee.old$ ab -n 100 http://127.0.0.1:4567/meals/2007-11-16-pumpkinpecan
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done


Server Software:
Server Hostname: 127.0.0.1
Server Port: 4567

Document Path: /meals/2007-11-16-pumpkinpecan
Document Length: 703 bytes

Concurrency Level: 1
Time taken for tests: 0.463 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 82400 bytes
HTML transferred: 70300 bytes
Requests per second: 216.15 [#/sec] (mean)
Time per request: 4.626 [ms] (mean)
Time per request: 4.626 [ms] (mean, across all concurrent requests)
Transfer rate: 173.93 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 3 5 4.8 4 46
Waiting: 0 4 4.3 4 42
Total: 3 5 4.8 4 46

Percentage of the requests served within a certain time (ms)
50% 4
66% 4
75% 4
80% 4
90% 4
95% 11
98% 23
99% 46
100% 46 (longest request)
So in the end, Sinatra/Couch is roughly 33% slower than cached Rails. I think I can live with that.

Tuesday, March 17, 2009

Complete Recipe Upload to CouchDB

‹prev | My Chain | next›


I got the JSON dump of recipes in pretty good shape last night simply by mucking with the various options that ActiveRecord's to_json method recognizes (:include, :except, and :methods). That accounts for most of a recipe's infomation, but not the photos.

So how to deal with photos? Continue to use a CouchDB-tailored attachment_fu (or switch to a CouchDB tailored paperclip)? Fortunately, the answer is a little more direct than that—CouchDB itself supports attachments.

My preference would be to use the nicer REST API that CouchDB provides for "Standalone Attachments" (see the CouchDB HTTP Document API, toward the bottom for more details). That is a recent addition, not available in the stock 0.8 that is packaged in Ubuntu's universe. Rather than shaving that yak, I will stick with "Inline" CouchDB attachments for now.

The thing about inline attachments in CouchDB is that they need to be Base64 encoded (and stripped of newline characters). So we need to grab the images from the file system (via attachment_fu that the old code uses), Base64 encode it, and put it into CouchDB. For CouchDB to recognize them, they need to be added in the record's _attachments key.

To grab it from the file system:
jpeg = File.open(recipe.image.full_filename).read
Base64 encoding (and removing newlines) is easy enough:
require 'base64'
Base64.encode64(jpeg).gsub(/\n/, '')
To get this into the JSON data structure (and thus into CouchDB), define an _attachements method in the Recipe class:
class Recipe
def _attachments
{
self.image.filename =>
{
:data => Base64.encode64(File.open(self.image.full_filename).read).gsub(/\n/, ''),
:content_type => "image/jpeg"
}
}
end
end
Next update the to_json method to include the _attachments method in the :methods option:
json = recipe.to_json(
:methods => [:tag_names, :_id, :_attachments],
:include => {
:preparations =>
{
:include =>
{
:ingredient => { :except => :id }
},
:except => [:ingredient_id, :recipe_id, :id]
},
:tools => { :except => [:id, :label] } },
:except => [:id, :label])
Lastly, put this into the DB with a simple RestClient put:
RestClient.put "http://localhost:5984/eee-meals/#{recipe._id}", json, 
:content_type => 'application/json'
Viewing the record in futon, we see that it does have a JPEG image attachment:


Clicking the attachment returns the original image:


At this point, we have a nice mechanism for transporting the data from the old relational DB into CouchDB. It needs to be expanded to run through all recipes (and maybe to add some error handling). Also, we still have to get this working for meal documents. The spike, and that is what this has been—a spike to understand how to transfer data from the legacy Rails application into CouchDB—is complete.

Monday, March 16, 2009

Recipe.to_json

‹prev | My Chain | next›

As described the other day, I am not immune to the siren's song of adding the really cool feature that will certainly be used someday. As always is the case when I do these things, it costs me later. In particular, I added the following to my cooking blog's Recipe class:
  def to_hash
{
:title => title,
:date => date.to_s,
:label => label,
:image => image,
:serves => serves,
:cook_time => cook_time,
:prep_time => prep_time,
:inactive_time => inactive_time,
:serves => serves,
:tags => tags.map(&:name),
:ingredients => ingredients.map(&:name)
}
end

def to_json
to_hash.to_json
end
OK, so maybe it wasn't that cool. This must have pre-dated to_json being part of Rails core. The best explanation that I can is that it was done for a presentation, but that's a lousy reason for it to still be in production code. At any rate, I remove it from my local copy and enter into a script/console session. The default Recipe#to_json is not going to cut it:
>> r = Recipe.first
=> #<Recipe id: 24, title: "Caper and Red Wine Vinaigrette", label: "salad",
summary: " A vinaigrette is so simple and quick to prepare ...",
serves: 2, prep_time: 5, cook_time: 0, inactive_time: 0, image_old: "vinaigrette_7195.jpg",
date: "2006-04-04",
instructions: " \n We combine the vinegar and a small pinch of salt...",
recipe_group_id: nil, howto: false, author_id: nil, published: true>
>> r.to_json
=> {
"label": "salad",
"prep_time": 5,
"title": "Caper and Red Wine Vinaigrette",
"inactive_time": 0,
"published": true,
"howto": false,
"date": "2006\/04\/04",
"id": 24,
"image_old": "vinaigrette_7195.jpg",
"recipe_group_id": null,
"serves": 2,
"author_id": null,
"cook_time": 0,
"summary": " A vinaigrette is so simple and quick to prepare ...",
"instructions": " \n We combine the vinegar and a small pinch of salt..."
}
Fortunately, it is easy to include relationships as well as to exclude certain unnecessary attributes. What I eventually end up with is:
>> r.to_json(:methods => [:_id, :tag_names], 
:include => { :preparations => { :include => {:ingredient => {:except => :id }},
:except => [:ingredient_id, :recipe_id, :id] },
:tools => {:except => [:id, :label]} },
:except => [:id, :label])
The resulting JSON should include a key, tag_names, pointing to an array built from the results of a call to the tag_names method (def tag_names; tags.map(&:name) end).

Additionally, the resulting JSON should include two other arrays built from Recipe associations: preparations (of ingredients) and tools used to make the recipe. The resulting JSON data structure is pretty darn close to what I would eventually like to use:
=> {  
"prep_time": 5,
"title": "Caper and Red Wine Vinaigrette",
"inactive_time": 0,
"published": true,
"howto": false,
"tools": [
{
"title": "Bowl",
"amazon_title": "Pyrex Mixing Bowls",
"asin": "B0000644FE"
}
],
"date": "2006\/04\/04",
"_id": "2006-04-04-salad",
"image_old": "vinaigrette_7195.jpg",
"recipe_group_id": null,
"serves": 2,
"tag_names": [
"dinner",
"first-course",
"side",
"salad",
"vegetarian"
],
"author_id": null,
"cook_time": 0,
"summary": " A vinaigrette is so simple and quick to prepare ...",
"preparations": [
{
"brand": null,
"quantity": 1.0,
"order_number": null,
"unit": "teaspoon",
"description": null,
"ingredient": {
"kind": "red wine",
"name": "vinegar"
}
},
{
"brand": null,
"quantity": null,
"order_number": null,
"unit": null,
"description": null,
"ingredient": {
"kind": null,
"name": "salt and pepper"
}
},
{
"brand": null,
"quantity": 1.0,
"order_number": null,
"unit": "tablespoon",
"description": null,
"ingredient": {
"kind": "extra-virgin olive",
"name": "oil"
}
},
{
"brand": null,
"quantity": 0.25,
"order_number": null,
"unit": "teaspoon",
"description": "rinsed",
"ingredient": {
"kind": "packed in salt",
"name": "capers"
}
},
{
"brand": null,
"quantity": null,
"order_number": null,
"unit": null,
"description": null,
"ingredient": {
"kind": null,
"name": "salad greens"
}
},
{
"brand": null,
"quantity": null,
"order_number": null,
"unit": null,
"description": null,
"ingredient": {
"kind": "green, Picholine",
"name": "olives"
}
}
],
"instructions": " \n We combine the vinegar and a small pinch of salt..."
}
The tools data structure is simple enough (though I do not recall why we stored the Amazon.com title):
    {
"title": "Bowl",
"amazon_title": "Pyrex Mixing Bowls",
"asin": "B0000644FE"
}
The preparation data structure is a little more interesting and somewhat reminiscent of Cooking for Engineers. For instance, given some capers, we measure out 1/4 teaspoon of them and rinse them:
    { 
"brand": null,
"quantity": 0.25,
"order_number": null,
"unit": "teaspoon",
"description": "rinsed",
"ingredient": {
"kind": "packed in salt",
"name": "capers"
}
}
I can envision building richer data structures in the the future to describe higher level steps in a recipe, like mixing:
{ 
"description": "tossed",
"children": [
{
"quantity": 0.25,
"unit": "teaspoon",
"description": "rinsed",
"ingredient": {
"kind": "packed in salt",
"name": "capers"
},
},
{
"quantity": 1,
"unit": "bunch",
"ingredient": {
"name": "salad greens"
}
}
]
}
That's pie in the sky at this point, but I think I'll keep the data structure as-is for now. All that is left is to dump the data into CouchDB. Next time...

Sunday, March 15, 2009

Running Old Code

‹prev | My Chain | next›

Following the instructions that I left myself, I checkout my old code:
svn co http://svn.eeecooks.com/svn/eeecode/trunk eee.old
After editing my config/database.yml file to point to the datbase restored last night, I fire up script console only to find that I need to re-install the activerecord-postgresql-adapter. Ugh... another gem that didn't survive the rubygems upgrade.

Maybe it is just me, but the PostgreSQL adapter always seems to involve a little bit of black magic, starting with installing the pg gem, not the requested activerecord-postgresql-adapter:
cstrom@jaynestown:~/repos/eee.old$ gem install pg
WARNING: Installing to ~/.gem since /usr/lib/ruby/gems/1.8 and
/usr/bin aren't both writable.
Building native extensions. This could take a while...
Successfully installed pg-0.7.9.2008.10.13
1 gem installed
Somehow, I got myself in a state in which no matter what I tried, I would get the error:
psql: FATAL:  Ident authentication failed for user "rails"
It seemed that a couple of times of changing the value host in my config/development.yml between localhost, 127.0.0.1, and not specifying it at all resolved the problem:
development:
adapter: postgresql
database: eeecooks_development
username: rails
password: secret
host: localhost
All I can figure is that I had a stray tab in the YAML configuration file, because the only way to reliably reproduce now is to not include the host line. Ah well, at least it works:
>> cstrom@jaynestown:~/repos/eee.old$
cstrom@jaynestown:~/repos/eee.old$
cstrom@jaynestown:~/repos/eee.old$ ./script/console
Loading development environment (Rails 2.1.2)
>> Recipe.count
=> 578
And:
cstrom@jaynestown:~/repos/eee.old$ ./script/dbconsole -p
Welcome to psql 8.3.6, the PostgreSQL interactive terminal.

Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit

SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)

eeecooks_development=> select count(*) from recipes;
count
-------
578
(1 row)
Up next is playing with JSON serializing—gonna need it at some point to get the data into CouchDB and it readily available now...

Saturday, March 14, 2009

Backing Up PostgreSQL

‹prev | My Chain | next›

For my chain tonight, I thought it worth verifying that I could generate JSON from my current application.

Sadly, this turns out to mean yak shaving. You see, I had made the unfortunate choice to override to_json (and to_hash) in the original code. For the life of me, I can not remember why I did this. The only JSON that I ever recall using was for a BMore on Rails presentation that I did back when the codebase was still on Rails 1.2.x (pre-dating core to_json?). This is what happens with inside-out development—weird artifacts of things I thought I might have needed in the future. Instead, they are causing me trouble now that the future it today.

Sooo....

I grab a copy of my production database, which is running on PostgreSQL:
pg_dump -d  -U  -h 127.0.0.1 > eeecooks_pg.sql
gzip eeecooks_pg.sql
Next up is installing PostgreSQL on my local Ubuntu. Following along with the Ubuntu community documentation, I install the package:
sudo apt-get install postgresql
And create a rails user and a database with which to work:
cstrom@jaynestown:~$ sudo -u postgres psql postgres
[sudo] password for cstrom:
Welcome to psql 8.3.6, the PostgreSQL interactive terminal.

Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit

postgres=# \password postgres
Enter new password:
Enter it again:
postgres=# \q
cstrom@jaynestown:~$ sudo -u postgres createdb -O rails eeecooks_development
To restore the downloaded copy, unzip and then use the psql command:
cstrom@jaynestown:~$ gunzip -dc eeecooks_pg.sql.gzip
cstrom@jaynestown:~$ psql -U rails -h 127.0.0.1 -d eeecooks_development -f eeecooks_pg.sql
A quick check of the database shows that it restored OK:
cstrom@jaynestown:~$ psql -U rails -h 127.0.0.1 -d eeecooks_development
Password for user rails:
Welcome to psql 8.3.6, the PostgreSQL interactive terminal.

Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit

SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)

eeecooks_development=> select count(*) from recipes;
count
-------
578
(1 row)
Tomorrow night, I will connect the rails application code, remove the to_json hackery and decide where to go next.

Friday, March 13, 2009

First Draft: Recipe Feature "Story"

‹prev | My Chain | next›

After an intense day at my day job, my chain is somewhat light tonight. Rather than make the ultimate call on the framework that I will use or the hosting service, I add a Rails Recipe story.

Feature: Recipe Details

So that I can accurately reproduce a recipe at home
As a web user
I want to be able to easily recognize important details

Scenario: Viewing a recipe with several ingredients

Given a recipe for Buttermilk Chocolate Chip Pancakes
When I view the recipe
Then I should see an ingredient of "1 cup of all-purpose, unbleached flour"
And I should see an ingredient of "¼ teaspoons salt"
And I should see an ingredient of "chocolate chips (Nestle Tollhouse)"

Scenario: Viewing a recipe with non-active prep time

Given a recipe for Crockpot Lentil Andouille Soup
When I view the recipe
Then I should see 15 minutes of prep time
And I should see that it requires 5 hours of non-active cook time

Scenario: Viewing a list of tools used to prepare the recipe

Given a recipe for Chicken Noodle Soup
When I view the recipe
Then I should see that it requires a bowl, a colander, a cutting board, a pot and a skimmer to prepare

Scenario: Main site categories

Given a recipe for Mango and Tomato Salad
And site-wide categories of Italian, Asian, Latin, Breakfast, Chicken, Fish, Meat, Salad, and Vegetarian
When I view the recipe
Then the Salad and Vegetarian categories should be active
I am not sure how I feel about these stories. First off, they do not detail the behavior of the system, rather characteristics of the system. Even so, they are important characteristics of EEE Cooks that we have cultivated over the years and need to remain in place when we transition back to a document based store.

In addition to the lack of comfort with the intrinsic nature of the stories, I am also bothered by the Given steps. They do not accurately describe the preconditions needed to reproduce the Then statements. Ultimately, that is a function of my abusing scenarios to describe characteristics, so I am willing to live with it.

Ultimately, these are not so much stories as characteristics of a legacy system that must be preserved. I am OK with abusing Cucumber—at least until the transition is made.

Thursday, March 12, 2009

Cucumber, Merb and CouchDB - Win (Finally)

‹prev | My Chain | next›

Sometimes it really pays to look at the source code.

Recall from yesterday's link in the chain that I was having difficulty reminiscent of http://www.ruby-forum.com/topic/176289. Despite my best efforts to follow the steps described, I was still getting:
Loading /home/cstrom/repos/eee-merb/config/environments/development.rb
Feature: See a meal # features/browse_meals.feature

So that I can see an old meal
As a web user
I want to browse a single meal by permalink
Scenario: View Meal # features/browse_meals.feature:6
Given a "Breakfast Elves" meal # features/steps/browse_meals.rb:1
When I view the meal permalink # features/steps/browse_meals.rb:8
Then the title should include "Breakfast Elves" # features/steps/browse_meals.rb:12
undefined local variable or method `response' for # (NameError)
./features/steps/browse_meals.rb:13:in `Then /^the title should include "(.+)"$/'
features/browse_meals.feature:9:in `Then the title should include "Breakfast Elves"'


1 scenario
2 steps passed
1 step failed
rake aborted!
Rather than posting a follow-up question, I looked at the commit history for the project. There are relatively few around the time of the ruby forum thread, so I poked through and found this commit. The examples in there all contain response_body.should rather than the response.should that I had used in my Sintra spike. So let's give that a try...

From the Top

To eliminate all doubt, I will start from scratch. First, I need to install the gem:
cstrom@jaynestown:~/repos/eee-merb$ gem install jsmestad-merb_cucumber
WARNING: Installing to ~/.gem since /usr/lib/ruby/gems/1.8 and
/usr/bin aren't both writable.
Successfully installed jsmestad-merb_cucumber-0.5.1.3
1 gem installed
Then I remove the old generated Cucumber files:
cstrom@jaynestown:~/repos/eee-merb$ merb-gen cucumber --session-type webrat -d
Loading init file from /home/cstrom/repos/eee-merb/config/init.rb
Loading /home/cstrom/repos/eee-merb/config/environments/development.rb
Generating with cucumber generator:
[DELETED] features/support/env.rb
[DELETED] lib/tasks/cucumber.rake
[DELETED] autotest/cucumber_merb_rspec.rb
[DELETED] features/steps/result_steps.rb
[DELETED] features/authentication/login.feature
[DELETED] features/authentication/steps/login_steps.rb
[DELETED] features/steps/webrat_steps.rb
[DELETED] bin/cucumber
[DELETED] cucumber.yml
And re-generate:
cstrom@jaynestown:~/repos/eee-merb$ merb-gen cucumber --session-type webrat
Loading init file from /home/cstrom/repos/eee-merb/config/init.rb
Loading /home/cstrom/repos/eee-merb/config/environments/development.rb
Generating with cucumber generator:
[ADDED] features/support/env.rb
[ADDED] lib/tasks/cucumber.rake
[ADDED] autotest/cucumber_merb_rspec.rb
[ADDED] features/steps/result_steps.rb
[ADDED] features/authentication/login.feature
[ADDED] features/authentication/steps/login_steps.rb
[ADDED] features/steps/webrat_steps.rb
[ADDED] bin/cucumber
[ADDED] cucumber.yml
I am not using merb-auth, so I delete the generated authentication example:
cstrom@jaynestown:~/repos/eee-merb$ rm -rf features/authentication/
Next up, I add my Before and After CouchDB hooks (see discussion from the other night as to the why). To do this, append to features/support/env.rb the following:
require 'rest_client'
Before do
uri = "http://127.0.0.1:5984#{repository(:default).adapter.uri.path}"
RestClient.put uri, { }
end

After do
uri = "http://127.0.0.1:5984#{repository(:default).adapter.uri.path}"
RestClient.delete uri
end
Remove the db:automigrate dependency from lib/tasks/cucumber.rake by commenting it out on line 16:
dependencies = ['merb_cucumber:test_env'] #, 'db:automigrate']
Finally, define features/browse_meal.feature:
Feature: See a meal

So that I can see an old meal
As a web user
I want to browse a single meal by permalink
Scenario: View Meal
Given a "Breakfast Elves" meal
When I view the meal permalink
Then the title should include "Breakfast Elves"
And the steps in features/steps/browse_meals.rb (using response_body instead of response):
Given /^a "(.+)" meal$/ do |title|
@permalink = title.gsub(/\W/, '-')

Meal.create(:id => @permalink,
:title => title)
end

When /^I view the meal permalink$/ do
visit("/meals/show/#{@permalink}.html")
end

Then /^the title should include "(.+)"$/ do |title|
response_body.should have_selector("h1", :content => title)
end
Running rake features now gives us our desired results:
cstrom@jaynestown:~/repos/eee-merb$ rake features
(in /home/cstrom/repos/eee-merb)
Loading init file from /home/cstrom/repos/eee-merb/config/init.rb
Loading /home/cstrom/repos/eee-merb/config/environments/development.rb
Feature: See a meal # features/browse_meals.feature

So that I can see an old meal
As a web user
I want to browse a single meal by permalink
Scenario: View Meal # features/browse_meals.feature:6
Given a "Breakfast Elves" meal # features/steps/browse_meals.rb:1
When I view the meal permalink # features/steps/browse_meals.rb:8
Then the title should include "Breakfast Elves" # features/steps/browse_meals.rb:12


1 scenario
3 steps passed


Finished in 0.002594 seconds

0 examples, 0 failures

Shutting Down the Spike

I did try re-adding the cucumber task's dependency on db:automigrate back. Unfortunately, I still get the rake borted messages of Unknown property 'views' and couch logs to the effect of:
[Fri, 13 Mar 2009 00:26:39 GMT] [info] [<0.592.0>] HTTP Error (code 404): {not_found,missing}
[Fri, 13 Mar 2009 00:26:39 GMT] [info] [<0.592.0>] 127.0.0.1 - - "GET /eee-test/_design/Merb::DataMapperSessionStore" 404
[Fri, 13 Mar 2009 00:26:39 GMT] [info] [<0.593.0>] HTTP Error (code 404): {not_found,missing}
[Fri, 13 Mar 2009 00:26:39 GMT] [info] [<0.593.0>] 127.0.0.1 - - "GET /eee-test/_design/Meal" 404
[Fri, 13 Mar 2009 00:26:39 GMT] [info] [<0.594.0>] 127.0.0.1 - - "PUT /eee-test/_design/Meal" 201
I am unsure if the problem is caused by the attempt at accessing the Merb::DataMapperSessionStore or the initial Meal attempt.

No matter—I have it working well enough that I could make a go at outside-in development of my chain on Merb. So the question is: Sinatra or Merb?