david chelimsky
david chelimsky
david chelimsky
on software in process and practice
story runner in plain english
8
posted by david
sun, 21 oct 2007 23:29:00 gmt
houston, we have plain text!
i just committed a first stab at a plain text story runner. it’s in rspec’s trunk and will be (in some form) part of the next release.
big thanks to pat maddox for the storypartfactory (which is now called storymediator) and to all on the rspec-users list who contributed their ideas and thoughts to the discussion about plain text stories.
keep in mind that this is brand new and very experimental. i do not recommend that you start converting all your projects to using this.
that said …
a bit of background
[update: modified to use and for multiple givens, whens or thens]
the initial implementation of story runner supported syntax like this (slightly modified from dan north’s article introducing rbehave):
story "transfer to cash account",
%(as a savings account holder
i want to transfer money from my savings account
so that i can get cash easily from an atm) do
scenario "savings account is in credit" do
given "my savings account balance is", 100 do |balance|
@savings_account = account.new(balance)
end
and "my cash account balance is", 10 do |balance|
@cash_account = account.new(balance)
end
when "i transfer", 20 do |amount|
@savings_account.transfer_to(@cash_account, amount)
end
then "my savings account balance should be", 80 do |expected_amount|
@savings_account.balance.should == expected_amount
end
and "my cash account balance should be", 30 do |expected_amount|
@cash_account.balance.should == expected_amount
end
end
scenario "savings account is overdrawn" do
given "my savings account balance is", -20
and "my cash account balance is", 10
when "i transfer", 20
then "my savings account balance should be", -20
and "my cash account balance should be", 10
end
end
while this is a really cool start, there are a couple of problems. one is that we’re constrained in the way we phrase things. because the arguments become part of the phrase, we have to structure each phrase so that the argument comes at the end.
the other problem, for me, is that the differing levels of abstraction in the two scenarios make it difficult to read.
enter blockless steps and step matchers
the first step in resolving this problem was to decouple the expression of the story from the steps, which is accomplished with the use of step matchers. here’s how the story above might look:
step_matchers = stepmatchers.new do |add|
add.given("my savings account balance is $balance") do |balance|
@savings_account = account.new(balance.to_f)
end
add.given("my cash account balance is $balance" do |balance|
@cash_account = account.new(balance.to_f)
end
add.when("i transfer $amount") do |amount|
@savings_account.transfer_to(@cash_account, amount.to_f)
end
add.then("my savings account balance should be $expected_amount" do |expected_amount|
@savings_account.balance.should == expected_amount.to_f
end
add.then("my cash account balance should be $expected_amount" do |expected_amount|
@cash_account.balance.should == expected_amount.to_f
end
end
story "transfer to cash account",
%(as a savings account holder
i want to transfer money from my savings account
so that i can get cash easily from an atm),
:step_matchers => step_matchers do
scenario "savings account is in credit" do
given "my savings account balance is 100"
and "my cash account balance is 10"
when "i transfer 20"
then "my savings account balance should be 80"
and "my cash account balance should be 30"
end
scenario "savings account is overdrawn" do
given "my savings account balance is -20"
and "my cash account balance is 10"
when "i transfer 20"
then "my savings account balance should be -20"
and "my cash account balance should be 10"
end
end
a bit nicer, yes? the matchers coming first is a bit noisy, but that could be extracted to another file, or perhaps we can add a means of associating them with the story after the story has already been parsed so they can move below the story.
that bit aside, look how much cleaner the story reads now. and we can do a couple of additional things to make it even nicer. one thing you might notice is that the line about transfering (when “i transfer 20”) doesn’t specify which way the transfer goes. we can improve on that by enhancing the step matcher:
step_matchers = stepmatchers.new do |add|
...
add.when("i transfer $amount from $source to $target") do |amount, source, target|
if source == 'cash' and target == 'savings'
@savings_account.transfer_to(@cash_account, amount.to_f)
elsif source == 'savings' and target == 'cash'
@cash_account.transfer_to(@savings_account, amount.to_f)
else
raise "i don't know how to transfer from #{source} to #{target}"
end
end
...
that lets us write the step as
when "i transfer 20 from savings to cash"
as you can see, this is a big step towards making stories more clear and flexible.
more on step matchers
another thing you may have noticed is that the step matchers are grouped together somewhat arbitrarily. thanks to a couple of handy convenience methods, you can easily build up libraries of these step matchers and make them as broad or as granular as you like. perhaps we want the account matchers available to many stories, but the transfer matcher only to this one. here’s how you can handle that:
class accountstepmatchers < spec::story::stepmatchers
step_matchers do |add|
add.given("my savings account balance is $balance") do |balance|
@savings_account = account.new(balance.to_f)
end
add.given("my cash account balance is $balance" do |balance|
@cash_account = account.new(balance.to_f)
end
add.then("my savings account balance should be $expected_amount" do |expected_amount|
@savings_account.balance.should == expected_amount.to_f
end
add.then("my cash account balance should be $expected_amount" do |expected_amount|
@cash_account.balance.should == expected_amount.to_f
end
end
end
step_matchers = accountstepmatchers.new do |add|
add.when("i transfer $amount") do |amount|
@savings_account.transfer_to(@cash_account, amount.to_f)
end
end
here we’ve created a subclass of stepmatchers, instantiated one and added an additional ‘when’ that will only be available to this instance.
goodbye quotes!
once we were able to get rid of the blocks, the quotes made no sense. so we’ve added support for true plain text stories. so now our example can like this:
story: transfer to cash account
as a savings account holder
i want to transfer money from my savings account
so that i can get cash easily from an atm
scenario: savings account is in credit
given my savings account balance is 100
and my cash account balance is 10
when i transfer 20
then my savings account balance should be 80
and my cash account balance should be 30
scenario: savings account is overdrawn
given my savings account balance is -20
and my cash account balance is 10
when i transfer 20
then my savings account balance should be -20
and my cash account balance should be 10
that gets stored in a plain text file and you can run it by running a ruby file that looks like this:
require 'spec'
require 'path/to/your/library/files'
require 'path/to/file/that/defines/account_steps.rb'
# assumes the other story file is named the same as this file minus ".rb"
runner = spec::story::runner::plaintextstoryrunner.new(file.expand_path(__file__).gsub(".rb",""))
runner.step_matchers << accountsteps.new
runner.run
and that’s it! it’s that simple. this is still in its very early phases and i’m certain there will be enhancements as people gain experience with it.
if you want to check it out yourself, grab the trunk and do the following:
cd trunk/rspec
ruby examples/stories/calculator.rb
ruby examples/stories/addition.rb
the first example uses ruby with blockless steps. the second example uses a plain text story stored in examples/stories/addition.
also, with a couple of small tweaks we’ll be able to consume plain text from any source (not just a local file) and feed it into the plaintextstoryrunner. this means that we’ll be able to do things like email scenarios to an app that consumes email and runs the scenario against the system and emails you back a report! crazy, huh?
lastly, just a reminder, this is only in trunk right now (as of rev 2764), so if you want to explore it you’ll have to get the trunk.
enjoy!!!!!
tags bdd, rspec, stories
meta
8 comments,
permalink,
rss,
atom
simple matchers made simple
3
posted by david
sat, 08 sep 2007 12:36:00 gmt
although rspec supports custom matchers, it has always been a bit more work than is ideal for simpler situations. this could be attributed to the desire for a system which would be flexible.
but now, with a bit of convention-over-configuration kool-aide, we offer you the simplematcher.
the simplematcher snuck its way into rspec’s source when we merged in the story runner (formerly rbehave). dan north had wanted a simpler way to create custom matchers, and so he created one and used it throughout the specs for the story runner.
and now we bring it to you (today if you use trunk, otherwise next release).
here’s how you use it:
def beat(hand)
return simple_matcher("hand that beats #{hand.to_s}") do |actual|
actual.beats?(hand)
end
end
full_house.should beat(flush)
=> nil #passes
straight.should beat(flush)
=> expected hand that beats flush, got straight
admittedly, these are only useful for very simple cases. but what’s in a name?
meta
3 comments,
permalink,
rss,
atom
159
2
posted by david
fri, 17 aug 2007 18:12:00 gmt
apparently i’ve reached some sense of balance. i’m 1/2 way there (somewhere). 1:59 since it all began, 1:59 left to go …
meta
2 comments,
permalink,
rss,
atom
rspec basics peep code is out
9
posted by david
sun, 08 jul 2007 15:56:35 gmt
topfunky has just released the first of a 3 part peepcode series on rspec entitled rspec basics. i’ve gone through it myself and i am impressed.
i would recommend this for anybody who is just getting started with rspec and rails. as its title suggests, it covers the basics: a bit of bdd philosophy, getting things set up, writing simple examples, all the way through describing models.
as for advanced users, i’d recommend this to those of you who are interested in a few helpful tips and tricks. there is some material on the textmate bundles and integrating with growl. in fact, the approach to validating models is simple and pragmatic, and one that i suspect will become the standard in time.
the only constructive criticism i’d offer is that the discussion of the philosophy of the rspec team doesn’t recognize the roots of our philosophy in test driven development. it seems to present writing focused examples, getting them to fail first, etc, as our idea. it is not. just ask google.
regardless, it is very exciting to see the beginnings of quality educational material on rspec and bdd emerging. keep your eyes open in this space. i suspect there is quite a bit more on the horizon.
tags bdd, peepcode, rspec
meta
9 comments,
permalink,
rss,
atom
template.expects_render
2
posted by david
thu, 28 jun 2007 05:00:00 gmt
one thing that’s been missing from rspec_on_rails for a while is a clean and consistent way to mock or stub calls to partials from inside partials. in fact, even mocking partials inside non-partial templates has been buggy. for example, let’s say you’re describing a template that should include a partial named ‘_thing’. you might want to do something like this:
assigns[:thing] = thing = object.new
template.should_receive(:render).with(
:partial => 'thing',
:object => thing
)
render 'things/index.html.erb'
now if that is the only example in your entire suite that renders ‘things/index.html.erb’, no problem. in other words, in most cases, this is a problem.
the problem
it turns out that rails compiles erb templates in memory the first time they are encountered and continues to use the compiled version throughout a given process. this is a good thing for performance. it is, however, a challenge for testability. why? because when we stub methods using rspec, mocha or flexmock, we replace the real methods with implementations from the mocking framework. those methods are part of what gets compiled. and that means weird stuff.
if you mock a method in the template and the template gets compiled, then every other access to that template accesses the mocked method (even accesses in other examples). conversely, if you mock a method in a template that’s already been compiled, well, it just doesn’t get hooked up at all and the mock expectation fails.
the problem with mocking partials inside partials is that a response only has one instance of actionview::base, so if you mock one call to render on that instance, you mock them all. this means that you simply can not use a standard mocking framework to mock the call as they are simply not designed to pass some calls on to the real object and intercept others.
rspec’s solution
to solve this, i created a proxy that delegates to a mock object, but that mock behaviour is not added to actionview::base directly. when actionview::base receives #render, it asks the proxy if it is interested in the call based on the arguments that were passed in. if so, it passes it on to the mock proxy for later verification, and otherwise swallows the call, the way a mock normally would. if it is not, however, interested in the call, execution of the render method continues as normal.
i have to confess that this feels a bit dirty. i come from a land of fairly strict rules about what mocks should and should not do, but now live in a land in which a lot of rules i learned before are being challenged. this is the land of ruby and rails. and so i grit my teeth, and do what seems pragmatic.
view examples
the result is that you’ll now be able to do this in view examples:
assigns[:thing] = thing = object.new
template.expects_render(:partial => 'thing', :object => thing)
render 'things/index.html.erb'
you can even do this if the thing you’re rendering in the example is a partial which contains a sub-partial:
thing = object.new
template.expects_render(:partial => 'thing', :object => thing)
render :partial => 'outer_partial', :locals => {:thing => thing}
controller examples too
i also added the same behaviour to controllers in controller examples. this essentially restores the old controller.should_render behaviour that we gave up in rspec 0.9, but with the syntax similar to that in the view examples above:
controller.expects_render(:action => 'login')
get 'index'
thanks to jake scruggs for pairing on this with me. it might not have happened had he not offered to help.
when can i use it?
this is committed to rspec’s trunk and will be released with rspec-1.0.6, sometime very soon.
tags mock, rails, render, rspec
meta
2 comments,
permalink,
rss,
atom
pending("insert reason here")
2
posted by david
sat, 23 jun 2007 16:04:00 gmt
in rspec-1.0, we introduced a not yet implemented feature. when you say …
it "should do something"
... with no block, the summary report lists that example as not implemented.
37 examples, 0 failures, 1 not implemented
as i started to use this i found myself doing stuff like this:
it "should do something"
# do
# here.is(the).actual(implementation).but(commented).out
# end
this made me sad. i hate having things that are commented out like that, even if the summary report draws my attention to it.
then came a conversation with dan about rbehave. in his article introducing rbehave, dan talks about identifying pending scenarios so instead of getting failures while he’s working on the objects that must implement the behaviour, he gets a nice list of scenarios that should pass pending the completion of those objects. we discussed the similarities and differences between the not yet implemented feature in rspec and the pending feature in rbehave and agreed that rspec should have the pending method.
and so it has come to pass.
rspec (trunk, as of rev 2118 – will be included in 1.0.6) still supports calling #it with no block, but now also supports the #pending method, allowing you to say:
describe "pending example (using pending method)" do
it %q|should be reported as "pending: for some reason"| do
pending("for some reason")
end
end
describe "pending example (with no block)" do
it %q|should be reported as "pending: not yet implemented"|
end
and hear:
$ ruby bin/spec examples/pending_example.rb -fs
pending example (using pending method)
- should be reported as "pending: for some reason" (pending: for some reason)
pending example (with no block)
- should be reported as "pending: not yet implemented" (pending: not yet implemented)
finished in 0.006639 seconds
2 examples, 0 failures, 2 pending
the #pending method raises a spec::dsl::examplependingerror, which gets reported, in this case, as “pending: for some reason”. if you leave off the block the example will be reported as “pending: not yet implemented”. either way, the summary will combine these two types of pending examples as just “pending”.
tags rbehave, rspec
meta
2 comments,
permalink,
rss,
atom
real confusion over mock concepts
1
posted by david
fri, 15 jun 2007 13:23:00 gmt
various dictionaries define mock (the noun) as an imitation, a counterfeit, a fake. the term “mock object” initially meant exactly that – an imitation object, which serves as a control (i.e. invariant) in a test, allowing you to limit the variables in your test to the object being tested.
over time, new, more specific definitions have emerged and confusion has ensued. now we are calling these things “test doubles”, of which there can be several types including what we now call mocks and stubs. and to make matters worse, if i use what we now call mocks, i’m a mockist and if i use what we now call stubs i’m a classicist. why can’t we just use both of these tools when each is appropriate and dispense with the name calling?
but i digress.
the main problem that i see with all of this is that the thing that separates the different kinds of test doubles is really how they behave at the individual message/method level. we have frameworks called mocking frameworks that create objects you can train to supply pre-defined responses (in which case its acting like a stub), record messages (in which case it’s a recording stub), and even verify that specific messages are received (in which case its a mock). the same object can have all of these behaviours, so the struggle over what to call the object seems to miss the point.
and to make matters more confusing, mocking frameworks in dynamic languages like ruby give you the ability to treat any real object in your system in the same way as you treat a generated mock, fake, test double, test spy, etc, etc, etc. in rspec, for example, you can do this:
real_collaborator = realcollaborator.new
real_collaborator.should_receive(:some_message)
object_i_am_describing = interestingclass.new(real_collaborator)
object_i_am_describing.do_something_that_should_send_some_message_to_collaborator
so what can we call this object? it’s real, but it behaves like a mock because i tell it to. this has always been considered a mocking no-no for many reasons. for example, you risk replacing methods that other methods in the same class rely on, which can lead to some unexpected test failures or, worse, passing tests that should be failing. in practice, i find that i only do this when dealing with other frameworks that rely on class methods (like ruby on rails’ activerecord).
but, again, i digress.
fighting confusion with more confusion
of late, i’ve gotten into the habit of talking about these things at the method level. you have an object (test double or real object) that can have method stubs and message expectations, either of which can return stub values. i’m hopeful that these are self-explanatory, but in the interest of minimizing confusion:
by “test double” i mean the meszaros definition. essentially, a test-specific substitute for a real object.
by “method stub” i mean a no-op implementation of a method that may or may not be called at any time during the test. a method stub returns a stub value unless it returns nil, none, void, nirvana, etc.
by “message expectation” i mean an expectation that a specific message will be received by the test double.
by “stub value” i mean a single, pre-defined value that will be returned by a method stub regardless of whether or not it is associated with a message expectation.
i recognize that i risk adding to the confusion instead of minimizing it, however i think this makes the whole thing easier to understand. what do you think?
tags mocks, stubs, test_doubles
meta
1 comment,
permalink,
rss,
atom
speaking at the rails edge
3
posted by david
wed, 06 jun 2007 17:03:16 gmt
mike mangino of elevated rails and i are joining forces to do a talk at the rails edge in chicago in august. we’ll be talking about how to use rspec and selenium together to drive the development of rails applications.
hope to see you there!
tags edge, rails, rspec, selenium
meta
3 comments,
permalink,
rss,
atom
rspec 1.0 - belated thanks
8
posted by david
mon, 04 jun 2007 05:18:52 gmt
the decision to release rspec 1.0 happened quite spontaneously at railsconf 2007 in portland. i heard more than one person there say they like rspec’s sensibilities and might use it if not for the changing api and upgrade problems, adding that they’d consider using it when it goes 1.0.
aslak was in portland as well, so he and i talked it over and decided that the time was right to put a stake in the ground and release rspec-1.0. brian takita was there as well, and was happy to join in the effort. so the three of us sat down to close up some holes and ship it.
but the really cool thing was that a few other people got wind of our plan and sat down with us to help make it all happen. big thanks go out to kurt schrader, chad humphries, ken barker and dustin tinney for joining the three of us. it was a blast hanging out with all you guys and the effort is seriously appreciated.
meta
8 comments,
permalink,
rss,
atom
oxymoron: testing behaviour of abstractions
2
posted by david
sun, 03 jun 2007 19:59:17 gmt
the question of how test behaviour implemented in abstract classes came up on the rspec list yesterday. this is something that comes up periodically, so i thought it worth posting about. over the years i’ve seen and tried a few different approaches to dealing with this problem and i’ve come to an approach that feels right to me. it stems from two basic principles:
abstract classes don’t have any behaviour
only remove duplication that actually exists
abstract classes don’t have any behaviour
abstract classes are a structural tool that we can use to share implementation between concrete classes. sometimes they actually represent abstract types, but all too often they’re misused as convenient placeholders for shared implementation. when that happens, their presence clouds meaning.
in statically typed languages like java and c#, you can spot this misuse by looking at the types being referenced in method signatures. if no methods accept the abstract class, then it isn’t really functioning as an abstract type in your system.
this is a bit more subtle in ruby because we don’t have an abstract class construct and we think in duck types instead of static or runtime types. abstract classes are really more of a convention than a language feature.
regardless, this rule of thumb steers me away from testing abstract classes directly.
only remove duplication that actually exists
this is a general rule of thumb that i like to apply whether i’m dealing with examples or subject code. only remove actual duplication. if you start with something general before you have anything specific, there is a tendency to make assumptions about what the duplication will be, and those assumptions are often misguided.
in terms of abstractions, this rule of thumb guides me to extract abstraction when i see duplication rather than imposing it up front.
putting the two together
the question on the list was specifically about how to test methods that live in in application in a ruby on rails app. for the uninitiated, application is the base class for all controllers in an mvc framework. ruby doesn’t support abstract classes, so you can actually initialize application, but afaik the rails framework never does.
based on the two principles, and given that i write executable examples first, i start by describing the behaviour of a concrete controller, developing methods directly on that controller against those examples. when a new controller comes along that should have the same behaviour, i’ll often duplicate the examples and the behaviour first and then extract the duplication into shared examples and the application controller. this way, if there are any differences at all in the expected behaviour (which there often are) it easier to figure out the exact bits that i want to extract.
once i’ve got the abstraction in both the examples and code i can just plug them in to the third, fourth, etc examples.
other schools of thought
there is a runtime cost to pay when you’re running the same examples for multiple subclasses of an abstract class. you could argue that this is wasteful because the same implementation is being tested multiple times, which flies in the face of the goal of fast running test suites. this argument might lead you to write tests directly for the abstract class.
i can’t really disagree with the performance cost, yet i still prefer to approach it as i do because i find it more clear and less brittle to be describing the behaviour of concrete types rather than the behaviour of abstract types (which doesn’t really exist).
tags bdd, behaviour, rspec, testing
meta
2 comments,
permalink,
rss,
atom
older posts: 1 2 3 4
searching...
archives
october 2007
(1)
september 2007
(1)
august 2007
(1)
july 2007
(1)
june 2007
(6)
may 2007
(7)
april 2007
(4)
march 2007
(3)
february 2007
(6)
january 2007
(3)
tags
autotest
bdd
behaviour
edge
flexmock
mocha
mock
mocks
nested_resources
ood
rails
rbehave
rspec
ruby
selenium
spec_ui
stubs
test_unit
testing
web_spec
blogroll
dave astels
aslak hellesøy
dan north
jay fields
steve freeman
pat maddox
object mentor blogs
luke redpath
brian takita
yurii rashkovskii
miscellany
rspec
articulated man
syndicate
articles
comments
david chelimsky
powered by typo /
styled with scribbish
david chelimsky Précédent 808 Précédent 807 Précédent 806 Précédent 805 Précédent 804 Précédent 803 Précédent 802 Précédent 801 Précédent 800 Précédent 799 Précédent 798 Précédent 797 Précédent 796 Précédent 795 Précédent 794 Précédent 793 Précédent 792 Précédent 791 Précédent 790 Précédent 789 Précédent 788 Précédent 787 Précédent 786 Précédent 785 Précédent 784 Précédent 783 Précédent 782 Précédent 781 Précédent 780 Précédent 779 Suivant 810 Suivant 811 Suivant 812 Suivant 813 Suivant 814 Suivant 815 Suivant 816 Suivant 817 Suivant 818 Suivant 819 Suivant 820 Suivant 821 Suivant 822 Suivant 823 Suivant 824 Suivant 825 Suivant 826 Suivant 827 Suivant 828 Suivant 829 Suivant 830 Suivant 831 Suivant 832 Suivant 833 Suivant 834 Suivant 835 Suivant 836 Suivant 837 Suivant 838 Suivant 839