a view from afar

Drop this into your Rakefile and let the heckling begin!

First, you need basic RSpec tasks. I used the New Gem Generator to get my current project up and running, so I've got a tasks/rspec.rake file already in place. To avoid future conflicts when/if I choose to regenerate newgem's files, I put the rest of my tasks in tasks/rspec-extra.rake.

First you need the basic spec/coverage tasks. If memory serves, I nicked these from the RSpec project's own Rakefiles, slightly modified. (I don't actually need the .autotest exclusion, I personally only use ~/.autotest for configuration.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14

"Run all specs with rcov and store coverage report in doc/output/coverage"
Spec::Rake::SpecTask.new(:spec_rcov) do |t|
  t.spec_files = FileList['spec/**/*.rb']
  t.rcov = true
  t.rcov_dir = 'doc/output/coverage'
  t.rcov_opts = ['--exclude', 'spec,\.autotest']
end

desc "Verify that coverage is 100%"
RCov::VerifyTask.new(:verify_rcov => :spec_rcov) do |t|
  t.index_html = "doc/output/coverage/index.html"
  t.threshold = 100
end

Next up, a heckle task.

Currently, RSpec only lets you run Heckle via the spec command line, and not via a Rake task (and only in trunk as of revision 2804.) Also there's couple of bugs that affect this next bit. First, spec --heckle will heckle your code even if you have failing specs; second, spec --heckle will return successfully even if it catches.

The former is easy to work around: make your heckle task depend on a spec task (in this case, verify_rcov). The latter is more difficult: my solution is to run spec --heckle_ through IO.popen and record certain heckle messages, so it can produce a summary at the end (the first version I wrote of this was REALLY short, most of the code below is for the summary). Heckle will heckle all inner modules of the module passed on the command line, so you only need to specify the top-level module for your project (which I've moved to the top as a local variable).

If you have a good 1-1 correspondence between class and spec files, this will incur a lot of redundant re-runs of your specs (as it will run every spec for every mutation). My fatigued brain tells me the algorithm runs in O(N^2) time:

  N methods * k1 avg mutations/method * N methods * k2 avg specs/method

but it might be lying... it does that a lot this late at night. Either way, O(N) time should be possible without loss of coverage if you have symmetrical code and specs, but I'll worry about this when the run time of the task below becomes unbearable.

And, in true heckle spirit, this rake task lets you know exactly what it thinks of your specs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

desc "Heckle each module and class in turn"
task :heckle => :verify_rcov do
  root_module = "Celestial"
  spec_files = FileList['spec/**/*_spec.rb']
  
  current_module, current_method = nil, nil
  heckle_caught_modules = Hash.new { |hash, key| hash[key] = [] }
  unhandled_mutations = 0
  
  IO.popen("spec --heckle #{root_module} #{spec_files}") do |pipe|
    while line = pipe.gets
      line = line.chomp
      
      if line =~ /^\*\*\*  ((?:\w+(?:::)?)+)#(\w+)/
        current_module, current_method = $1, $2
      elsif line == "The following mutations didn't cause test failures:"
        heckle_caught_modules[current_module] << current_method
      elsif line == "+++ mutation"
        unhandled_mutations += 1 
      end
            
      puts line
    end
  end
  
  if unhandled_mutations > 0
    error_message_lines = ["*************\n"]
    
    error_message_lines << 
      "Heckle found #{unhandled_mutations} " + 
      "mutation#{"s" unless unhandled_mutations == 1} " +
      "that didn't cause spec violations\n"

    heckle_caught_modules.each do |mod, methods|
      error_message_lines <<
        "#{mod} contains the following poorly-specified methods:"
      methods.each do |m| 
        error_message_lines << " - #{m}"
      end
      error_message_lines << ""
    end
    
    error_message_lines <<
      "Get your act together and come back " +
      "when your specs are doing their job!"
    
    puts "*************"
    raise error_message_lines.join("\n")
  else
    puts "Well done! Your code withstood a heckling."
  end
end