While starting a new rails app, I noticed that I could simply skip installing Test::Unit from the start. This left the initial application ready for Rspec to be installed and configured.

rails new my_app --skip-test

Pretty simple. Kicking myself for removing it manually each time I decided to use Rspec.