While starting a new rails app, I noticed that I could simply skip installing
Test::Unit from the start. This left the initial application ready for Rspec to be installed and configured.
rails new my_app --skip-test
Pretty simple. Kicking myself for removing it manually each time I decided to use Rspec.