I've got a Speed Four and absolutely love it. Completely different engine, being an IL4, which will probably take a bit of getting used to from an SV. More revs and gear changes, but for me the fun comes in keeping the engine buzzing.
Handling and brakes wise there is absolutely no comparison to an SV, it's a league above. The only criticism I have is that there's a pretty annoying flat spot at 3k rpm which means it's not as "practical" for commuting on but above that there's no other flat spots (until you touch the rev limiter
). If you can, I'd probably recommend you try get a test ride on one to see if you like it. I was fortunate enough that my brother had one so I got to try before I bought.
Price wise I reckon you can find one to fit your budget. I paid £2,000 for my 06 bike in autumn last year, which was a steal but there are bargains to be had. Yellow and blue bikes are the least popular so are generally cheaper than black, green or orange.