FYI, there was a similar thread about my concerns a while ago.
http://forums.sv650.org/viewtopic.php?t=21396
As an update, I am really enjoying the biking, but I feel that it's definately good to choose what kind of riding I do. TBH, I ride the SV for pleasure only, and because I restrict myself to that I feel I reduce the risks. I rarely ride in a hurry, in places I don't know, in city traffic, in bad weather, in darkness, and with pillions and/or heavy luggage. Yes that means I generally ride on fast country roads and they can be dangerous (I almost hit a rabbit yesterday) but it's not a problem to slow right down as there is hardly any traffic.
Biking however has become a way for to meet new people and stay in touch with people I know. I also enjoy the physical aspect of looking after it and modifying it. I can and have spent hours of fun on it without going anywhere!
I think that if you like bikes, and don't spend time on them, you'll regret it one day. Just take it easy and use your head - and have fun!