Worrying just shows you have a health respect for your own life and those others of whom you care about.
That concern is yours and your alone, and you will find a way to deal with it as suits you best.
My only advice is it maybe time to take a break from riding if your worrying is detrimental to your control of the vehicle.
Taking things 'easier' isn't detrimental - if you are enjoying your ride and your control is smooth and safe then all you have done is simply reset the boundries you are prepared to ride to. It's when you suddenly bottle it half way round a corner, screwing up your line, snatching at the brake and wobbling like a weeble in an earthquake...
... then it could be affecting you too much and a break to gets things straight may be in order.
Take all that with a pinch* of salt though... it's said by a person riding around with a pair of cruthes straped to their back, so I have no regard, respect or sense of preservation about myself.
* Pinch = bucktfull.
