Conventional brake fluid is
Polyalkalene Glycol Ether according to
http://www.apracing.com/motorcycle/fluid/index.htm
I can't find this (or similar description) listed in rubber resistance tables like
http://www.baxterrubber.com/resistance-pg1.html
(unless it comes under the general grouping of "glycols")
but having said that, the sort of rubbers usually used for gasoline fuels generally have quite good resistance to most "non aggressive" liquids, so I'd guess you'll be OK.