The drowsiness lasts a few hours (2-4?), but the decreased sex drive doesn't usually last as long (1-2) hours.
As long as I've been taking them regularly, both of these symptoms are more manageable. I've never felt it was unsafe to drive my car or anything, but your mileage may vary.