I've done it before. It's not particularly difficult, just very time consuming. And at the end, you're left with a distribution that's not really that useful without repackaging everything you did into a package manager so you can do updates without borking it.
Great as a learning tool to see how the whole GNU/Linux stack works, but not something you'd use practically.