Selection technically isn't modification, since the modification had to have already occurred for it to be selected for. However, modification certainly did occur, and all crops are genetically modified. Indeed, all living creatures are genetically modified, as without modification, evolution can't occur.
The public fear of GMO's is largely due to Monsanto, who aggressively protect their GMO crop patents to the point where farmers who just happened to have some seeds blow into their fields have been sued.
The issue with GMO's isn't the modification, it's the lax patent laws that allow companies like Monsanto to exploit people for profit, giving a bad name to the field as a whole, in spite of the immense potential good it can do, for which Golden Rice is a prime example.