Another problem is that schools for the intellectually gifted are perceived as "elitist" etc. I suppose in a way they are, but no one seems to have any trouble identifying and nurturing other talents, e.g. sport, music, whatever. Ideally, we should have places where all children can find a niche, whether they want to study advanced math or tinker with pipes and become the next generation of plumbers. Just about everyone has at least one thing they're good at, or good enough to eventually turn it into a satisfying (if not necessarily spectacular) career.
But I'm reluctant to say much about this. I honestly don't know how or even whether we should go to great lengths to nurture great talent. If I had kids, it would be more important to me that they be happy than that they become the next Von Neumann, even if they actually had that potential.
The other side of the coin is the somewhat notorious "Chinese mother" thing in which even untalented children are badgered and bullied daily, and end up being Olympic gymnasts or concert pianists, and personal happiness be damned. Some say it's a good thing: once you practice enough at something to get good at it, you will begin to enjoy it even if you didn't before. I wouldn't know. The libertarian in me balks at the idea of forcing anyone to do anything. :-)