The two obvious natural physical characteristics that goes with 'smarts', and many book on anthropology or physiology will back it up, is size of the brain and the number and complexity of neural pathways. They can go hand in hand as the bigger brain will generally have more pathways, but you can also increase the numbers and complexities of these pathways by exercising your brain, read studying, and practice. A person with brain damage can grow new pathways and learning/relearning functions and memories. Brain tissue dies just like tissue in the rest of the body and if you don't exercise it, replacement of these pathways will differ or not happen at all.
It's not a direct correlation, as in a person with a larger brain will be smarter, but the potential is there. In human evolution, the human race evolved and subsequent species had larger brains, from ~300 cc in Australopithecus to todays ~1500 cc in homo sapiens giving rise to increased potential for an in skills. Cro-Magnon's brain was 10/15% larger than modern man, but not as convoluted. I believe it was noticed after Einstein death that his brain was ~15% bigger than the normal human. The more complex brains will also have more creases. Another physical attributes is the different development of the different areas of the brain, giving rise to different strengths and weaknesses in certain skills such as languages, spacial acuity, problem, solving and such. The brain is generally symmetrical, but not perfectly so. Then growth of new pathways explained why young children learn so much so quickly especially when compared to later stages of life.
There are also correlation between species that involves brain chemicals and DNA adding to the complexity, but that's beyond my abilities to easily explain.
Add to that the better diet, emotional support, social pressures to succeed,increased access to learning materials and facilities and you increase that end result.