The best age to downsize will differ from person to person. For a lot of people, their motivation for downsizing will dictate the time they will feel the need to downsize to a smaller house.
For instance, some people will be motivated by upcoming retirement and freeing up funds to enjoy their golden years. For other people the decision to downsize is prompted by children leaving the home, freeing up empty bedrooms.