There isn’t a particularly positive image of built up guys. Rather than be impressed, people are likely to be intimidated. This is somewhat true with the opposite sex, but even more so in business- very unacceptable in Japan where the privelidge of intimidating people is strictly based on seniority!
There may also be a social class thing going on. In a similar way to what is happening with suntans, the Japanese are at the transition point between burly meaning a manual worker (therefore poor) and burly meaning plenty of time and money for a gym and nutrition drinks (therefore rich)