There was a fascinating discussion about this on the BBC Radio 4 sociology programme Thinking Allowed. Perhaps the most interesting thing was the story of a researcher who found convincing evidence that there was no particular yellow element to East Asian skin but ignored it because it was a racial assumption that he couldn’t change his mind about, despite the complete lack of scientific evidence behind it.
Apparently all early visitors to East Asia from Europe described the Japanese as white. The explanation for the later idea that they were yellow seems simply to be that yellow caused a nice round quartet with black, white and red.
Available to download on this page.
Japan Explained FASAQ 