For most people English is a hobby. In Birmingham, the only French lessons available twice a week were exam classes- and most Japanese see learning English like the English see learning French. The other factor is that some may be keener than they seem- the Japanese are as reluctant to draw attention by being too enthusiastic as they are about being “the nail that stands up” out in any other way.
How can the same nation that has office workers who work 12 hour days and station masters who run up and down the platform in white gloves to make trains run on time produce such unmotivated language learners?
Japanese mothers seem to control their kids with an absolutely expert use of the kind of emotional blackmail that a Western parent might feel embarrassed about, plus a good dollop of indulgence and other manipulations. The net total result is a kid who will feel so upset themselves at displeasing their parents (mainly the mother), that no other punishment is needed- a child that will often grow up into an adult with much the same mentality. Another reason you would rarely see temper tantrums etc. if you are not part of the family is that a Japanese child quickly learns that different behaviour is suitable for different circumstances, and there is no such thing as “acting naturally” or “being true to yourself”.
More on this question in this New York Times article