Why did anime films become so popular in Japan?
The main reason was a lack of budget to compete with the special effects etc. of Hollywood sci-fi films etc. Plastic monster films like Gojira (Godzilla) and the British Thunderbirds puppets were other attempts to overcome the same budgetary constrictions. In Japan, there is also a very long tradition of writing comics to draw on.
Why did tentacle sex appear in Japanese anime?
Although there are examples of ukiyo-e prints involving octopuses, in anime it was basically a way to avoid censorship of showing certain private parts.