Deputy Prime Minister Taro Aso on Monday retracted his remark the previous day that could be taken as blaming swelling social security costs and other issues associated with Japan's aging population and declining birthrates on those who do not have children.

"There are lots of strange people who say the elderly are to blame, but that is wrong. The problem is those who don't have children," Aso, who doubles as finance minister, said in a seminar held in his constituency in the southwestern prefecture of Fukuoka.

When grilled by an opposition lawmaker during a lower house committee session Monday, who criticized Aso's insensitivity to the feelings of people who want to have children but cannot, the heavyweight said, "I'll retract the remark if it's caused any misunderstanding."

The 78-year-old politician, who is known as gaffe-prone, also drew flak when he made a similar comment in December 2014.

Toshihiro Nikai, the secretary general of the conservative Liberal Democratic Party to which Aso belongs, also came under fire when he said in a speech in June last year, "These days some people have the selfish idea that it is better not to give birth to children."