Prime Minister Shinzo Abe grumbled Wednesday about his ruling party's second-highest lawmaker's controversial remarks calling people who opt not to have children "selfish."

"It should be up to each person to decide whether to have children or marry," Abe said during a debate of party leaders in the Diet.

Toshihiro Nikai, secretary general of the Liberal Democratic Party, said Tuesday in a speech, "These days some people have a selfish idea that it is better not to give birth to children."

The 79-year-old Nikai's remarks were the latest in a series of gaffes by Japanese politicians demanding that younger people have babies to alleviate the country's rapidly graying population.

Nikai, regarded as the No. 2 figure in the ruling party after Abe, has three sons. He also said in the speech, "In order for everyone to be happy, we should have many children and develop our country."

(Toshihiro Nikai makes controversial remarks in Tokyo on June 26)

Abe, noting he has no children, said it is natural for each person to make their own choices.

Also referring to the gap between the rate of people who wish to have children and the percentage who actually do, Abe stressed the need to create a society where people can have babies if they wish.

Abe's government has sought to promote the empowerment of women and expand the number of women in the workplace as a key pillar of its growth strategy amid a lingering labor shortage.

Nikai's remarks have drawn criticism from both ruling and opposition lawmakers.

Yuichiro Tamaki, co-head of the opposition Democratic Party for the People, accused Nikai of promoting certain family values, saying the ruling party headed by Abe has a "very old-fashioned way of thinking."

Last month other LDP members also came under fire for remarks on childbearing, with Kanji Kato, a third-term lower house member, saying newlywed couples "must raise at least three children."

From the ruling bloc, Noritoshi Ishida, policy chief of the LDP's junior coalition partner Komeito party, said Nikai's remarks were "out of line," as there are people who cannot have children and each family has its own thoughts on the matter.