The No. 2 figure of Japan's major ruling party on Monday supported Yoshiro Mori to stay on as the head of the Tokyo Olympic organizing committee despite his recent sexist remarks that have drawn criticism at home and abroad.

"I would like him to work hard to meet the expectations of the people around him," Toshihiro Nikai, who has been serving as secretary general of the Liberal Democratic Party since 2016 said in a press conference.

Related coverage:

FOCUS: With Olympics at stake, Mori seen as too influential to push aside

Female Rugby Union board member says Mori's comment directed at her

Tokyo Olympic chief Mori's remarks given "gold medal" for sexism

Nikai, an influential figure of the party headed by Prime Minister Yoshihide Suga, said it will "not be a problem" for Mori to remain as the president of the committee because he has already retracted the comments he made last week about women talking too much during meetings.

The comments have sparked anger in Japan with many calling for Mori to step down. The organizing committee said Monday that about 390 Olympic and Paralympic volunteers have so far withdrawn their registrations in the wake of the remarks.

Commenting on the increasing number of volunteers deciding to turn down their roles, Nikai said he hopes "their thoughts will change once the situation settles down."

Suga and other LDP lawmakers, including Olympic Minister Seiko Hashimoto, a former Olympian herself, have also backed Mori, an 83-year-old former prime minister from the party, to continue preparations for this summer's Olympics and Paralympics.

On Wednesday, Mori told an online session of the Japanese Olympic Committee that meetings with women tend to "drag on" because they often talk too much. He took back the comments and apologized the following day.

Both the International Olympic Committee and International Paralympic Committee released comments stating that Mori's apology closed the matter.