Dick Pound, a senior member of the International Olympic Committee, said a games held with grandstands full of only Japan residents would run counter to the Olympics' cosmopolitan spirit.
Speaking to Kyodo News in a telephone interview on Monday, Pound said local organizers are exploring a simplified Olympics and that a move beyond 2021 is still seen as unrealistic.
Shutting out international spectators due to worries about travel during the coronavirus pandemic is also not yet on the cards.
"I think that will not be the Olympics in a sense that we understand them," Pound said.
"We don't know what the travel restrictions will be (in 2021), whether there are some countries that may not be permitted to travel, or permitted to enter Japan. That needs to be decided on the basis of scientific assessment of the risk on a country-by-country basis."
Japan is currently refusing entry to non-Japanese people who have been to any of the 129 designated countries across the world within the past 14 days, including the United States, China and European nations, except under special circumstances.
Pound emphasized the importance of the Olympics bringing people together, something that a games attended just by residents of Japan would lack to some extent.
A sense of global friendship, inclusiveness and universality are among the ideals of the Olympic Movement, aspirations that would be difficult to achieve if the world is kept away from Japan.
Pound dismissed the idea of pushing back each of the next three summer Olympics by four years, saying both Paris 2024 and Los Angeles 2028 are too far into the organizing phase to make such a major change.
"I think the only chance for the games to be held in Tokyo is 2021," Pound said, reiterating what IOC President Thomas Bach said in May.
The Tokyo Olympics are currently scheduled to be held from July 23 to Aug. 8, 2021.
"It seems fairly clear that it cannot be postponed beyond 2021...In a sense, (2021 is) possible, it's certainly not unviable, I think everybody involved in the games would prefer to do that than have a cancellation."
The Canadian lawyer, a former IOC vice president and World Anti-Doping Agency president, said it is "almost impossible to answer in July of 2020" when the IOC will reach a final decision on whether the postponed Tokyo Games can go ahead.
Last month, Toshiaki Endo, a Tokyo 2020 executive board member, said organizers will need to monitor the novel coronavirus situation until next spring before deciding on the fate of the Tokyo Games.
Pound said the IOC and local organizers, as well as Olympic broadcasters, are looking into possible options such as reducing spectator numbers and simplifying the opening ceremony without compromising too much on entertainment value.
When asked about a July 16 Reuters article that quoted him as saying a Tokyo Games cancellation would inevitably mean the Beijing Winter Games in 2022 would face the same fate, the 78-year-old said it is not just the coronavirus that could disrupt the games in China.
"The political, boycott aspect about it is more problematic, in a sense that nobody knows how that develops over time. We had, back in 2008, there were many, many calls to boycott because of China's human rights record, and in the end there was no concerted boycott. Maybe this time the situation is more serious," he said.
For now, the IOC and Tokyo organizers are focusing on what they can control, hoping the situation becomes safe and clear, he said, adding that there is not yet enough science-based information to make an informed decision.
Japanese Prime Minister Shinzo Abe and IOC chief Bach agreed in a phone call in March to delay the games. Pound said the final decision will also have to be a joint decision between Tokyo organizers and the IOC.
It will also involve international sports federations and NOCs, who will be consulted, and public health authorities, who will complete a risk assessment of the rapidly evolving coronavirus situation and its impact on the Olympic Games, he said.
"I think the principle that made the Olympic Games such a worldwide celebration still exists (and) the idea of international competition in various sports is still of interest. I think people recognize the Olympics are special," Pound said.
"The concept has a universal appeal and therefore people motivated by solidarity will find a way to make it work."