The UN health body outlined its stance in a statement late Friday after a group of 150 international doctors, scientists and researchers sent an open letter outlining fears that the arrival of half a million tourists for the Games could cause the virus to spread more rapidly around the world.
But WHO said moving the Olympics would not have a major impact on the spread of Zika.
"Based on current assessment, cancelling or changing the location of the 2016 Olympics will not significantly alter the international spread of Zika virus," it said.
Nearly 1,300 babies have been born in Brazil with the irreversible defect since the mosquito-borne Zika began circulating there last year.
Addressed to the world health body, the letter said that pressing on with the Games in Rio, the second-most affected city in Brazil, would be "irresponsible" and "unethical".
"An unnecessary risk is posed when 500,000 foreign tourists from all countries attend the Games, potentially acquire that strain, and return home to places where it can become endemic," said experts from the United States, Britain, Canada, Norway, the Philippines, Japan, Brazil, South Africa, Turkey, and Lebanon, among others.
"Should that happen to poor, as-yet unaffected places (eg, most of South Asia and Africa) the suffering can be great," it added.
But the WHO statement said Brazil was "one of almost 60 countries and territories which to-date report continuing transmission of Zika by mosquitoes.
"People continue to travel between these countries and territories for a variety of reasons. The best way to reduce risk of disease is to follow public health travel advice," it said.
The Olympics and Paralympics will take place between August 5 and September 18, which is wintertime in Brazil, with the WHO noting earlier this month that it was the season when there were "fewer active mosquitoes and the risk of being bitten is lower".