Most of the 20 million items it contained, including the oldest human remains ever found in the Americas, are believed to have been destroyed.
The cause of the blaze is not known. No injuries have been reported.
The museum, located in a building that once served as the residence for the Portuguese royal family, celebrated its 200th anniversary this year.
The fire started on Sunday evening, after the facility had closed for the day.
Aerial images broadcast on Brazilian television showed it spreading throughout the building.
Brazil's President Michel Temer said in a tweet that it was a "sad day for all Brazilians" as "200 years of work, research and knowledge were lost".
Roberto Robadey, a spokesman for the Rio fire department, is quoted by the Associated Press news agency as saying that the hydrants closest to the museum were not working and that firefighters had to get water from a nearby lake.
By Monday morning the fire was under control and some of the museum's pieces had been rescued, he added.
A stark metaphor for a city in crisis
By Katy Watson, BBC South America correspondent
This isn't just Brazilian history that's gone up in flames. Many see this as a metaphor for the city - and the country as a whole.
Rio de Janeiro is in crisis. Growing violence, a deep economic decline and political corruption have combined to make the city a shadow of what it once was. It was only in 2016 that it was hosting the Olympic Games - an event into which Brazil poured billions of dollars.
But the hangover from the sporting event has hit Rio hard. Add to that the fact that federal spending has been slashed, and with violence on the rise, tourism numbers have also declined.
This was a museum that many saw as long ignored and underfunded - now, with devastating consequences for Brazil's heritage.
What did the museum contain?
It was one of the largest museums of natural history and anthropology in the Americas.
Its millions of artefacts included fossils, Brazil's largest meteorite, dinosaur bones and a 12,000-year-old skeleton of a woman known as "Luzia", the oldest ever discovered in the Americas.
The building was also home to items covering the centuries from the arrival of the Portuguese in the 1500s to the declaration of a republic in 1889.
The ethnology collection had unique pieces from the pre-Columbian era and artifacts from indigenous cultures.
Pieces from Greco-Roman times and Egypt were also on display at the museum.
Portugal's royal family transferred the court to the building in 1808, when the country faced with the threat of invasion from Napoleon.
The museum was established in 1818, with the aim of promoting scientific research by making its collection available to specialists.
What has the reaction been?
Marcelo Moreira - a journalist with TV Globo in Rio - told the BBC: "It's very sad... This museum has a great history. It represents a lot for Brazilian history and Brazilian culture. It's really a big loss for Brazil."
In an interview with Globo, the museum's director said it was a "cultural tragedy".
One museum employee told the network that project managers had had "tremendous difficulty" trying to secure funding for "sufficient" resources for the palace.
