The vote is regarded as illegitimate both by the acting Ukrainian government and by the West, but is widely expected to pass. Crimea is predominantly ethnic Russian, and its residents say they fear the government that took over when pro-Russia President Viktor Yanukovych was ousted last month will oppress them.
Since Yanukovych fled to Russia, Crimea has come under control of local militia forces, as well as heavily armed troops under apparent command from Moscow.
On Saturday, Ukrainian officials said Russian forces backed by helicopter gunships and armored vehicles had advanced about 6 miles (10 kilometers) over the Crimean border into another Ukrainian region, where they took control of a village that holds a natural gas distribution facility.
If the referendum passes, Russia faces the prospect of sanctions from Western nations, but Moscow has vigorously resisted calls to pull back in Crimea.
In Sevastopol, the Crimean capital where the Russian Black Sea Fleet is based under a lease agreement with Ukraine, enthusiasm for the referendum was high with voters lining up outside polling stations before they opened.
"Today is an important day for all Crimea, Ukraine and Russia," said voter Manita Meshchina. "I think that people are expecting the majority of people will vote `yes.' What it means is that people believe and think they need to be with Russia."
In Sevastopol, more than 70 people surged into a polling station within the first 15 minutes of voting.
"Today is a holiday," said one of them, 66-year-old Vera Sverkunova. Asked how she voted, she broke into a patriotic war song: "I want to go home to Russia, it's been so long since I've seen my mama."
Dalton Bennett in Sevastopol and Jim Heintz in Kiev contributed to this story.