The United States does have referendums at the state level and all 50 states have some power of referendum offered to the people (the most common being legislature prescribed referendums to the people of general laws, which all states have. Constitutional Amendments done in such fashion exist in every state but Delaware). Wikipedia lists the United States as an example of a (Semi-) Direct Democracy in its article on the matter.
The lack of Federal Referendum comes from a number of reasons, but the most commonly cited one was the Founding Father's distrust of direct democracy, seeing the form as a mob rule at best. They wanted a government where no one branch of government had enough power to run roughshod over another branch, and that no majority could overrun a minority (you can see this in the design of such elements as the Separate but Equal Branches, the use of the Electoral College, the bicameral nature of Congress) that there was a concerted effort to protect the minority (in the sense of party politics... they still had slavery as a legal thing for the better part of 90 years). They didn't fight against the tyranny of Britain to establish their own tyranny... and they didn't all agree with each other and wanted to make the fight easier and less bloody than the last one they had.
Another reason is that, rules as written, the Federal Government wasn't supposed to be dealing with the citizens all that often... that was mostly done at the state level or even smaller community levels. The Federal Government was more supposed to deal with two broad areas of topics: interactions between the states, and interactions with other nations. As a federation, the United States government is pretty much on par with a more strict EU. Each state, upon independence, was originally seen as their own separate nation that collectively agreed to surrender certain duties of a nation (namely the ability to declare war, the ability to create diplomatic policy, and the ability to regulate commerce leaving their territory), but retained every other ability of a government bound by constitutions. If you didn't need to do business outside of the state... and you didn't need to do business outside the United States... and you didn't need to fight a war with another country, you really didn't have much business with the Federal Government. At best, your interaction with federal agents was getting your mail from the friendly neighborhood postman. In the modern nation, there are a few more interactions, but again, not terribly many for the ordinary citizen. The phrase "All politics is local" is true, as at most, any given U.S. Citizen will have three national level ballot questions: who do you want to represent your congressional district in the House, who do you want to represent your state in the Senate, and who do you want your state to give its Electoral College votes to for President... all three are asking local questions that don't rise beyond the state level.
Finally, there's the issue of size. Switzerland, which has direct democracy, has a population of 8 million people compared to the United States' 320 million people, or roughly 40 times the population of Switzerland. It's a lot of ballots to manage to make a popular national decision and would have been such a daunting task that it would have been nearly impossible to do until relatively recently. And like you said, Brexit is a good argument on why certain questions shouldn't be punted to the people (the Swiss avoid this by imposing neutrality, thus it doesn't seek to have a lot of decision making that deals with international issues. The U.S. tried this too, but the first half of the 20th century had a nasty habit of bringing war to the States (not to mention there was a lot of popular support for joining the wars) and by the end of World War II, they got themselves locked in a game of chicken with the U.S.S.R. that required them to get involved with the world writ large... with varying degrees of success.