None of the EU treaties specifically deal with this explicitly and this is a complex legal topic.
(There are claims that the "Vienna Convention on Succession of States in respect of Treaties" would solve that but so many countries haven't signed it that this can basically be ignored)
Firstly there isn't really a set procedure how countries leave the EU (much less what has to happen so that they get thrown out).
Article 50 only says that the European Council and that state will work it out somehow.
The new Scottish state isn't automatically a EU member - it has never applied to become a member of the EU and much less was accepted to be one - so it obviously can't be a member. (This is the current legal status)
To be able to join the EU the normal way a country needs to be able to fulfill the Copenhagen criteria. (The European Council judges whether those are fulfilled)
Broadly speaking those are:
Membership requires that candidate country has achieved stability of institutions guaranteeing democracy, the rule of law, human rights,
respect for and protection of minorities, the existence of a
functioning market economy as well as the capacity to cope with
competitive pressure and market forces within the Union. Membership
presupposes the candidate's ability to take on the obligations of
membership including adherence to the aims of political, economic and
monetary union.
(http://www.europarl.europa.eu/enlargement/ec/pdf/cop_en.pdf)
Now those are rather broad terms. Those broad terms are a little more
refined in the Community acquis.
I wasn't able to find an explicit mention of it but I'm sure a state needs to actually be "Sovereign state" to be able to join the EU. The concept of statehood currently boils down to:
The constitutive theory of statehood defines a state as a person of international law if, and only if, it is recognized as sovereign by other states.
So that is an further obstacle for Scotland - AFAIK no country has made a statement whether they will recognize Scotland.
If they would fulfill those criteria the European Council (after consulting the Commission and "receiving the assent of the European Parliament" (Vote with absolute majority)) can then start the process of letting the state join. That needs to be ratified by the Member States.
There is a high chance that Spain would block because they don't want Catalonia to secede.
The Yes-Movement doesn't want to go to that process (Turkey has, for various reasons, started negotiating to get into the EU in 2005 and still isn't in).
As Scotland joined the EU in 1973 this is not the starting position from which the Scottish Government will be pursuing independent EU membership. Article 49 does not appear to be the appropriate legal base on which to facilitate Scotland's transition to full EU membership.
(http://www.scotland.gov.uk/Publications/2013/11/9348/10)
They propose that the treaties would just be changed so that Scotland just becomes a member.
Article 48 of the Lisbon treaty allows for changes to the treaty - The proposal would need to be accepted by a simple majority of the European Council and then needs to be ratified by all member states.
(Note that the Lisbon treaty ITSELF had some trouble getting through because of the popular vote in e.g. France & Ireland. I'm not sure if a change based on Article 48 would make a new popular vote necessary).
Nobody really knows what will happen when Scotland is independent.
BUT the referendum on September 18 is under UK law (Edinburgh Agreement) and states that Scotland doesn't become a independent country as soon as the election is counted but only after negotiations with the UK - and everyone hopes that everything will be sorted out during those negotiation period.