In general a Meta-Theory is "theory about theories":
A metatheory or meta-theory is a theory whose subject matter is some
theory. All fields of research share some meta-theory, regardless
whether this is explicit or correct. In a more restricted and specific
sense, in mathematics and mathematical logic, metatheory means a
mathematical theory about another mathematical theory.
In fact the term Meta typically represents the abstraction layer of a concept (any concept; thus the Meta version of this very site) (see this for Etymology).
I would argue that there is no well-defined concept of Meta-Theory for the Political Science, or International Relations fields of study. Currently different authors give it a different meaning even if slightly. In any case if you need a base you can check one of the encyclopedic sources (Blackwell Encyclopedia of Sociology):
A metatheory is a broad perspective that overarches two, or more,
theories. There are many metatheories – positivism, post-positivism,
hermeneutics, and so on – of importance in sociology and other social
sciences. Two of the best known and most important are methodological
holism and methodological individualism. (...)
You will also find literature regarding this subject, namely:
That being said as a rule of thumb a Meta-Theory in International Relations is typically associated with challenging the assumptions made by a theory, for example (see also: Hermeneutics):
By re-evaluating the intention of persons of interest (politicians
for example) as to differentiate between idealism and opportunism.
By challenging the "intemporal" assumptions of a [social, economic, etc.] theory.
By putting into question the current historical perspective of a
political movement.
- etc.
EDIT: a possible example for a Meta-Theory is challenging the basic assumptions of Marxism (with origin in the works of Karl Marx and Friedrich Engels). For example Marx had contemporary philosophers that considered themselves Socialists but criticized Marx "Socialism" (thus "Marxism" and not "Socialism"). Another example are the multiple interpretations of Adam Smith Invisible Hand and the assumptions they make.