Does it exist?

Theoretically, yes.

What's that study about?
I've only heard of applications.

I don't know about the "theoretical" part, but statistics is arguably the most important branch of mathematics. Every science that deserves to be called science has statistics as its backbone because all measurements have some degree of inherent error, from sampling error all the way down to the Heisenberg uncertainty principle. Maybe "theoretical statistics" is just the theory of statistics, or statistics applied to something theoretical.

Theoretical as in "non-applied". Pretty much the same as pure mathematics; just the theory.

I'm hesitant to acknowledge statistics as a branch of mathematics because of it's focus on applications. While most branches of mathematics can be motivated by themselves, statistics require applications for that. Although a lot of the mathematics have been made due to practical needs, it have also been developed by mathematicians whose motive was learning mathematics and curiosity about it.
The parts of mathematics that are used practically does border to other sciences, but most branches have theoretical aspects, statistics seems to be the only exception. A lot of the currently applied mathematics have been developed without the intent of practical applications and was not used as such until long after the discovery, also this is (seemingly) different for statistics.

Edit: Summarizing, it's more like an application of mathematics (or a way to apply it) than actual mathematics.

It is true that statistics was heavily motivated by science, but that does not necessarily make it any less "pure" than other branches of math in my opinion. It is well-founded logically, and it is abstract. Dice have no memory, and neither do mathematical truths.

Aside from that, it is easy to forget that science and mathematics borrow heavily from each other in some respects, and that some of the greatest scientists are also some of the greatest mathematicians (Newton, Einstein, Hilbert, Gauss, and Euler off the top of my head dabbled in both math and science).

Back to the point, I don't really know of statistics being used for anything but science (almost by definition), but if it makes you feel better, not only is math heavily utilized in science, but science is being applied to mathematics in some cases (as with experimental mathematics)!

Well-founded logic does not make it mathematics. The discipline of logic is certainly also well-founded logic and even though it's based on set theory, it's not mathematics.

Einstein was not a mathematician. He was a great theoretical physicist and had good idea. The physical aspects of his idea was certainly great, but he needed help with the mathematical aspects of the theory.

The same person working with both science and mathematics does not mean anything about their relation. There likely are several people working at restaurants during the days and as prostitutes at night to earn enough money, but that does not tell anything about a connection between the professions.

Here's a qoute from the preface of the book Investigating Statistical Concepts, Applications, and Methods by Beth L. Chance and Allan J. Rossman (http://www.rossmanchance.com/iscam/preface.pdf):

At this point, it just becomes a matter of semantics. The distinction between what is math and what isn't is largely arbitrary and there's no well-established definition, so debating that would be pointless. Statistics is founded on logic, and there are a bunch of funny-looking symbols. Good enough for most people!

As for the quote, the author's focus seems to be the actual process of collecting data, which certainly lies outside the realm of mathematics. It's just that when someone says "statistics", the thing that pops into my mind is the actual analysis of collected data. Statisticians are certainly scientists that use the mathematical concepts made rigorous through the subject of statistics, but I wouldn't come to the conclusion that statistics is not really mathematics (whatever THAT is) just because statistics has specific applications in the real world.

Also, statistics doesn't necessarily have to be entirely useful! I've played around with some interesting distributions, probability problems, fuzzy logic, etc. It just so happens that most (or perhaps all) of what we consider statistics has direct real-world applications. I use matrices all the time in studying dynamics of aircraft, and in stress states of materials. Does that make linear algebra not math?

Einstein was indeed a theoretical physicist, perhaps not a mathematician. Maybe a couple of those examples are a bit misleading. As for the connection between physics and mathematics, physics (especially the theoretical kind) does indeed have a lot of overlap with math. In both, there are axioms, and then logical consequences. It's just that their goals are a bit different. In math, we come up with axioms and look for consequences (theorems). In physics, we are searching for the "axioms" of reality indirectly through experiment, under the assumption that reality does have some form of order to it. Then things get even blurrier when talking about purely theoretical stuff like M-theory and loop quantum gravity, which are certainly axioms that so far have been shown to work as a basis for reality. It's just that they aren't practical, and Occam's razor dictates that they be dismissed until they are proven correct.

And that's all I have to say about that.

Just out of curiosity, what would be your definition of mathematics?

Linear algebra is one specific branch of mathematics, that focus on the abstract objects called matrices (pretty much synonymous to the study of linear relations). Dynamics of aircrafts is part of a branch in physic (I don't know which), that happen to be using the mathematical results from linear algebra. The study of aircraft dynamics is not mathematics.
The difference between statistics and linear algebra is in this case is that collecting the data and (especially) interpreting the mathematical analysis of it is included in statistics, while the corresponding elements in the linear algebra example is included in physics rather than linear algebra.

The exact definition of mathematics isn't defined and I don't have one. I can however give vague description. Mathematics is a study focusing on the relation between abstract objects and their properties.