If you haven’t, I would start with Wasserman’s All of Statistics. If you’re not ready for that book, I would learn the required amounts of calculus and linear algebra first and then come back.
Statistics is so hard to learn because it’s a branch of mathematics that people pretend isn’t a branch of mathematics and so they end up teaching it very poorly; for example, people try to teach you intuitions instead of teaching you theorems, but the intuitions aren’t precise enough to prevent you from misunderstanding what the theorems really say. Stuff like “for all distributions there exists an N such that […]” versus “there exists an N such that for all distributions […]” is vitally important to correctly representing the claims of statistical theory, but those kinds of distinctions are exactly what tends to get lost in non-mathematical treatments of the field.