How to check if my gridded X and Y are inside a polygon?

Not really.

T = 273u"K" + 15u"K"*randn(501,501)
x = range(-5,5; length=501)u"cm"
y = range(-5,5; length=501)u"cm"

plot(x, y, T; st=:surface)

Of course you need all of your samples, but you really don’t need to gridify x and y.