It turns out to be a GMT issue. I finally figured this out, with some help from the GMT experts:
You want to use @~ for special characters, @+ for superscripts and @- for subscripts. In my example, it would be written this way: (@~\155@~mol kg@+-1@+) .
Here is the list of Octal codes:
https://docs.generic-mapping-tools.org/6.2/cookbook/octal-codes.html