Turns out you can also use multiset_permutations from Combinatorics. The input set needs N² 1 values and the same number of -1 values. Choosing any N² subset of these (in every permutation) will give the same as the above solution.
for perm in multiset_permutations([ones(Int, N^2); -ones(Int, N^2)], N^2)
S = reshape(perm, N, N)
end