I have a similar issue with C++ objects (High memory usage when creating lots of temporary C++ objects) and my explanation is the same as @Benny’s. Unawareness of the Julia runtime of the actual size of foreign objects seems to be the root issue here.
Unfortunately, there seems to be no solution for the general case, only manually triggering GC every so often.