Is there a size limit of array in PyCall.array2py?

I was trying benchmarking package to test performance of sum using python built-in function. I tried converting array with 10^8 elements to a python list using PyCall.array2py and jupyter notebook got hanged I even tried doing the same on my terminal but it also crashed and I had to reboot. So does the Abstract array in PyCall.array2py has a limit or it depends on the system? Because I tried creating list with same number of elements in Python and it worked.

No, there shouldn’t be a size limit (other than memory). 10^8 elements work for me:

julia> @time PyCall.array2py(rand(10^8));
 34.658970 seconds (100.00 M allocations: 2.235 GiB, 26.95% gc time)

(I double-checked, and PyCall calls PyList_New and PyList_SetItem, which take 64-bit integers … assuming you are using 64-bit Julia and Python.)

2 Likes

Thanks @stevengj :smiley: I guess my poor machine is unable to support the operation​:disappointed::sweat_smile:. Although it would be good if the function has some timeout limit or show some message while its processing so that it doesn’t looks like it is just stuck there.

Would it be an option for you to use Numpy arrays on the Python side?
Transfer of Julia arrays to Numpy arrays is much more efficient than converting them to Python lists.

I think @shriaas2898 might be trying to follow lecture 1 from https://github.com/mitmath/18S096 , which benchmarks summation in C, Python lists, NumPy arrays, and Julia arrays?

But you don’t need 10^8 items for an accurate benchmark. (I used 10^7 in the notes, which is probably overkill too.)

Yes @stevengj I was exactly doing that😅
I have now settled for 10^6 which works fine in all the scenerios. And as @lungben mentioned numpy arrays would be better in otherwise scenerios.