I was trying benchmarking package to test performance of sum using python built-in function. I tried converting array with 10^8 elements to a python list using PyCall.array2py and jupyter notebook got hanged I even tried doing the same on my terminal but it also crashed and I had to reboot. So does the Abstract array in PyCall.array2py has a limit or it depends on the system? Because I tried creating list with same number of elements in Python and it worked.
No, there shouldn’t be a size limit (other than memory). 10^8 elements work for me:
julia> @time PyCall.array2py(rand(10^8));
34.658970 seconds (100.00 M allocations: 2.235 GiB, 26.95% gc time)
(I double-checked, and PyCall calls PyList_New and PyList_SetItem, which take 64-bit integers … assuming you are using 64-bit Julia and Python.)
2 Likes
Thanks @stevengj 
I guess my poor machine is unable to support the operation:disappointed:
. Although it would be good if the function has some timeout limit or show some message while its processing so that it doesn’t looks like it is just stuck there.
Would it be an option for you to use Numpy arrays on the Python side?
Transfer of Julia arrays to Numpy arrays is much more efficient than converting them to Python lists.
I think @shriaas2898 might be trying to follow lecture 1 from GitHub - mitmath/18S096: 18.S096 three-week course at MIT , which benchmarks summation in C, Python lists, NumPy arrays, and Julia arrays?
But you don’t need 10^8 items for an accurate benchmark. (I used 10^7 in the notes, which is probably overkill too.)