# Sparse arrays and function arrays in CUDAnative.jl

**URL:** <https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754>\
**Category:** GPU\
**Tags:** cudanative\
**Created:** [July 10, 2017, 8:28am UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754 "2017-07-10T08:28:32Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [July 10, 2017, 8:28am UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/1 "2017-07-10T08:28:32Z")

</div>

Is there any simple way to make, write to and read from sparse arrays on the device through CUDAnative.jl?

Also, is it possible to define an array of device functions that is moved to the device and then each function can be accessed and called by its index during a kernel call?

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [July 10, 2017, 8:47am UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/2 "2017-07-10T08:47:50Z")

</div>

We haven’t looked at sparse arrays yet, but CUDAnative supports arbitrary structs (as long as they are not mutable) so you can redefine your own sparse array type if necessary. You might also be able to use Cxx.jl and CUDAnativelib.jl to call cuSPARSE on the device, but I haven’t tried that yet so your mileage might vary.

> [@mohamed82008](#):
>
> Also, is it possible to define an array of device functions that is moved to the device and then each function can be accessed and called by its index during a kernel call?

Can you give a bit more context why you want to do that?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [July 10, 2017, 8:52am UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/3 "2017-07-10T08:52:48Z")

</div>

> [@vchuravy](#):
>
> You might also be able to use Cxx.jl and CUDAnativelib.jl to call cuSPARSE on the device

That would be awesome if it works.

> [@vchuravy](#):
>
> Can you give a bit more context why you want to do that?

I would like to use SymEngine to get symbolic derivative expressions then use them to define functions on the GPU to use for some optimization algorithm on the GPU. The functions should be accessible by the variable index, with respect to which the derivative was taken. There could be as many as 1M functions btw.

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [July 10, 2017, 9:26am UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/4 "2017-07-10T09:26:53Z")

</div>

> [@mohamed82008](#):
>
> That would be awesome if it works.

Just double checked and cuSPARSE doesn’t define a device level API so direct calls are not possible.  
But you can call cuSPARSE from the host and interweave that with CUDAnative code as long as the data formats are compatible.

> [@mohamed82008](#):
>
> I would like to use SymEngine to get symbolic derivative expressions then use them to define functions on the GPU to use for some optimization algorithm on the GPU. The functions should be accessible by the variable index, with respect to which the derivative was taken. There could be as many as 1M functions btw.

Hm, I see. The problem is that you have to compile that code, then take the function pointer (e.g. what `cfunction` does but GPU aware) and then emit a `ccall` to that function pointer. It might be doable, but I don’t think we have the infrastructure for that yet and I don’t know if that will be efficient.

If I understand you correctly you effectively want a switch statement since you don’t dynamically add new functions? For CUDAnative.jl tthe code hast to be statically know and we can’t do dynamic dispatch. So if you can precompute your function selection to something like

```julia
if fidx == 1
  f1(x)
elseif fidx == 2
  f2(x)
...

```

You could generate that code and don’t need to deal with function pointers on the GPU.

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [July 10, 2017, 9:36am UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/5 "2017-07-10T09:36:50Z")

</div>

> [@mohamed82008](#):
>
> The functions should be accessible by the variable index, with respect to which the derivative was taken. There could be as many as 1M functions btw.

Would it work for you to just directly splice the function call from the vector of functions into an expression, then eval the expression and execute that function (now with static dispatch) on the GPU?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [July 10, 2017, 9:45am UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/6 "2017-07-10T09:45:44Z")

</div>

@vchuravy

> [@vchuravy](#):
>
> But you can call cuSPARSE from the host and interweave that with CUDAnative code as long as the data formats are compatible.

So if I pass a sparse matrix to cuSPARSE and apply some decompositions, then solve some system of equations, the pointer to the solution I get back should be possible to pass to CUDAnative directly through a second kernel call so that basically all the work is done on the GPU, the solution doesn’t have to be moved back to the CPU, right?

> [@vchuravy](#):
>
> For CUDAnative.jl tthe code hast to be statically know and we can’t do dynamic dispatch. So if you can precompute your function selection to something like
> 
> if fidx == 1  
> f1(x)  
> elseif fidx == 2  
> f2(x)  
> …
> 
> You could generate that code and don’t need to deal with function pointers on the GPU.

That was my first thought, a horrendously long if statement in a single function but I was worried about the practicality of every thread/warp having to go through all the conditions to reach the intended one, may not be the most efficient approach. So I thought if I could define many device functions using metaprogramming and then pass them as an array, that it would be better. But if I understand your point correctly, until the functions are compiled, we cannot define the GPU pointer to point at them, so that’s why ccall is necessary as an after-step. Please correct me if I am wrong.

@sdanisch

> [@sdanisch](#):
>
> Would it work for you to just directly splice the function call from the vector of functions into an expression, then eval the expression and execute that function (now with static dispatch) on the GPU?

You mean to use metaprogramming inbetween kernel calls to define functions as necessary which then get compiled and spliced in a kernel call? That seems like it could work for me. But I am not sure I understand how this being possible fits with @vchuravy’s answer, could you please elaborate further.

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [July 10, 2017, 9:51am UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/7 "2017-07-10T09:51:59Z")

</div>

> [@mohamed82008](#):
>
> could you please elaborate further.

It’s just a different solution that only works if you can statically decide the dispatch for each function 😉  
@vchuravy solution would also work when your function calls depend on the value of the variables, but introduces branches, which would be especially bad if you have ~1 million branches 😉

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [July 10, 2017, 9:56am UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/8 "2017-07-10T09:56:27Z")

</div>

Note that generated functions work on the GPU as well. So as long as your meta code is pure, this can also work:

```nohighlight
const look_up_table = Function[...]
@generated function derivative_kernel(a, b)
    quote
        $(look_up_table[static_index])(a, b)
    end
end

```

You just need to figure out how to derive a static index just from the function argument types.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [July 10, 2017, 10:01am UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/9 "2017-07-10T10:01:22Z")

</div>

How about something like this

```julia
dfdx1(x) = 2x[1]+x[2]
dfdx2(x) = x[2]^2

dfdx = [dfdx1, dfdx2]

for i in 1:2
	@eval begin
		function kernel_derivative_x$(i)(x)
		    return $(dfdx[i])(x)
		end
	end
end

```

Actually in this code function `kernel_derivative_x1` and `dfdx1` are the same, my bad, so what I really wanted to do is to take `dfdx` to the GPU so that all the functions can be compiled and callable by `dfdx[i](x)` from inside a kernel call.

I believe a generated function is what I might be looking for, so your previous comment kind of answers my question. I will try and see what happens, thanks 🙂

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 10, 2017, 10:14am UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/10 "2017-07-10T10:14:41Z")

</div>

> [@mohamed82008](#):
>
> define an array of device functions that is moved to the device and then each function can be accessed and called by its index during a kernel call

Better do this in Julia, as @vchuravy and @sdanisch are explaining above, because our compiler doesn’t support taking the (device) pointer of a device function. Indirect calls also are a somewhat underused feature of CUDA C, and I wouldn’t know if NVPTX supports lowering it.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [July 10, 2017, 12:21pm UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/11 "2017-07-10T12:21:10Z")

</div>

Doesn’t CUDAnative support Windows 😅

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 10, 2017, 12:29pm UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/12 "2017-07-10T12:29:35Z")

</div>

> [@mohamed82008](#):
>
> Doesn’t CUDAnative support Windows 😅

Nothing holding it back, but none of the people actively working with CUDAnative uses Windows so it probably won’t work out of the box ☹ Definitely needs a build from source, and I’d presume that any/all of LLVM.jl/CUDAdrv.jl/CUDAnative.jl might need some tweaks to the build scripts to get everything working (@musm has been doing some work on this). But other than that, I don’t expect any big issues, unless Windows codegen would be somehow incompatible with our assumptions for reusing most of host codegen for the GPU.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [July 10, 2017, 12:32pm UTC](https://discourse.julialang.org/t/sparse-arrays-and-function-arrays-in-cudanative-jl/4754/13 "2017-07-10T12:32:08Z")

</div>

I see, I was looking for a good reason to go back to Linux anyways, seems like I found one 🙂