Dagger use cases?

Obviously there are no upper-layer applications .
As the user, It can’t be used to solve real problems.

I highest priority need DataFrames.jl on Dagger, OnlineStats.jl on Dagger, Transducers.jl on Dagger, To achieve spark’s or flink’s functions…

1 Like

First, this JuliaHub tool isn’t a great metric to determine whether there are users of Dagger: GitHub - JuliaFolds/FoldsDagger.jl, GitHub - JuliaParallel/DaggerArrays.jl: Experimental Distributed Arrays package, and GitHub - JuliaGPU/DaggerGPU.jl: GPU integrations for Dagger.jl depend on Dagger, yet they don’t show up in “Direct Dependents”, and at a minimum, DaggerGPU is a registered package. Obviously this listing is inaccurate and needs fixing. (Edited above: This is a JuliaHub tool, not a GitHub tool, and pointed out that DaggerGPU is the only registered package).

Second, just because DataFrames, OnlineStats, etc. don’t directly depend on Dagger, that doesn’t mean that Dagger can’t be used to solve real problems (which is a rather rude statement to make, but I’ll ignore that).

Dagger’s DTable, for example, wraps other tables and distributes them, which thus doesn’t require the ecosystem to depend on Dagger directly; instead, users can use any table type they want to back their distributed table. You can then use OnlineStats to do processing of that distributed table, computing statistics and performing aggregations, because the DTable is composable. No Dagger dependency in OnlineStats needed!

Similarly, Dagger.@spawn can wrap basically any function, and doesn’t require the function to know anything about Dagger to work; this is the same as how Threads.@spawn doesn’t always require the spawned function to know that it’s running on a different thread; the composable nature of Julia allows both of these spawning mechanisms to “just work”.

It’s true that Dagger does need to see further ecosystem uptake to become a better composable foundation for distributed computing, but it’s still possible to use it today to solve problems. My original post that this thread was split from was intended to query the community to find out how that’s currently working out, which will help me inform Dagger’s development.

11 Likes

That’s not a GitHub screenshot, it’s a JuliaHub one. It only tracks other registered packages, which is a small sliver of usages. Indeed, the package stats show it’s been downloaded 5000 times by 2000 unique IPs since Sept 1.

@zsz00, your initial comment here is far harsher than necessary; there’s no need to be so confrontational — especially when the main developer is asking how to make the package better for its current users.

5 Likes

I guess it’s strange that DaggerGPU does not show up on the dependents page though? OTOH, Dagger does show up on the dependencies page of DaggerGPU https://juliahub.com/ui/Packages/DaggerGPU/5ydcV/0.1.2?page=1. Maybe it’s trimmed off by the heuristics on JuliaHub or something?

3 Likes

Sorry, updated the text to point out that this is a JuliaHub tool! Also pointed out @tkf 's point that DaggerGPU should show up.

2 Likes