On Machine Learning and Programming Languages

[viralbshah]

Markdown, ideally done as a PR to the julia blog would be the easiest way to get it going.

-viral

[innerlee]

okay, working on it.
ref link

[viralbshah]

Thank you!

-viral

[ScottPJones]

@mauro3

My suspicious is that Chris cloned himself a few times. Even with Julia productive as it is, that is the only possible explanation!

I sure wish he could! Besides the great productivity of using Julia, and him being super smart, I think part of his success is attributable to how well he has inspired others to contribute to the project (much like Tom with Plots.jl, and Jeff and others with the Julia language!)

[MikeInnes]

This is very cool!

[rickhg12hs]

@viralbshah

I personally think that many ML frameworks are slowly becoming more like programming languages, and something like Julia + native AD + Flux/KNet + GPU / other hardware codegen makes for a really compelling ML platform.

It really does seem compelling! Unfortunately I have no GPU but I need some CNNs… Flux doesn’t have them and Knet takes forever. Knet takes 1000’s of seconds per training epoch on my I7 for its example MNIST CNN. TensorFlow.jl doesn’t pass it’s own MNIST tutorial. Mocha.jl and MXNet.jl seem to be the only ones that work. So for my projects I’ll need to use them or jump to Python/TensorFlow.

[denizyuret]

See https://github.com/ilkarman/DeepLearningFrameworks for some benchmarks. Knet CNNs were never optimized for cpus. This is not a high priority because you can’t train anything serious on a cpu. Amazon has gpus for 10-20c an hour on spot price if you want to experiment

[davidbp]

I think CPUs have a lot to offer, it is just that the competition in the CPU marked was almost non existent. Some frameworks (MXNET:129/25), even when the model is quite GPU friendly, are 4 times faster in a 4000$ gpu than in a 600$ CPU. What about using a 18 core xeon/16 core threadripper? I’m not sure results would be that different (if the code is optimized for both architectures). Nevertheless, the easier it is for scientist to swtich tools the better. The CPU/GPU war will continue and all companies will bring us more power for less price. If all of us use products from a single company (no matter which one) bad stuff is going to happen to the consumers.

[denizyuret]

Good point. I will look into other frameworks’ cpu kernels to see what
tricks they pull. In the meantime I think 1080Ti goes for about 600USD

[RoyiAvital]

I posted above on Intel MKL DNN.

Though they only guarantee optimization to Intel CPU it is a step forward from having only nVidia.
Hopefully it will work reasonably good on AMD as well.

[MikeInnes]

If anyone is interested in helping us wrap MKL DNN, or other good implementations, it’d certainly be a welcome addition. It would be great to ship these things with our ML packages where possible.

[innerlee]

A preview of Chinese translation is available here. Feedback are welcome! You can comment on this pr

[viralbshah]

Thanks to innerlee and reviewers, the post is live now.

https://julialang.org/blog/2017/12/ml&pl-cn

Would be great to send it to Chinese language tech sites. I note that csdn.net carried our why Julia post. Some other suggestions are in the PR above.

[Joshua_Bowles]

Wow. Than you for this blog post. I have been lamenting what seems like a standardization on Python in the ML/DL community (though it really seems more like historical accident than intentional engineering).

I’ve looked over F#, Go and Haskell for a decent combo of software engineering and mathematical computation. The story in Go is “train in Tensorflow, serve in go”… There are high quality and valiant efforts in Go to bring in data-science (https://github.com/gonum, http://www.pachyderm.io/, http://gopherdata.io/)… F# has https://fslab.org/getting-started/; haskell has http://www.datahaskell.org/… but the general communities of these languages seem not to be oriented around (applied) math-based computation, and so these projects are led by a small dedicated minority.

I’m looking to start my own company and none of the languages I’ve mentioned seem “just right”…

I installed julia a long time ago (v 0.2.1) and tried out a basic tutorial. I just came back to it last night to experiment and I think it might be the right fit fro me. I’ve already enrolled in the Coursera Julia Scientific Programming: https://www.coursera.org/learn/julia-programming and am super excited…

It seems like julia addresses many of these practical real world business concerns:

1. I don’t have a 16 hours a day only for development…
2. i can’t afford a buggy, slow, difficult, version-nightmare deployment;
3. I can’t be out on an island building core libraries
4. I can’t afford to spitball and release, only to come back and either re-write or spend countless hours figuring out how to improve performance/scalability of APIs.

In other words, it seems [please correct me if I’m wrong, sources and resource welcomed!]:

1a. julia is an easy enough language to be productive in [of course, subjective as this is…],
2a. a good deployment story… naive expectation here…
3a. again naive, but looking over all the packages seems like there is enough to get things built
4a. another niave view… not sure what the real deploy story is …

My other concern is Type systems. From the bit of reading I did last night it seems like you could prototype quickly using no type declarations, but as you moved towards productionizing things you could begin to stablize by building up a more obviously typed program? Does that seem realistic?

Thanks again for the blog post.

[ScottPJones]

@Joshua_Bowles

3. I can’t be out on an island building core libraries

@Joshua_Bowles

3a. again naive, but looking over all the packages seems like there is enough to get things built

It all depends on what you are trying to do in Julia. If you are dealing with strings, database access, data in different encodings (common with databases and CSV files), then you may end up like me, doing #3.
Hopefully that situation will be better soon.

@Joshua_Bowles

My other concern is Type systems. From the bit of reading I did last night it seems like you could prototype quickly using no type declarations, but as you moved towards productionizing things you could begin to stablize by building up a more obviously typed program? Does that seem realistic?

IMO, except for typing needed to correctly dispatch between different methods of the same function, for optimization purposes, or to handle differences such as between strings where code points are represented as a single code unit vs. one or more code units, then leave the type declarations off as much as possible for functions. Julia does a great job at inferring types, and generating very good code specialized for those types (which is why Julians concerned with performance talk a lot about type-stability )

For structs, there you do want to use concrete types for the fields if possible, and if not, parameterization is usually the best choice (also, for performance).

[ChrisRackauckas]

@ScottPJones

IMO, except for typing needed to correctly dispatch between different methods of the same function, for optimization purposes, or to handle differences such as between strings where code points are represented as a single code unit vs. one or more code units, then leave the type declarations off as much as possible for functions. Julia does a great job at inferring types, and generating very good code specialized for those types (which is why Julians concerned with performance talk a lot about type-stability )

The question is about typing for safety, not performance. Yes you can type for safety and you’ll get good errors, though note they are runtime instead of compile-time errors. In theory one could use the compiler tools to do some static analysis though, but I don’t know of anyone who has done that.

@Joshua_Bowles

i can’t afford a buggy, slow, difficult, version-nightmare deployment;

This is the killer feature to me about Julia. Honestly, I don’t understand build structures well, and if make fails I scratch my head and Google. It makes C/Fortran/C++ a nightmare for me to use. But Julia packages and libraries can be Julia the whole way down, and so that gets rid of all installation nightmares. In fact, package installation problems only seem to happen when there’s a binary dependency! So yes, these reasons make Julia a “productive C++” to me instead of a “fast Python”.

[Joshua_Bowles]

awesome, thanks.

I’ll clarify too: Not looking for a silver bullet here, no PL can do everything. What I’m specifically targeting here is that I’ll be processing a lot of airplane and airport and geo data, running the gambit on machine learning and deep learning techniques… I need/want the ability to write a backend API around my models… an API that can be used by other services (for example, a front end website).

I’m not looking to build websites with julia or crud apps. I need a great tool to build models, and way to serve those models via an API.

My current option is training with Python-Tensorflow, and then using the go API to tensorflow to serve the model as an API. (BTW, this bugs me cause now its a 3 language solution… python, c++, and go [and if you consider TF graphs as DSL or something like SQL then its 4-languages]).

I can easily bear the brunt of pre-processing data in go to overcome lack of support in julia (and of course contribute my own julia stuff…), but there is a critical mass of support that needs to be there to just get off the ground and it seems Julia already has that (for example, i needed to do some NLP 5 years ago and looked to go, but basic packages for statistics and matrix multiplication were not there so, i didn’t have the time to build those before building basic NLP libs too).

All in all, I’m psyched on julia, and think I can definitely make it work with what I’m doing.

thanks again

[ScottPJones]

@ChrisRackauckas

The question is about typing for safety, not performance.

OK, that wasn’t clear to me. For us, productizing meant making sure we scaled as we expected, and for safety, adding types I would say that adding typing doesn’t really help, the only thing IMO that helps with dynamic languages such as Julia (and M/MUMPS/CachéObjectScript that I worked on before) is really thorough unit-tests, backed by coverage tools that ensure that at least you’ve tested all paths with your unit-tests.

[ScottPJones]

@Joshua_Bowles

I’m not looking to build websites with julia or crud apps.

Yeah - I don’t know many people who are (we are using Node.js and TypeScript I believe for UI).
I do hope there will be some better support in the future for NLP in Julia, I think that’s a more promising field for Julia than as a language for building websites.
We are doing all sorts of fun stuff in Julia, and although we’ve had to use other languages, and roll our own support for some core things we found lacking in Julia, we’ve been fairly pleased with it.

[Joshua_Bowles]

nice.
Yeah I am thinking here about type safety… But it’s also not something I’m dogmatic about. Rust/F#/Haskell has a lot of interesting things you can do with Types… but it’s kind of not clear how that equates into productivity, safety, and refactoring.

I think static type systems and/or static analysis is super important for the ML/DL future… but it’s kind of not obvious where that should happen (e.g., Python is dynamic typed, but TensorFlow has a static type system for building the graph… so it seems a matter of interpretation if that is overall “static”).

I’ve also deployed plenty of ruby on rails and python projects, so my sense of what is “easy” to deploy is definitely a bit more weathered than “put the binary out on the server”, from this perspective it seems julia deployment is good.

I look forward to moving through tutorials and trying out Knet/Flux

Thanks for the reply