What can we do to make Julia grow fast?

Community
82

I don’t think its too early to advocate for those types of changes though. As @Antoine_Jardin said above, the package ecosystem will continue to grow and the sooner we can create a standard where functions work with variable labels and the likes, the easier it will be to compete with stata in the future.

I am trying to learn julia so I can help contribute to this ecosystem, but one thing that concerns me is that for my field, applied economics, Stata is by far the best tool, yet it seems like a package like GLM is trying to emulate R and Python, which is a less intuitive process for regression. On the other hand, I totally understand that it is difficult to implement xtreg y x [pw = weight], fe(village) cluster(group) while DataFrames hasn’t finalized their type system.

83

These are exactly the features that make Julia appealing to me, so I’d have to disagree. Einstein once said “make everything as simple as possible, but not simpler”. The Julia language is actually quite elegant in my opinion, and lives up to this standard by Einstein quite well. (that’s an opinion, of course).

The self-reflection and metaprogramming in Julia is actually very elegant. It takes some time reading the documentation and experimenting with examples to really learn all the features, but the self-reflection features of the Julia language actually help with that and make things easier by allowing you to inspect program statements abstract syntax tree, which is an amazing concept to me.

Those are the features that make me choose Julia over anything else. But hey, if you are actually quite happy with Matlab’s version of “simplicity”, then there’s nothing wrong with preferring that. Nobody is requiring you to use Julia’s metaprogramming features in your own code, but others find those features very useful and are drawn to Julia to use those features and they should not be discouraged from using it.

84

@chakravala, You’re missing my point.

I wasn’t talking about Julia as a language.
Julia is pretty simple and one could always pick the set of features one would like to employ.

I was talking about the part @Tamas_Papp said that what’s needed is more contribution all around.
For that, the bar set for coding of those packages should be not at the level of the developers as it is high and those are experienced programmers in addition to be utterly smart people.

NASA have great and talented people.
But I’m happy they do Space Shuttles and Toyota is doing my daily driving (Actually I’m happier with Alfa Romeo, but that’s another story).

Because things for the people should simple, namely mind the gap between your capabilities as the core user / main developer to those you want to join your eco system, if you want them as contributors and not only users.

85

No, I understood your point. You want to be able to read package code and you don’t want package developers to use metaprogramming as I understand it. My point was that the people who are drawn to Julia and make packages are also drawn to Julia and make packages because they can use features like metaprogramming. One of my packages for example would be impossible or very difficult to create without using the abstract syntax tree metaprogramming. The reason people use these features in their packages is because it lets them do the stuff they want to accomplish.

Also, not everyone’s motivation for making a code available for open source is the same. Just because someone made a source code available, doesn’t mean you have to expect everyone who develops a package to have the time available to make it easy to read. Of course, it’s beneficial to have easy to read code, but it’s not like people are being paid to maintain easy to read code here.

It would help the Julia ecosystem to grow faster if people did make their code easy to read though.

Nevertheless, I re-iterate my point that ensuring code-readability should not stop people from using features like metaprogramming in the packages they create.

86

RIP Space Shuttle. 1981 - 2011. never forget :pensive:

87

I don’t think that’s the problem, really. Of course the syntax would have to be a bit different, but for example the FixedEffects package already supports this kind of thing via:

reg(df, @model(y ~ x, fe = village, weights = weight))

What is hard is designing a general system which would allow fitting any kind of model while being consistent.
There’s an ongoing discussion in StatsModels about this.

The instability of data frames clearly doesn’t help stabilizing modeling packages, but the basics are already there and the required changes shouldn’t be that large as they only affect the interfaces. What we need is actually implementing the algorithms to estimate all of the model families we want.

1 Like
88

This is promising! I hope the ecosystem becomes more cohesive than R’s lfe4, felm, and lm. That is a reason why I support Julia.

2 Likes
89

This just came up, maybe of relevance to this thread:

6 Likes
90

An open letter to recruiters.

I often get emails from recruiters that want R/Python/SAS skills. I am thinking about drafting an open letter to such recruiters that tells them why they should list Julia as one of skills they seek.

Reasons to list Julia as a skill they seek

  1. This way it will get the attention of innovative and forward thinking anlaysts
  2. Positions them well for the future
  3. Positions them as innovative recruiters that are at forefront of latest industry trends

Just a thought at this stage, any suggestions welcome.

6 Likes
91

AFAIK recruiters do what companies ask them to do. You will probably get e-mails from recruiters looking for Julia skills when there is a market demand for it.

That said, many of these e-mails are unsolicited (= spam). The Julia community has been remarkably free from that, it would be nice to keep it that way (but that is probably hoping too much in the long run).

7 Likes
92

This is why my employer lists Julia under the desired skills. I got to edit a job description, wrote Julia as a bit of an inside joke, now it’s in every data science job listing through inertia.

I’m sure a Haskell writer could be a good addition to our team, but no one is advocating for it, so it doesn’t get added to the job description.

4 Likes
93

Throwing some wild idea here :slight_smile:

I’m working on the game industry, which has a huge population of programmers and they have almost zero interest in Julia.

I think if Julia becomes an official scripting language for Unreal or Unity3D Engine like Unreal script. Hundred of thousands game programmer will be interested in Julia, and quite possibly become a contributor.

So what is scripting language? now it will be a boring and lengthy explanation about realities of the game industry. so, stop reading this if you are not interested.

Game development is really hard and gruesome, there are many reasons to it but I will focus on one issue about data structure and handling.

Inherently, a game has to deal with a lot of unorganized data. For example, an RPG game has hundreds of heroes and monsters and each has tens of unique properties like HP, MP, stamina, attack power, defense point, critical probability… etc. And It is game designers job to manage all those chaotic numbers with some arbitrary rules of gaming. But most of game designers are not a computer scientist, nor mathematician or statistician. They are someone who wanted to make fun game. They found out later that in order to make a fun game, most of the real work is managing a staggering number of data, and even have to balance those data right :scream:

How much stronger 30 level ogre have to be compared to that 15 level gobline? make matters worse, how much loot ogre and goblins have to drop?

So game designers excessively use excel to calculate and balance all those numbers. Excel is great and all… but it cannot really handle complicate data, so many game development team has to implement other means of handling data.

One of those is teaching game designers to do some scripting with a high-level language like Unreal script or Lua.

As a game designer, I’ve done some coding with VBA, Lua and a little bit of C#. But after I learn to code in Julia I feel strongly that Julia is far more superior as tools for game designer than any other high-level language (Including Python). Julia is easy and intuitive for a non-programmer like me, and it can be as powerful as C++ if a user wants or needs to.

In short, if Julia is being used by game industry, I believe many game programmers will join Julia community. (Programmers won’t make games with Julia, but they will have to make tools for game designers, and solve problems game designers will encounter)

I know implementing Julia as a scripting language for the game engine is a far-fetched idea. And best way to spread Julia in the game industry is making huge success game like COC with Julia.

Thaks! hope you enjoyed reading this.

13 Likes
94

That’s a nice view you are sharing here, thanks for that. I was fascinated by the game technology and Unreal Engine for a while so your post made me curious. It is my impression that most programming in Unreal Engine is visual in nature, with optional hard coding usually done in C++, and the game developer has access to nice data structures in the visual environment that can be used to achieve many data management tasks without ever needing to use textual code.

So how do you view Julia coming into this ecosystem? Are you referring to perhaps implementing some functions or modules in Julia and calling them from the blueprint or event graph? Or using something like Cxx.jl to wrap the gaming engine and enable Julia scripting? Also Unreal Engine might a be a little too object-oriented so I wonder if some compatibility issues could arise when trying to wrap it in Julia. Having a game engine fully implemented in Julia sounds like a daunting task, so I wonder if there could be intermediate steps that actually add value. Feel free to contact me privately.

95

I’m not a programmer, so I fear I cannot answer your question correctly but I will try my best with pieces of knowledge I have.

Unreal advertise that anyone can make games with Blueprint. Maybe a small indie game or student projects can be done with Blueprint. But a commercial game project doesn’t have much use for Blueprint, except prototyping experimental game design.

Most of the game development is done by hard coding. But after building core system with robust language, we enable game designers to make actual contents with scripting. This is where visual scripting might comes in. However, visual scripting is slow, hard to debug, and impossible to track history.

For example, in order to game designer to make fight sequence like this, game designers have to be able to make things interact with script.

kill_zombie_count == 5 && activate_event(1)
etc…

96

Interesting. I would be surprised if these were not possible to do with blueprints though. But I agree that visual scripting is slow and hard to debug, I guess I have a lot to learn about this fascinating world! Maybe a topic to revisit in a year or so. Thanks for your response. I guess I should stop here before people following this thread get too annoyed.

97

@YongHee-Kim Thats a really interesting suggestion! I think maybe this should be its own new thread though?

98

Hi @outlace, thank you for the thread. If you are in an academic environment, one thing that I find very effective is porting class material to Julia and sharing them with the professors and instructors.

Talking about Machine Learning… I’ve just ported the homeworks of CS229 at Stanford this quarter while I was taking the class, and shared them with the instructors. Next year the most popular machine learning class in the world will have the Julia option :wink:

17 Likes
99

Just 3 humble little ideas.

  1. Inspiration: Blender has projects. They create small projects (movies) which help them polishing new features testing ideas advertising product. Producer, director, sponsors … Probably Julia could have similar projects. (maybe also as method to introduce new language features, new libraries etc)

  2. Engage universities: we have at least 2 teachers here (if you know them maybe you could @ them). Which could probably teach students with working on this community project. (maybe not only future coders but also testers, documentarists, team leaders too)

  3. One of thousands possibilities: I remember that I liked python’s grin grep like tool. But then I read blog where author told that ag written in c is much faster (read useful). Julia aims to be fast and super maintainable. We could show it! (or test it :stuck_out_tongue: ) There is still place here. (I am sure that you could add other proposals! :slight_smile: )

1 Like
100

If Julia wants to attract a lot of people it would be good to have a GUI with menus with the most important functions. And it would be great to have an ETL GUI interface like Rapidminer or Knime.

1500×1000 82.7 KB

101

Please don’t shoot messenger!

This post seems to be (at least to me) as good description about situation of GUIs in Julia …

So IMHO we miss several layers of functionality needed for something similar to Knime. :frowning: