Dear all,
It is time again this year to launch our annual User and Developer Survey. For those who didn’t see last year’s survey, please do see: Julia User - Developer Survey 2019
We plan to launch this survey in a few days. I’m sharing it here for feedback and suggestions on improvements. Results will be shared during JuliaCon like last year.
Thanks,
-viral
I would add in the lists of technical aspects most/least liked (I think that there may be opinions on both sides):
And among the non-technical:
In question #15, maybe asking for both “Atom” and “Juno” is redundant.
And I don’t know if it is suitable for the objectives of that survey, but perhaps it could be interesting to ask about convenience or necessity of professional, paid support for users (either as part of the existing questions or as a new one).
.I think SQL should be listed. it may not be that relevant to Julia now, but since it is often used for data management and Julia has a sizable data science component, it would be good to bring the two closer together.
Many questions lack an answer “none of the above” or “other” which is important. If my preferred answer is not represented, I have to choose one that I do not identify with to continue.
One - you could ask a question about what the users would like to see next in Julia – what would they prioritize for future Julia enhancements?
Two - localize the survey in additional languages. Perhaps the user groups could help with the translation. Perhaps just target a few this year – say, French, Spanish, German, Russian, Hebrew, Japanese, Mandarin, and Hindi. Or pick 3.
Three - an open-ended question on, “What are some interesting things you have done with Julia?”
When asking questions about what you like/dislike about Julia I think it is useful to make the distinction between
My guess is that most of us have little issue with the language itself (we all love it!). Instead, what I believe we are mostly concerned with is how the process of developing Julia code can be improved.
I’d say absolutely “yes” to this, if it helps to obtain answers from a more representative sample of users, not biased by the language of the survey. But before asking people to do that, we should consider how much support has Julia for each language, rather than the size of the speaking population.
Only for languages with a significant support (updated translation of the manual, active discussion forums, etc.) it would be worth to translate the survey, lest answers would be lost from users because the survey is not in their language. I don’t know if there are languages in that situation right now.
And expanding on the previous comment: at least if translations of the survey are done, it would be intersting to add a question about the quantity and quality of support of Julia in their speaking language.
Questions
- What is the country where you currently llive/workive / work?
- What is the country where you are originally from?
- In what language(s) are you fluent? Please select all that apply.
- Which of the following best describes your race or ethnicity?
- Are you a: X Man Woman
- Do you identify as LGBTQ?
- Do you identify as underrepresented in science or computing because of your:
should not be asked. (Q 32+34 could perhaps be asked but I prefer not.)
They don’t provide meaningful insights, nor do they adress anything of importance to the Julia language.
If the survey should adress social aspects, it is not deep enough and in general I think, the survey in this form is not able to do this and it shouldn’t.
Stay on the technical aspects of Julia on this survey.
In general, before you start a survey, ask yourself, what do you want to know, what is important, what are the main questions I have, which I want to be answered with the outcome of the survey. With these questions in mind, the survey should be designed. Putting up a survey with just a number of questions, like “oh this could be interesting, and this, too…” doesn’t typically result in good answers and good coverage. A survey should be short (max ~15min), but a short survey must be focused. Better create multiple short surveys for different independent aspects, launch them at different times. Good surveys can be relaunched multiple times to create information about how the answers develop over time. This can not be done with changing surveys, they need to be (mainly) stable for this.
Use neutral questions:
Don’t ask: “Do you like this feature?” Don’t like! - … - Like! - Don’t know!
But ask: “If you think about this feature, which would best fit your personal impression:” Doesn’t fit my needs! - … - Completely satisfying! - …
Possible number of answers should be not too few and too much. Like above, just “Don’t like” - “Like” are too few, but allowing grades of 0(hate it) to 10 (love it) is too much. Typically 5 answers plus “no answer” (together 6) is good.
5 answers (+ “no answer”) do have a middle answer = “I am neutral”. This is not always desired. If you want to force a side, use 4 answers (+ “no answer”).
“No answer” should always be available! This is important to distinguish between “the user can’t or don’t want to answer this” or “the user forgot to answer this” or "the user is neutral about this (if the middle/neutral answer is available, see above).
…stopping here now.
Generally well-designed surveys ask for demographic information because it helps to adjust the results for statistical analysis (response rates, representativeness, reducing noise in Y/Y changes, etc). This is quite standard in survey statistics.
Also, Q35 and Q36 are useful for understanding representation of various groups in computer science/programming/the Julia community. This is a relevant issue that people care about, so it is useful to have data.
imho:
in the Golang survey …
there is a useful question:
in my case:
on the other hand in the “2020 Julia developer survey” I will answer
so my SAS ‘familiarity’ info will be lost…
Similar info in the GoLang Survey
“Respondents who have been using Go the longest have different backgrounds than newer Go developers. These Go veterans were more likely to claim expertise in C/C++ and less likely to claim expertise in JavaScript, TypeScript, and PHP. One caveat is that this is self-reported “expertise”; it may be more helpful to think of it instead as “familiarity”. Python appears to be the language (other than Go) familiar to the most respondents, regardless of how long they’ve been working with Go.”
I disagree here.
Generally a well-designed survey may ask for demographic/gender/wealth/… information because it may help to adjust…
This can be true and if some questions are asked to help e.g. to decide for representativeness, this can be a sign of the design being well of a specific survey. Turning this into
well-designed surveys ask for demographic information
is wrong.
In the current form the special questions here do not help “for understanding representation of various groups”. For this the survey as a whole should be representative but the bias is already integrated. The numbers from those questions will be questionable. Reduce those questions to a minimum: maybe ask for gender (female,male,diverse,…), but asking e.g. for LGBTQ (Q35) is already discriminating like e.g. asking if you are black, yes or no. And I am absolutely sure, that it will not help to adjust for anything.
The most important aspect of a well designed survey is, that the designer knows what question to answer with the outcome of the survey. Only with this known, you can add meaningful demographic questions for statistical purposes.
But, I think, we do not need to have the perfect survey. A not so well designed, lets say, naive survey, is a good thing too and gives a lot of good numbers to help for future development, to go into the right direction. What I have said about a survey in general, should be considered if there is time and energy to do this (it is quite some work to invest) and it can not always be done. And if this is the case, be aware of it, and skip questions which just pretend to be meaningful. If you still think these questions are important, I am fine with it, too.
You are perfectly free to not answer, as well as to abandon the entire survey, should you somehow feel uncomfortable with any question. The survey is further anonymous, so whatever you answer can not be held against you, or used to discriminate against you.
In the end, what you don’t ask you will never know. If the community wants to strive for openness an inclusivity, it’s important to know where it fails in order to know where improvement is needed.
I am sorry but I do not understand this sentence. One can always pick “prefer not to respond”, and the survey is anonymous in any case. I don’t understand how you imagine that any answer can be used to discriminate against someone in any sense.
In any case, these questions were asked last year too, and were found to be useful. Please check out
But for Julia it would be best, to have as many people as possible to answer the survey! Asking questions where some people may feel uncomfortable and abandon the survey is, again, introducing bias, and this is not what you (Julia) want.
Side note: Don’t use so much “you”. It is not about me. I, personally, have no problem at all with this survey, I just want to see it better designed for the benefit of Julia, the language and this community.
Being discriminating in something does not mean, somebody has used something to discrimate against someone.
Asking discriminating questions or in a discrimating way is not yet that is has been used against somebody. It is just bad conduct and should be avoided.
The act of being used against somebody lies in the data itself, not in the question. So don’t raise the data if not necessarily needed.
This is a general aspect. I do not say, this community is discriminating, I would say the opposite!
In question 6, I would pick one of the following.
It’s unclear to me what the difference between them is. I think they should be one option.
Personally, I would be much more inclined to adopt Julia as my language of choice and take the survey if the survey and the community would focus, entirely and exclusively, on technical merits and possible improvements.
I realize this comment from a beginner probably does not carry much (if any) weight, but I still felt compelled to make it.
I would add a question about if you have develop your own package, and maybe the number. Also, if you have develop your own package the current state
(register, not registered but installlable from Gituhb, Deprecated). Also, it could be interesting in that case to ask about the packages as documentation level: (using Literature and/or Documenter, commenting the functions, or pending), or any other interesting ask about developing packages. There are many packages and several people in the survey will have develop packages, it should be good to receive their feedbacks about it.