@TimG. I’m with you—I speak sort of a domain-specific language version adapted to the basics with a few more advanced topics. I’m never going to cease to be a newbie insofar as the more advanced topics to, and I’m good with that.
Ouch. Point taken and corrected.
Which point? Just confused because my last comment was responding to someone else’s argument.
1 Like
“Substandard code” (in the unedited post, I’m mortified)
I don’t think Benny was referring to technocrat’s code when he wrote this, but it does remind me of a similar feeling.
I’m still quite a novice programmer myself, especially around setup and tooling. I wrote a Julia package to simplify and automate some of the tedious actions my co-workers still do in Excel. I wrote a .bat file they can double click to call the Julia package manager and set up the proper environment. I wrote detailed (formatted and screenshot) instructions on how to download all the needed tools, start Julia, and call my package functions. I tested everything myself and then sent it out with much fanfare.
Upon first use, a Git permission security issue I never saw coming stopped the whole project in its tracks. I’ve still not taken the time to figure out implementing a solution.
I tried very hard to argue and prove coding to be easier than their current Excel methods, but now because of this error, it’s my fault that they are turned off to Julia and coding in general as being too complicated to be worth trying.
I find it unfortunate that there is a requirement for them or me to learn to troubleshoot all this auxiliary tooling before they get to try running Julia code themselves. (Here I’m using “tooling” very generally to mean everything else we’re forced to interact with to get Julia code to run.)
5 Likes
There is no reality where everyone can skip the work of “auxiliary tooling.” Anyone who has made an easily downloadable executable or app has done that work to ensure end-users on a wide variety of platforms only need a few clicks or commands. That includes Excel, and it shouldn’t be surprising that people prefer better software shipped by industry pros. To reiterate, I don’t think there’s anything wrong with your colleagues using Excel. If they’re able to do all their work in visible interactive spreadsheets, it’s unlikely they’re doing anything on a scale worth writing programs. I’ll go even farther; if Excel is sufficient for their purposes or you have not proven that Julia offers significant enough benefits for them, they have no good reason to try Julia.
3 Likes
My rule of thumb for spreadsheets is the visibility test: if you can see the entire sheet on a full size screen you should be ok with care. If you have to scroll and scroll along either axis, not.
Back when mortgage backed securities and their derivatives were still a thing, I heard a tale of an investment bank that was running a model on a loan tape with 10^5 records x 50 variables that took overnight in Excel.
I think this may under-estimate the sheer inertia of people sticking with the familiar - and the consequent inefficiencies involved.
In my work, we run a specific project every year. As part of preparation in previous years, a team member downloaded a frequently-updated dataset from a public website every day (as a CSV) and then manipulated this manually in Excel to highlight different aspects of the data for different team members. This took hours each day, was repetitive and error-prone. No one considered this wasteful. It was just doing what needed to be done with the tools available and Excel sufficed.
This year, a new team member thought there was probably a more efficient way to do things and involved me for the first time. With a comparatively small investment of my time, I wrote a Julia script to automate the whole process. The script runs in three minutes and saves one team member literally hours of tedious and repetitive work every day and the product quality is higher and more reliable, too.
I don’t highlight this as an example of “why Julia” specifically, but rather to illustrate by example that, even if Excel is considered sufficient, it may be far from optimum. Reducing barriers to entry increases the chance that the better ways offered by Julia will be adopted. The difference between what I offer my team and @Nathan_Boyer offered his is that I need to run my script myself. I haven’t tried to package it to be shared in the team - the barriers are too great.
I think similar scenarios where familiar tools are pushed further and further as a result of inertia and lack of knowledge of alternatives are very common. The resulting inefficiencies may be significant.
8 Likes
I’ll weigh in again with a war story. In 1980 I found myself as the new planning director for a fishing port in far, far western Alaska charged with organizing a capital improvements program for badly needed infrastructure. The city employed 150 with a bi-monthly payroll. It took the City Treasurer, the City Clerk and four assistants a week to put out each using a “one write” system. The rest of the accounting and budgeting process was no better. I recommended automating the process, which led to installing an IBM System 34 mini-mainframe. Staff was skeptical, to say the least. But that skepticism turned to acceptance when they saw it in action. Within a few cycles it took one junior clerk a half-day. That’s just another example of @TimG’s experience. Perhaps the moral is that things just have to get bad enough that folk are willing to take the plunge.
4 Likes
I’d say those examples align perfectly with proving “significant enough benefits”, unless they see job security in manual inefficiency. If we’re trading horror stories, I have one about introductory programming gone wrong. Someone on a forum (not about programming) complained that they were taking days to make a plot in Excel and they couldn’t even make it look as nice as the plots in some related publications. Days after a recommendation of Matplotlib via the beginner-friendly Anaconda distribution, they proudly showed a plot but complained it didn’t save any time or work. We asked to see the code, which was structured like:
from matplotlib import pyplot as plt
x = [0, 1, 2, 3, ...]
y = [8.3135, 2.9587, 4.2352, 9.9654, ...]
plt.plot(x, y, ...)
What took days was manually copying several thousand values from a CSV file to the Python lists. We didn’t bother to dig into the likely similar process in Excel earlier because this individual was known for being stubborn and irate, and they did have some choice words for us “withholding” the existence of numpy.loadtxt, something they could’ve easily learned on their own if they didn’t dismiss documentation and tutorials as “elitist”.
If I had to glean a relevant point from that extreme anecdote, it’s that lowering barriers to entry is only part of the picture. It is possible for an app to automate everything to a couple clicks around a Julia script plus metadata (I’m imagining something like PlutoDesktop but for general-purpose, non-reactive scripts), but refusing to learn how to install packages and manage environments ourselves is inherently limiting, and the general combination of insufficient guidance and willful ignorance will ruin the effort in unpredictable ways. An unwillingness to learn is not something we can do on our end though, so we should probably hammer out more best practices on how to share Julia code and instructions. Troubleshooting is best done in advance and with support.
4 Likes
I think this is a relevant example and we have all seen similar.
While in general I think it is a good idea to engage non-technical users, the only model that a free forum can support is onboarding: it is recognized (by all parties) that one needs to learn new skills and concepts, and help and resources are provided to achieve that. Everyone is patient and supportive, because the implicit assumption is that there is learning taking place, and after a while the newcomer will be part of the community, and may possibly help others. I have seen many people on this forum who started out as Julia newbies, and are now giving excellent advice to others.
But this model is not sustainable with people who are unwilling or unable to invest the effort (note that I am not passing judgement here, it is perfectly fine not to want to learn Julia). They are asking for tech support, not help with their Julia journey: don’t give me links to this and that, give me the answer. Doing this for free is emotionally draining; it is only sustainable in a context where people get paid for their work.
This kind of service has to emerge sooner or later, as it did for all other languages. There is a niche between large projects for which one would engage existing companies (eg JuliaHub) and tiny questions on Stackoverflow or here which you can expect to be answered for free.
5 Likes
Which is in many cases can already perfectly be done by AI. Recently Gemini (paid version) helped me to get a driver working for a notoriously problematic piece of hardware on an old Linux notebook. We went “together” through several iteration loops, and I do not think a human support would do the job better. This was exactly the case where I only needed the answer - I am not interested in Linux kernel intricacies.
BTW “non-technical users” is not precise term. The majority of my former and current colleagues are not interested and would avoid programming, while being technical people - chemists, all kinds of engineers etc. Excel suffice. It does.
3 Likes