I’m working on a version of a parser using Tokenize that lints as it parses, tracks variable declarations and is round-trippable (i.e. holds whitespace/punctuation in the syntax tree) and in its early formulation it’s ~4 times as fast as the scheme version even with the extra work. I’m sure a proper attempt at speed could do better than an order of magnitude faster.
7 Likes
After reading through this thread, perhaps this piece of missing information for @abhi18av (and anyone else interested) is this:
If you are asking “But only - why femtolisp in particular?” the answer would be, as I understand (see this), Jeff Bezanson wrote femtolisp before beginning to work on Julia. So it was the right tool for the right person at the right time, and there has been no strong need or effort to change since.
1 Like
There was a while very early on where we were using Guile (IIRC – but it could have been some other Scheme implementation) instead, on the premise that a mature, well-tested Scheme – and one that can compile to C code at that – would be better and faster. But we found that it was pretty crashy and that more time was being spent debugging Guile issues than working on Julia itself, so we switch to Femtolisp, which was totally smooth change, only 2x slower, took zero compilation time, and of course has the huge benefit that any bugs in the implementation get fixed immediately by the guy who wrote it – namely, @jeff.bezanson.
So ultimately the reasons for Femtolisp are:
- Scheme is excellent for writing parsers since trees (aka S-expressions) are its forte.
- Femtolisp is a small, simple, highly embeddable and remarkably fast Scheme.
- We control it (and by “we” I mean Jeff) and can fix any bugs we encounter.
Tangentally related, but we also used the Boehm collector as our GC for a while very early on but also found that to be crashy. It could have been our usage of it which was at fault, but we switched to a simple custom mark-and-sweep and that was much more reliable.
8 Likes
Mm hmm, thanks everyone for the wonderful resources! I’ve learned a lot from this thread 
1 Like
That’s interesting, using Guile initially. I think that after Andy Wingo got involved, the language got a VM and wonderful S-exp based assembly. I’m really starting to grok compilers and language design.
Could you tell me more about why mark-and-sweep and not some other GC? And, I thought people prefer not having a GC so they can fine tune their applications more so how does that work out with julia.
Yes @andyferris, I did come to know about femtolisp by this femto-emacs. Only then did I become really interested it the use of femtolisp in julia.
2 Likes
The Tokenize.jl - I wouldn’t have found it myself! Appreciate it, @ararslan
@ScottPJones and @stevengj Regarding this thing about C++, then how does it work out with other LLVM targetting languages like Rust etc? Isn’t LLVM supposed to be a level platform to build upon.
2 Likes
In Julia, certain types in certain situations can be statically allocated on the stack (I think this maps to isbits-types but I’m not sure). An algorithm that doesn’t allocate any heap memory can avoid the GC.
For example:
function fib_rec(n)
(n == zero(n) || n == one(n)) && return n
fib_rec(n - 2) + fib_rec(n - 1)
end
If you benchmark this function, you’ll find it never triggers the GC, even for a composite type like Complex{Int64}.
3 Likes
It’s as far as I know only used in the Julia parser (and easteregg julia --lisp not even documented with --help-hidden). I don’t think it’s going away in the any time soon (since I read, it’s not "speed-critical), but it’s been discussed getting rid of FemtoLisp, there’s already a parser for Julia written in Julia-only code, but I’m not sure it’s always in sync, since Unicode symbols occupationally get added to the parser.
And a different parser (currently done in C mostly I believe):
1 Like
Femtolisp does parsing and lowering of the julia code. There is interest in bringing the parser to julia in order to give better errors
3 Likes
Well, before Javascript existed, Scheme was the proposed language for web. Too bad management stepped in and said they would like something catchy like Java so Brendan Eich had to quit on the idea 
I sometimes wonder how the world would look like if Scheme/LISP would have been used.
5 Likes
You sound like Gerald Sussman (creator of Scheme), who said at Jeff’s thesis defense that he really wanted to program Julia using S-exprs
5 Likes
I think it’s actually possible that at some point we will add a Julia S-expression mode. It would be a relatively reasonable way to allow for Julia’s parsing and lowering to be written in pure Julia without bootstrapping issues.
5 Likes
Not the only Mona Lisa there (and none in Julia; well, except for that FemtoLisp dependency?), part of this commit:
one good way to bloat a piece of software is to add several
ASCII pictures of the mona lisa
2 Likes
I think you’d have to ask @jeff.bezanson about it.
1 Like
If you’re feeling motivated to do it, documentation is always great of course. I do think many if not most Julia users and developers would prefer more of the compiler to be written in Julia. So I don’t see high active demand for detailed femtolisp documentation, e.g. you won’t find it on our roadmap. It’s entirely up to you.
7 Likes
Shameless self plug, but I hope that you know julia can (with a bit of effort) run on microcontrollers as well, right? If using julia on microcontrollers is the goal, I’d pursue that route instead of using femtolisp directly, since that’s really only used for the parser right now. Aside from that, it’s not a core component of the compiler or the stdlib.
2 Likes