I particularly like the ones about long context windows but not only.
That makes sense. The long-context work is especially interesting because it seems closely connected to where RLMs are heading. I’d also be curious to see whether some of these ideas could be practically explored in Julia, particularly around recursive inference and context management. That could make for a really interesting direction for the primer and the project overall.
Yeah, in general, long context seems to be a major direction, reinforced by better efficiency and of course new architectures. Medium to long term, I’m also expecting potentially a form of specialization and a need for orchestration. Honestly, in the short term, I would probably focus on infrastructure, on both the software and hardware sides. So, training Trinity Large TrueBase or Base looks like probably the safest option now. And development of Julia native model(s).
I agree that focusing on infrastructure first seems like a very practical direction. Beyond training a larger base model, I think it would be interesting to explore what a Julia-native stack for model development and inference could look like in practice. For example, having Julia-native tooling for efficient inference, context management, recursive workflows, and benchmarking could make it much easier to experiment with these newer architectures. It might also be useful to define a few concrete benchmarks so we can compare different approaches and see where Julia can offer real advantages rather than simply reproducing existing workflows.
With full respect, I don’t believe anyone intended to reproduce existing workflows, especially in the long term. And as far as my current understanding goes, Julia already seems highly competitive in terms of inference, at least for smaller models. But at least one important question remains open: what’s the legal framework?
I have tried to work with other recursive architectures in the past to be disappointed in the end and finding it a challenge. Maybe it was a bad experience.
Yes, working on tooling would be great, but I think at least another more more robust and probably necessarily larger model might be needed first to see the fruits of the tooling. I am thinking about making a 1B-3B model, with much more data and then maybe work on tooling. But reading the posts from Palli, I think that fast attention and quantization might be needed actually before. Tell me what you all think of this roadmap:
Fast attention > quantization > Train 1B model on more data > tooling
It would be a great help if some of the more academically inclined here want to join in and write some papers together during the process as well as plan the concrete next steps.
Let me know your thoughts
I’m afraid I don’t have much to add to the discussion at this moment. It’s been a pleasure. Thanks a lot. Wishing you the best.
Thanks, j_u. I really enjoyed the discussion as well. I think there’s still a lot worth exploring here, especially around how Julia could support these newer model architectures in a more native way.
I’ll keep thinking about the roadmap and the practical challenges around fast attention, quantization, and scaling the model. Hopefully we can turn some of these ideas into concrete experiments and see what works in practice.
Thanks again for sharing your thoughts, and I hope we can continue the discussion sometime in the future.
Absolutely, it seems like this space is developing rapidly. I’m always happy to make friends and exchange ideas. Again, it’s been a pleasure to have the opportunity to participate in this discussion.
It’s sensible, you mean FlashAttention. Then maybe a larger model, and let others do quantization? What do you mean by tooling?
I think training data, and what comes after pretraining, i.e. finetuning is very important. See my forked post. It can be done separately, and all such work reused here. I see over 50% of training data is on code, in a recent Microsoft model.
I really like the direction of this project. Getting a small GPT-style model running end-to-end in Julia, including training, tokenization, artifacts, and the REPL workflow, is a useful proof of concept.
The offline side of software is also an interesting area to explore, especially for tools that need to remain usable without a constant internet connection. I’ve been looking into the “Offline Mode” concept as well, and I’ve documented some related ideas here:
It would be interesting to see where KeemenaLM.jl goes next, particularly around local/offline inference, model optimization, and making the workflow easier for Julia users.