I particularly like the ones about long context windows but not only.
That makes sense. The long-context work is especially interesting because it seems closely connected to where RLMs are heading. I’d also be curious to see whether some of these ideas could be practically explored in Julia, particularly around recursive inference and context management. That could make for a really interesting direction for the primer and the project overall.
Yeah, in general, long context seems to be a major direction, reinforced by better efficiency and of course new architectures. Medium to long term, I’m also expecting potentially a form of specialization and a need for orchestration. Honestly, in the short term, I would probably focus on infrastructure, on both the software and hardware sides. So, training Trinity Large TrueBase or Base looks like probably the safest option now. And development of Julia native model(s).