AI-generated enhancements and examples for StaticCompiler.jl

Tooling
21

Have you read, understood and reviewed the code yourself? I don’t have particularly strong feeling about using these tools to generate code, but I do have strong feeling about having responsible humans review and understand their results.

I suspect you haven’t reviewed the code yourself here, so that makes me very hesitant to do the huge amount of work to even begin evaluating the quality, design decisions, and potential failure modes.

Showing up with no standing and no prior engagement and then dumping gigantic chatbot generated PRs that do tonnes of different things that could easily be broken down into smaller chunks, and then falsely claiming that it couldn’t possibly be broken down into smaller pieces is not a good way to start this process or get reviewers on board.

14 Likes
22

The PR can certainly be broken down further, starting with the Julia 1.12 upgrade.

I intend to get to it next week.

I the meantime, feel free to run through the examples using my build.

23

The added features don’t work. None of the analyses I looked at in the src/analyses/ directory do what they claim, and the “tests” all seem to check that they return nothing. I found this funny:

function estimate_method_targets(func_name::Symbol, types::Tuple)
    # Conservative estimate
    # In a real implementation, we'd query the method table
    return 2  # Assume multiple possibilities
end

Did you read any of this?

14 Likes
24

I will have to.

1 Like
25

It knows what it’s doing but tends to be… lazy.

› Why not actually implement all the analysis and do the tests properly?

• Fair point—we should either implement them or drop them. I can replace the

  • Devirtualization: walk code_typed for :invoke/Core._apply_iterate calls,
    collect callee types, count concrete vs multi-target dispatch using methods/
    hasmethod against the specialization signature. Populate CallSiteInfo with real
    num_targets, can_devirtualize, and dynamic_dispatch.
  • Escape analysis: stop guessing sizes; use code_typed to count allocation nodes
    (:call to Array, Base.allocatedinline, Expr(:new)), and mark escapes/can_promote
    based on SSA use analysis (does result feed stores or leave the function). Size
    estimates can use element type and literal dims when available.
  • Monomorphization: flag abstract parameter types by inspecting the signature and
    code_warntype/code_typed body to compute a simple specialization factor (e.g.,
    ratio of concrete to total slot types).
  • Constant propagation: scan IR for literal operands and dead branches; count
    and count potential leaks and double frees by following SSA uses and return
    paths.
  • Suggestions: tie into the above metrics (e.g., only suggest stack promotion when
    escapes==false and can_promote==true).
  • Tests: assert on report fields (scores, counts) for crafted fixtures that
    exercise each path (single vs multi-target call, escaping allocation, missing
    free, constant branch).
26

Ultimately, what is missing is a process to have AI self-review while keeping itself honest.

Plus, help a human review the code.

Yes, you can keep AI honest but you have to remember to prompt it which I didn’t at the very end. Something along the lines of “are there any tests stubbed out and have you taken any shortcuts?”

27

Well, I’m feeling pretty justified in not wanting to review the PR.

For now, I’ve closed it as slop, but please feel free to try again and submit discrete PRs that do discrete things like e.g. upgrade to 1.11 / 1.12, or add certain features / analyses.

That said, if you do go and submit new PRs, please actually review and understand the code you’re submitting first. If I wanted an LLM to generate 20,000 lines of unreviewed code with fake tests, I could do it myself.

6 Likes
28

Complaining that the LLM is lazy and then (presumably) asking it what we could improve is pretty meta :rofl:

4 Likes
29

It will literally say that this is too hard, I’m gonna take this easier option and stub this out for now. This is not the best example but it’s all I saved for now. I don’t know why it does that but it does.

So you do, actually, have to ask it if it has been honest and it will tell you if it has not been and exactly where.

30

(post deleted by author)

1 Like
31

Much of the actual work during software development is establishing a strong mental model of how your changes work together as part of a larger system and being able to communicate them effectively to the people reviewing and approving said changes. Whether the code is written by an LLM or not if you can’t do this chances are you need to do more work before having other people look at your changes.

11 Likes
32

@joelreymont I refer you to Chris Rackauckas’ excellent article on how to incorporate AI into your workflow.

I don’t know how aprocryphal or misquoted this is but a quote I enjoy is “A couple of months in the laboratory can frequently save a couple of hours in the library.”

I encourage you to employ the strategies Chris concludes on in his article (“library”) to save you the time you’ll spend (“in the lab”).

Though with the quality of what you’ve submitted, it does look like you spent little time at all reading “your” own code.

Us in the Julia community are going to repeatedly reject your PRs unless your present something human-provably worthwhile.

We’d love to have you as a genuine addition to the Julia community. But please, I emphasise: “genuine.”

33

I wouldn’t be so quick to dismiss this. You favor the term “cooking”, and to press the analogy, a meal isn’t usually served as a monolith. There are several courses, each dish has several components and sides, diners can savor a bite at a time. Emergency ration bars are technically monoliths, but they’re actually not trivial to make. Chances are you do in fact have several dishes if you change your perspective.

For example, one of the features you introduced is a heap allocation check. Diving into one example arr = [i for i in 1:n] in your blogpost, the underlying routine is:

julia> analyze_escapes(bad_code,(Int,))
EscapeAnalysisReport(AllocationInfo[AllocationInfo(true, 64, "line 19 (collect)", true)], 0, 0, :bad_code)

And the way that works is that check_allocation_expr examines lowered expressions for array-allocating functions like collect. Neat, let’s try another:

julia> foo() = [1,2,3];

julia> analyze_escapes(foo,())
EscapeAnalysisReport(AllocationInfo[], 0, 0, :foo)

Uh oh, a miss. The problem is that expression lowers to Base.vect, and that is not among the function names that check_allocation_expr checks. You could add that one, but it’s just a drop among all the functions that can allocate, and arrays are not the only thing that need allocations.

julia> bar() = Ref(1);

julia> analyze_escapes(bar,())
EscapeAnalysisReport(AllocationInfo[], 0, 0, :bar)

Another miss, and I haven’t even tried mutable structs. Meanwhile, the rest of your PR is blocked by this problem even if there were no other issues, and if this problem is fixed, the other completely unrelated problems in your PR (say, the overly strict check on abstract types and dubious verification score cutoff) in turn block it from being merged.

StaticCompiler may not even be the right place for these changes. Despite your intended application, static allocation checks do not necessarily concern AOT compilation without the runtime, and it could be a separate package entirely that is suggested as a dependency or as independent tooling (like StaticTools) for StaticCompiler. In fact, the AllocCheck.jl package does just that. I don’t know its limits, but it at least succeeds at the few examples here.

2 Likes
34

Thank you Benny!