Planning Large Projects

Not that I am aware of.

But note that in Julia elaborate towers of type hierarchies are not as important, useful, or encouraged as in C++ and friends.

So I would say that if you need a tool like this, you are writing heavily non-idiomatic Julia code.

I currently have no type hierarchies that extend beyond two “orders”. IE:

abstract type Fruit end

struct Apple <: Fruit
   color::String
end

If that’s heavily non-idiomatic julia code I don’t really know what to say…

Think this is the second time you’ve implied I was making bad design decisions without seeing my code? I am writing a physics package… There are, unsurprisingly a lot of things in physics that interact with one another and believe it or not multiple paradigms to view the same phenomena.

Please don’t take this personally. I think that warning people about the dangers of some approach is generally useful.

I may of course be wrong (but since you did not share any actual code to start with, it is hard to be more specific). Just experiment with what you find useful and you will find what works best for you.

That said, if there is even a remote chance that you will be showing your code to others (asking for advice, cooperating), please consider following the

https://docs.julialang.org/en/v1/manual/style-guide/

(type names capitalized etc)

1 Like

I wrote that in like 10 seconds to respond… Sorry to bother your eyes I will amend my most grievous error…
In my code I follow conventions set by the Julia community…

As @Tamas_Papp has stated, having some code (even just mock-ups) for us to look at would help us help you immensely. Of course, until we’ve seen some actual code, it wouldn’t be fair for us to judge said code as good or bad; so the least we can do is give very broad, non-specific advice which often will come across as implying that your code is bad (even thought that’s hopefully not the intent) because of the nature of the advice itself (i.e. we can’t complement your excellent craftsmanship if we can’t see what you’re working on).

Anyway, I will play devil’s advocate and say that I’d be interested in some tooling which can render the structure of a package or project in some sort of graph-like or hierarchical manner. Some packages (or even organizations, like DiffEq) have a lot of code and API surface, and it would be really cool to see it all laid out as a graph that connects the dots. It could also make large API refactoring efforts easier by showing at a glance everything downstream that needs changing when something upstream changes.

11 Likes

I think it could be an interesting project to build something with an interface kinda like Cthulhu.jl where instead of descending into @code_typed, it displays a type hierarchy like this [1]:

Number (Core)
	Complex (Base)
	Real (Core)
		AbstractFloat (Core)
			BigFloat (Base.MPFR)
			Float16 (Core)
			Float32 (Core)
			Float64 (Core)
		AbstractIrrational (Base)
			Irrational (Base)
		Integer (Core)
			Bool (Core)
			Signed (Core)
				BigInt (Base.GMP)
				Int128 (Core)
				Int16 (Core)
				Int32 (Core)
				Int64 (Core)
				Int8 (Core)
			Unsigned (Core)
				UInt128 (Core)
				UInt16 (Core)
				UInt32 (Core)
				UInt64 (Core)
				UInt8 (Core)
		Rational (Base)

and then the user can navigate through the hierarchy and see the output of methodswith on each type, or the names exported by each module.

Perhaps that sort of functionality would help the OP?


[1] This type tree is generated by

function showtree(T::Type, level=0)
    println("\t" ^ level, T, " ($(parentmodule(T)))")
    for t in subtypes(T)
        showtypetree(t, level+1)
    end
end

showtree(Number)

which I adapted from https://en.wikibooks.org/wiki/Introducing_Julia/Types.

5 Likes

Mason this is more what I’m talking about!

I would like to see how types are related, what functions accept certain types. Also what files those things are contained in from a project julia file (or multiple)?

I know in python there is a tool that does the following : https://www.logilab.org/blogentry/6883

I guess consider files as nodes in a graph? Or maybe have the option to consider types as nodes? I’m not sure the best way to lay out julia code like this, but something like this would be excellent for me.

It’s really hard to grok thousands of lines of code, even as the author sometimes, and if I weren’t the author I might be mortified. Moreso - I’d love to know if I’m doing anything stupid from a higher level - can I merge types to something more generic, do I have a lot of the “same” functionality that can be abstracted. Etc.

1 Like

@jpsamaroo I’m not asking for a code review or compliments or anything. I’m asking for tooling, I’m sorry if this hasn’t been made clear. I agree though a graph structure could easily be utilized to make code visual once it has reached a certain kind of “critical mass”. That’s what I’m trying to inquire about. Are there tools like this written for Julia? I know there are for other languages.

Sorry for being off-topic: just out of curiosity, since you mentioned it’s a physics package (and physics is free :wink: ), is there any reason why it’s closed source yet?

It’s closed source for one reason,

  1. The useful functionality in it is all in islands and I am trying to unify it.

It will be MIT licensed once it’s worth making public. It’s a pretty large undertaking to even get the essence of what I want done… I speculate it may take 2 months or more. Right now I have a bit of free-time (inbetween jobs) but once I start the new job I won’t have nearly as much free-time.

I really encourage you to make it public unless there are some good reasons to hide it (commercialising, licensing, NDA, younameit). I often see people being shy, hiding their stuff because they think it’s not “good/mature enough” or just have the feeling that they will be eaten alive.
The open source community is tremendously helpful and people really prefer to help other people instead of blaming them for writing bad code or whatever. You can totally assume that most of your stuff is just ignored, but if it’s public, it’s much easier to get help.

That said, I think you could have had some really nice feedback and suggestions within the last hours instead of wild guesses what you are up to and dealing with and some critical comments that you probably do something fundamentally wrong.
Not to mention that hiding such information often leads to things like the XY problem :wink:

7 Likes

By the way, GraphRecipes.jl has some similar features for visualizing trees

3 Likes

No I really mean as it currently stands the package is very jumbled. The real core utility of my package is glueing between domains. Without it, it’s a pile of scripts(what I have now). Once I finish the glue I will happily share, just like I did with my other package(https://github.com/caseykneale/ChemometricsTools.jl). There’s no money to be made here, just trying to draw people into using Julia… Optics is my hobby.

Alls I’m asking about is code inspection tools. It seems there aren’t many and that’s A-okay. I’m starting to see some good ideas on how to build one.

3 Likes

This is a delicious find! Hmmm… I’m wondering if I could spin up a prototype really quickly here. In an ideal world - a tool like this could be interactive, but man I can deal.

3 Likes

If you’re interested in making an interactive way to generate and move through trees of types / functions (though not fancy plot graphics), my humble package ReplMaker.jl might be of assistance. It won’t be straightforward, but it should be easier than wrestling with Julia’s REPL code directly.

3 Likes

Cool! I’ll consider it. I may try for a super hacked down version first…

There is another tool I’ve been kicking around making but haven’t done as well. This may be a bit of a distraction from my main goal but it sounds like fun and maybe people here will take ownership of it and make it way better…

After sitting down for a half hour or so I came to a daunting realization that performing this project would require basically rewriting a way to read the julia call stack… Does anyone who knows the insides of Julia know if there’s a way to highjack how julia compiles code?

Have a look at Cassette.jl.

1 Like

Cool! Alright I have some reading to do…

I doubt you really need to go down to that level, Julia has pretty good introspection tools that should help you getting the information you need, e.g. if you want to list all the types in a module you can do something like this:

using Distributions
mod = Distributions

fields = [getfield(mod, n) for n in names(mod, all=false)]
types = filter(x -> typeof(x) <: DataType, fields)
stypes = supertype.(types)
10 Likes