# Understanding Aqua output

**URL:** <https://discourse.julialang.org/t/understanding-aqua-output/100052>\
**Category:** General Usage\
**Tags:** aqua\
**Created:** [June 8, 2023, 1:55pm UTC](https://discourse.julialang.org/t/understanding-aqua-output/100052 "2023-06-08T13:55:46Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Eben60](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/eben60/32/13475_2.png) [@Eben60](https://discourse.julialang.org/u/Eben60)\
**Post date:** [June 8, 2023, 1:55pm UTC](https://discourse.julialang.org/t/understanding-aqua-output/100052/1 "2023-06-08T13:55:46Z")

</div>

Aqua errors with “The file `.../test/Project.toml` is not in canonical format”

Full output:

```julia
/Volumes/.../Julia/Mendeleev.jl/test/Project.toml: Test Failed at /Users/elk/.julia/packages/Aqua/BhBic/src/project_toml_formatting.jl:7
  Expression: result ⊜ true
   Evaluated: ⟪result: 😭 FAILED: /Volumes/.../Julia/Mendeleev.jl/test/Project.toml
    The file `/Volumes/.../Julia/Mendeleev.jl/test/Project.toml` is not in canonical format.
    
    --- Original Project.toml
    +++ Pkg's output
    @@ -1,10 +1,10 @@
     [deps]
    +Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"
     Coverage = "a2441757-f6aa-5fb2-8edb-039e3f45d037"
     Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
     Unitful = "1986cc42-f94f-5a68-af5c-568840ba703d"
    -Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"
     
     [compat]
    -Unitful = "v1.10"
    -Coverage = "1.6.0"
     Aqua = ">= 0.6"
    +Coverage = "1.6.0"
    +Unitful = "v1.10"
    
⟫ ⊜ true

```

`Project.toml`:

```julia
name = "Mendeleev"
uuid = "c116f080-063d-490a-9873-2b5b2cce4c34"
authors = ["Eben60"]
version = "0.3.1"

[deps]
Unitful = "1986cc42-f94f-5a68-af5c-568840ba703d"
UnitfulAtomic = "a7773ee8-282e-5fa2-be4e-bd808c38a91a"

[compat]
Unitful = "1.10"
UnitfulAtomic = "1"
julia = "1.6"

[extras]
Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test"]

```

`test/Project.toml`:

```julia
[deps]
Coverage = "a2441757-f6aa-5fb2-8edb-039e3f45d037"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
Unitful = "1986cc42-f94f-5a68-af5c-568840ba703d"
Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595"

[compat]
Unitful = "v1.10"
Coverage = "1.6.0"
Aqua = ">= 0.6"

```

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [June 8, 2023, 2:04pm UTC](https://discourse.julialang.org/t/understanding-aqua-output/100052/2 "2023-06-08T14:04:11Z")

</div>

That test is broken. There’s probably already an issue open for it. Three solutions are 1) disable the test for Project.toml format. 2) Make the changes that appear in the error report every time it is reported. 3) Look in Aqua for the code for this test. There is a tool (in Pkg I think) that does this formatting. Run this tool manually or in a script.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [June 8, 2023, 4:04pm UTC](https://discourse.julialang.org/t/understanding-aqua-output/100052/3 "2023-06-08T16:04:40Z")

</div>

Is it broken though? I think it’s just complaining about the lack of alphabetical order, which (in my experience) is ensured by default? Did OP edit the Project.toml manually to get Aqua at the bottom?

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [June 8, 2023, 7:04pm UTC](https://discourse.julialang.org/t/understanding-aqua-output/100052/4 "2023-06-08T19:04:53Z")

</div>

Sometimes you have to edit a Project.toml by hand rather than using `add` and `remove`, for instance to manage a test environment (the newer method is still experimental and is buggier). Furthermore, the manual explicitly allows manually updating `compat` entries. There is no requirement for the entries in a section to be alphabetized.

If the entries are not alphabetized, I get:

```julia
julia> Aqua.analyze_project_toml_formatting(GraphsExt)
1-element Vector{Aqua.LazyTestResult}:
 ⟪result: 😭 FAILED: /home/lapeyre/myrepos/julia_repos/GraphsExt/Project.toml
    Running `Pkg.resolve` on `/home/lapeyre/julia_repos/GraphsExt/Project.toml` will change the content.

```

But running `resolve` does _not_ change the content. Deleting `Manifest.toml` and running `resolve` does not change the content. `Project.toml` is left untouched and the Aqua test will fail again with the same message.

Using the `Pkg.compat` command _will_ alphabetize the file. But Aqua will also complain about the `extras` section, and I don’t know how to edit that except by hand.

Yes, sometimes the package manager alphabetizes `Project.toml`.  
But Aqua is enforcing something that is neither documented to be nor intended to be invariant, and the error message that it gives is incorrect.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [June 8, 2023, 7:13pm UTC](https://discourse.julialang.org/t/understanding-aqua-output/100052/5 "2023-06-08T19:13:52Z")

</div>

That is so weird! What did Aqua complain about in your case?

I agree that editing the extras section must be done by hand if you don’t trust the “new” `test/Project.toml` system (and it’s quite a bummer).

It would definitely be worth opening an issue on Aqua.jl, I don’t think it exists yet. FYI there are plenty of other bugs in the interaction with Pkg.jl:

- [Aqua.jl does not like comments in the `Project.toml` · Issue #100 · JuliaTesting/Aqua.jl · GitHub](https://github.com/JuliaTesting/Aqua.jl/issues/100)
- [Handling extensions in `Project.toml` · Issue #105 · JuliaTesting/Aqua.jl · GitHub](https://github.com/JuliaTesting/Aqua.jl/issues/105)

The issue to clarify the Aqua.jl docs:

- [Clarify errors in documentation · Issue #84 · JuliaTesting/Aqua.jl · GitHub](https://github.com/JuliaTesting/Aqua.jl/issues/84)

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [June 8, 2023, 7:38pm UTC](https://discourse.julialang.org/t/understanding-aqua-output/100052/6 "2023-06-08T19:38:18Z")

</div>

> [@gdalle](#):
>
> That is so weird! What did Aqua complain about in your case?

If you try it, you might not find it weird at all. You can construct a valid Project.toml file, in accordance with the documentation, and it will function well with all of the Julia tools. Except Aqua, which will complain if its not alphabetized. Take an alphabetized file and switch the order of any two entries in `compat` or in `extras`. This mimics what you could do when constructing the `Project.toml` file. You can use `add`, then fill in `compat` by hand. Then do `resolve`, `instantiate` whatever. You are in a completely functional and legit state, but `Aqua` will complain. And as I said, it will claim (it doesn’t follow strictly logically from the error message, but close enough) that `resolve` will fix it, which is not true.

I poked around `Aqua` before posting and saw one or two of your links. I have seen this discussed before more explicitly. I don’t recall on which forum, GH, slack, etc.

EDIT: Also, to expand on what I wrote above, Aqua in fact uses `Pkg.TOML` plus a little hand rolled code to alphabetize the file. Its a handful of lines. You could organize this in your local workflow if you want to enforce this invariant. And if you trust that what Aqua is doing will not get out of sync with what `add`, `remove`, etc. (but not `resolve`!) do.

I don’t know what else you could do… `add Example`, `rm Example` maybe? That’s a bit silly. Maybe there’s something better. If I want this Aqua test enabled I use the Aqua diff output to realphabetize by hand. But, honestly given the state of Pkg, Aqua, and the TOML formatting tools, the best thing is to disable this test. And by “state of” I don’t mean to imply that they are deficient, it’s just that they don’t require or claim to require that the file be alphabetized, they just do it sometimes.  
You could decide to enforce it as an additional style requirement, beyond what the standard Julia tools require. But then you’d want some reasonably automatic tooling to reformat it and document why you are doing it.

---

<div class="post-metadata">

**Author:** ![moble](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moble/32/23535_2.png) [@moble](https://discourse.julialang.org/u/moble)\
**Post date:** [June 9, 2023, 8:07pm UTC](https://discourse.julialang.org/t/understanding-aqua-output/100052/7 "2023-06-09T20:07:47Z")

</div>

IIRC, if you have CompatHelper running in your repo, the `compat` section will get alphabetized for you at some point. So it’s arguably a good idea to start it out that way, to minimize diff noise. 🤷

But I also ran into this issue got a confusing error message when I first started using Aqua.