# Do any options for fuzzing users' Julia code exist?

**URL:** https://discourse.julialang.org/t/do-any-options-for-fuzzing-users-julia-code-exist/57048
**Category:** Tooling
**Tags:** question, testing
**Created:** [March 12, 2021, 11:19pm UTC](https://discourse.julialang.org/t/do-any-options-for-fuzzing-users-julia-code-exist/57048 "2021-03-12T23:19:59Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![nsajko](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nsajko/32/221187_2.png) [@nsajko](https://discourse.julialang.org/u/nsajko)
#### Post date: [March 12, 2021, 11:19pm UTC](https://discourse.julialang.org/t/do-any-options-for-fuzzing-users-julia-code-exist/57048/1 "2021-03-12T23:19:59Z")

</div>

One of the truly great advantages of writing C or C++ code is the (multiple) possibilities for subjecting that code to _fuzz testing_, which is an outstanding way to find bugs in code:

> **[Fuzzing](https://en.wikipedia.org/wiki/Fuzzing)**
>
> In programming and software development, fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks. Typically, fuzzers are used to test programs that take structured inputs. This structure is specified, e.g., in a file format or protocol and distinguishes valid from invalid in...

[https://llvm.org/docs/LibFuzzer.html](https://llvm.org/docs/LibFuzzer.html)

Some people think that fuzzing is just for catching out-of-bounds accesses and the like, but it’s really much more useful than just that. If one has multiple independent implementations of a single algorithm with well defined input and output, it’s possible to generate random input for them with the fuzzer and then check if the outputs match. Solutions like the one used by Clang also attempt to maximize coverage during the fuzzing process. This and similar features make fuzzing **really useful**.

Even though I haven’t yet written any Julia code that could have benefited from a little fuzzing (I’m a newbie), I’m interested in **whether there are options available** , or if there is **interest in making some progress on that front?**

I suppose that Julia code is actually superbly _suited to fuzzing_, as Julia programs can modify their own syntax trees and (AFAIK) influence compilation in other powerful ways; and also perhaps because Julia already uses LLVM for code generation, so perhaps it could use libfuzzer just like Clang does?

The efforts in the Golang world to make fuzzing a first class citizen (like unit testing already is) are probably also worth taking a look at:

> <https://github.com/golang/go/issues/19109>
>
> Filing a proposal on behalf of @kcc and @dvyukov:
> 
> They request that cmd/go su…pport fuzzing natively, just like it does tests and benchmarks and race detection today.
> 
> https://github.com/dvyukov/go-fuzz exists but it's not as easy as writing tests and benchmarks and running "go test -race" today.
> 
> Should we make this easier?
> 
> \[Motivation\](http://tiny.cc/why-go-fuzz)
> \[Proposal\](https://docs.google.com/document/u/1/d/1zXR-TFL3BfnceEAWytV8bnzB2Tfp6EPFinWVJ5V4QC8/pub)

> <https://github.com/golang/go/issues/44551>
>
> This proposal is to add fuzz test support to Go. This will add a new \`testing.F\`… type, support \`FuzzFoo\` functions in \`\_test.go\` files, and add new \`go\` command behavior.
> 
> A \[design draft\](https://golang.org/s/draft-fuzzing-design) has already been published and iterated on based on feedback from the Go community. This is the next step to propose that this design draft become a language feature.
> 
> This feature will be considered experimental in Go 1.18, and the API will not be covered by the Go 1 compatibility promise yet. The functionality that goes into this release is expected to have bugs and be missing features, but should serve as a proof-of-concept for which Go developers can experiment and provide feedback. Since this will be an experimental feature to start, we also expect there will be room for growth for the mutator and fuzzing engine in future Go releases.
> 
> Below are the parts of the design draft which will not make it into 1.18, and will be considered later on:
> 
> \- support for fuzzing with \`-race\` and \`-msan\`
> \- support for fuzzing with \`-keepfuzzing\`
> \- deduplication of similar crashes caused by different mutations, which would be a prerequisite to implementing \`-keepfuzzing\` (to reduce noise)
> \- allowing special options while fuzzing (e.g. maximum input size)
> \- dictionary support
> \- customizable coverage instrumentation while fuzzing (e.g. to only instrument certain packages or files)
> \- custom generators for the mutator
> \- structured fuzzing support for struct and non-primitive types
> \- \[Stretch goal for 1.18\] structured fuzzing support for primitive types other than \`\[\]byte\` (e.g. string, int, float64)
> 
> 
> Edit: This previously said 1.17, but it was not merged to master to make it into 1.17, so this has been updated to say "1.18" instead.

---

<div class="post-metadata">

### Author: ![kernelmethod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kernelmethod/32/35480_2.png) [@kernelmethod](https://discourse.julialang.org/u/kernelmethod)
#### Post date: [March 13, 2021, 2:15am UTC](https://discourse.julialang.org/t/do-any-options-for-fuzzing-users-julia-code-exist/57048/2 "2021-03-13T02:15:04Z")

</div>

I don’t know of any such tools/libraries for Julia off the top of my head. A Google search would seem to suggest the [ConcolicFuzzing.jl](https://github.com/vchuravy/ConcolicFuzzer.jl) ([JuliaCon talk](https://www.youtube.com/watch?v=gSiQpKHZk7I)) is probably the furthest anyone has gone with fuzzing for Julia.

I definitely think it’s something people would be interested in, though! (I know I would be interested, anyways 🙂) I agree that with Julia’s metaprogramming facilities (plus libraries like Cassette.jl) it would probably be very well-suited to fuzzing.

---

<div class="post-metadata">

### Author: ![miguelraz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/miguelraz/32/631_2.png) [@miguelraz](https://discourse.julialang.org/u/miguelraz)
#### Post date: [March 13, 2021, 2:22am UTC](https://discourse.julialang.org/t/do-any-options-for-fuzzing-users-julia-code-exist/57048/3 "2021-03-13T02:22:19Z")

</div>

@maleadt made an amazing post about this a while ago.  
[https://blog.maleadt.net/2018/11/16/julia\_bugs/](https://blog.maleadt.net/2018/11/16/julia_bugs/)