# MultiResolutionInterators.jl: Tools for working with data that has macro-scale hierachical structure, where you don't always care about some of the levels

**URL:** https://discourse.julialang.org/t/multiresolutioninterators-jl-tools-for-working-with-data-that-has-macro-scale-hierachical-structure-where-you-dont-always-care-about-some-of-the-levels/10315
**Category:** Data
**Created:** [April 13, 2018, 5:50am UTC](https://discourse.julialang.org/t/multiresolutioninterators-jl-tools-for-working-with-data-that-has-macro-scale-hierachical-structure-where-you-dont-always-care-about-some-of-the-levels/10315 "2018-04-13T05:50:55Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)
#### Post date: [April 13, 2018, 5:50am UTC](https://discourse.julialang.org/t/multiresolutioninterators-jl-tools-for-working-with-data-that-has-macro-scale-hierachical-structure-where-you-dont-always-care-about-some-of-the-levels/10315/1 "2018-04-13T05:50:55Z")

</div>

I wanted to share this that I am working on right now.  
(Not yet registered)

[https://github.com/oxinabox/MultiResolutionIterators.jl](https://github.com/oxinabox/MultiResolutionIterators.jl)

I’m making it as a part in my reengineering of CorpusLoaders.jl.  
Basically a lot of Natural Language data has a lot of structure.

For example Wikipedia can be broken into  
**doc, section, paragraph, sentence, word, character**.

Depending on what you are doing you are probably not interesting in considering it at all those levels.  
So you want to basically drop some dimensions.

Here is the headline example, (full context for this is in the readme).

```julia
julia> animal_info = [
           [["Turtles", "are", "reptiles", "."],
            ["They", "have", "shells", "."],
            ["They", "live", "in", "the", "water"]],
           [["Cats", "are", "mammals", "."],
            ["They", "live", "on", "the", "internet"]]
           ]
2-element Array{Array{Array{String,1},1},1}:
 Array{String,1}[String["Turtles", "are", "reptiles", "."], String["They", "have", "shells", "."], String["They", "live", "in", "the", "water"]]
 Array{String,1}[String["Cats", "are", "mammals", "."], String["They", "live", "on", "the", "internet"]]

#...
# some code here
#...

julia> # Merge everything **except** words
       merge_levels(animal_info, (!lvls)(indexer, :words)) |> full_collect
22-element Array{String,1}:
 "Turtles"
 "are"
 "reptiles"
 "."
 "They"
 "have"
 "shells"
 ⋮
 "."
 "They"
 "live"
 "on"
 "the"
 "internet"

```

I’ld love to hear if this is useful in any other domains.  
And any other thoughts.  
(If anyone cares to do a code review and post an issue, that would be really awesome and i’ll owe you one. I’ve good for it 😉 )

---

<div class="post-metadata">

### Author: ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)
#### Post date: [April 13, 2018, 10:22am UTC](https://discourse.julialang.org/t/multiresolutioninterators-jl-tools-for-working-with-data-that-has-macro-scale-hierachical-structure-where-you-dont-always-care-about-some-of-the-levels/10315/2 "2018-04-13T10:22:23Z")

</div>

Hi,

I have created a small library around Flux used for nested multiple-instance learning, which resembles exactly what you have said. The whole purpose of the library is to classify the whole document, reflecting the structure without making the sample flat.

You can find it here

> **[GitHub - CTUAvastLab/Mill.jl: Multiple Instance Learning Library is build on...](https://github.com/CTUAvastLab/Mill.jl)**
>
> Multiple Instance Learning Library is build on top of Flux.jl aimed to prototype flexible multi-instance learning models. - GitHub - CTUAvastLab/Mill.jl: Multiple Instance Learning Library is buil...

but there are no examples at the moment. They might come, if I am not super lazy or if someone is interested in the problem. It implements these two papers:

> **[Using Neural Network Formalism to Solve Multiple-Instance Problems](https://arxiv.org/abs/1609.07257)**
>
> Many objects in the real world are difficult to describe by a single numerical vector of a fixed length, whereas describing them by a set of vectors is more natural. Therefore, Multiple instance learning (MIL) techniques have been constantly gaining...

> **[Discriminative models for multi-instance problems with tree-structure](https://arxiv.org/abs/1703.02868)**
>
> Modeling network traffic is gaining importance in order to counter modern threats of ever increasing sophistication. It is though surprisingly difficult and costly to construct reliable classifiers on top of telemetry data due to the variety and...

Tomas