Cxx.jl with dlib


#1

Hi!

I am using Julia v0.6.4 with Cxx.jl and OpenCV.jl to run some Image Processing codes. After some corrections in openCV_libs.jl file, I was able to run codes which used standard C++ libraries and OpenCV libraries, but for certain scripts I have to use dlib as well. I downloaded the latest version of dlib and compiled it successfully. I am now stuck trying to figure out how to run the C++ codes which use dlib library as well. I would really appreciate any help regarding this.

More Details

What I have tried so far: I copied the dlib folder which contains the *.h files and the built executables into the same folder where opencv and opencv2 folders are already present (include folder).
After this, I wrote this sample small C++ code:

#include <iostream>
#include <dlib/opencv.h>
#include <dlib/image_processing.h>
#include <dlib/image_processing/frontal_face_detector.h>
#include <opencv2/opencv.hpp>

using namespace dlib;
using namespace std;
using namespace cv;

int main()
{
	cout << "Hello World" << endl;
	return 0;
}

I tested it in Julia using Cxx.jl as follows:

ENV["PKG_CONFIG_PATH"] = "the config path" # I defined this part successfully
using OpenCV
using Cxx

cxx"""
#include <iostream>
#include <dlib/opencv.h>
#include <dlib/image_processing.h>
#include <dlib/image_processing/frontal_face_detector.h>
#include <opencv2/opencv.hpp>

using namespace dlib;
using namespace std;
using namespace cv;

int main()
{
	cout << "Hello World" << endl;
	return 0;
}
"""

@cxx main()

This worked successfully and I got the output:

Hello World
0

But, when I try to actually use the dlib functions, I end up getting the following error for example concerning dlib base64 library.

Thanks

Vishwesh


#2

Not that I could help, but what error?


#3

Hi @Tero_Frondelius!
Thanks for the reply. This is the error I am getting:
LLVM ERROR: Program used external function '_ZN4dlib6base64C1Ev' which could not be resolved!


#4

Hi!

An update regarding this matter, I tried a pure dlib example as shown below:

# C++ test Code
cxx"""
// The contents of this file are in the public domain. See LICENSE_FOR_EXAMPLE_PROGRAMS.txt

/*
    This is an example illustrating the use of the matrix object 
    from the dlib C++ Library.
*/


#include <iostream>
#include <dlib/matrix.h>

using namespace dlib;
using namespace std;

// ----------------------------------------------------------------------------------------

int main()
{
    // Let's begin this example by using the library to solve a simple 
    // linear system.
    // 
    // We will find the value of x such that y = M*x where
    //
    //      3.5
    // y =  1.2
    //      7.8
    //
    // and M is
    //
    //      54.2   7.4   12.1
    // M =  1      2     3
    //      5.9    0.05  1


    // First let's declare these 3 matrices.
    // This declares a matrix that contains doubles and has 3 rows and 1 column.
    // Moreover, it's size is a compile time constant since we put it inside the <>.
    matrix<double,3,1> y;
    // Make a 3 by 3 matrix of doubles for the M matrix.  In this case, M is
    // sized at runtime and can therefore be resized later by calling M.set_size(). 
    matrix<double> M(3,3);
    
    // You may be wondering why someone would want to specify the size of a
    // matrix at compile time when you don't have to.  The reason is two fold.
    // First, there is often a substantial performance improvement, especially
    // for small matrices, because it enables a number of optimizations that
    // otherwise would be impossible.  Second, the dlib::matrix object checks
    // these compile time sizes to ensure that the matrices are being used
    // correctly.  For example, if you attempt to compile the expression y*y you
    // will get a compiler error since that is not a legal matrix operation (the
    // matrix dimensions don't make sense as a matrix multiplication).  So if
    // you know the size of a matrix at compile time then it is always a good
    // idea to let the compiler know about it.




    // Now we need to initialize the y and M matrices and we can do so like this:
    M = 54.2,  7.4,  12.1,
        1,     2,    3,
        5.9,   0.05, 1;

    y = 3.5,  
        1.2,    
        7.8;


    // The solution to y = M*x can be obtained by multiplying the inverse of M
    // with y.  As an aside, you should *NEVER* use the auto keyword to capture
    // the output from a matrix expression.  So don't do this: auto x = inv(M)*y; 
    // To understand why, read the matrix_expressions_ex.cpp example program.
    matrix<double> x = inv(M)*y;

    cout << "x: \n" << x << endl;

    // We can check that it really worked by plugging x back into the original equation 
    // and subtracting y to see if we get a column vector with values all very close
    // to zero (Which is what happens.  Also, the values may not be exactly zero because 
    // there may be some numerical error and round off).
    cout << "M*x - y: \n" << M*x - y << endl;


    // Also note that we can create run-time sized column or row vectors like so
    matrix<double,0,1> runtime_sized_column_vector;
    matrix<double,1,0> runtime_sized_row_vector;
    // and then they are sized by saying
    runtime_sized_column_vector.set_size(3);

    // Similarly, the x matrix can be resized by calling set_size(num rows, num columns).  For example
    x.set_size(3,4);  // x now has 3 rows and 4 columns.



    // The elements of a matrix are accessed using the () operator like so:
    cout << M(0,1) << endl;
    // The above expression prints out the value 7.4.  That is, the value of
    // the element at row 0 and column 1.

    // If we have a matrix that is a row or column vector.  That is, it contains either 
    // a single row or a single column then we know that any access is always either 
    // to row 0 or column 0 so we can omit that 0 and use the following syntax.
    cout << y(1) << endl;
    // The above expression prints out the value 1.2


    // Let's compute the sum of elements in the M matrix.
    double M_sum = 0;
    // loop over all the rows
    for (long r = 0; r < M.nr(); ++r)
    {
        // loop over all the columns
        for (long c = 0; c < M.nc(); ++c)
        {
            M_sum += M(r,c);
        }
    }
    cout << "sum of all elements in M is " << M_sum << endl;

    // The above code is just to show you how to loop over the elements of a matrix.  An 
    // easier way to find this sum is to do the following:
    cout << "sum of all elements in M is " << sum(M) << endl;




    // Note that you can always print a matrix to an output stream by saying:
    cout << M << endl;
    // which will print:
    //   54.2  7.4 12.1 
    //      1    2    3 
    //    5.9 0.05    1 

    // However, if you want to print using comma separators instead of spaces you can say:
    cout << csv << M << endl;
    // and you will instead get this as output:
    //   54.2, 7.4, 12.1
    //   1, 2, 3
    //   5.9, 0.05, 1

    // Conversely, you can also read in a matrix that uses either space, tab, or comma
    // separated values by uncommenting the following:
    // cin >> M;



    // -----------------------------  Comparison with MATLAB ------------------------------
    // Here I list a set of Matlab commands and their equivalent expressions using the dlib
    // matrix.  Note that there are a lot more functions defined for the dlib::matrix.  See
    // the HTML documentation for a full listing.

    matrix<double> A, B, C, D, E;
    matrix<int> Aint;
    matrix<long> Blong;

    // MATLAB: A = eye(3)
    A = identity_matrix<double>(3);

    // MATLAB: B = ones(3,4)
    B = ones_matrix<double>(3,4);

    // MATLAB: B = rand(3,4)
    B = randm(3,4);

    // MATLAB: C = 1.4*A
    C = 1.4*A;

    // MATLAB: D = A.*C
    D = pointwise_multiply(A,C);

    // MATLAB: E = A * B
    E = A*B;

    // MATLAB: E = A + B
    E = A + C;

    // MATLAB: E = A + 5
    E = A + 5;

    // MATLAB: E = E'
    E = trans(E);  // Note that if you want a conjugate transpose then you need to say conj(trans(E))

    // MATLAB: E = B' * B
    E = trans(B)*B;

    double var;
    // MATLAB: var = A(1,2)
    var = A(0,1); // dlib::matrix is 0 indexed rather than starting at 1 like Matlab.

    // MATLAB: C = round(C)
    C = round(C);

    // MATLAB: C = floor(C)
    C = floor(C);

    // MATLAB: C = ceil(C)
    C = ceil(C);

    // MATLAB: C = diag(B)
    C = diag(B);

    // MATLAB: B = cast(A, "int32")
    Aint = matrix_cast<int>(A);

    // MATLAB: A = B(1,:)
    A = rowm(B,0);

    // MATLAB: A = B([1:2],:)
    A = rowm(B,range(0,1));

    // MATLAB: A = B(:,1)
    A = colm(B,0);

    // MATLAB: A = [1:5]
    Blong = range(1,5);

    // MATLAB: A = [1:2:5]
    Blong = range(1,2,5);

    // MATLAB: A = B([1:3], [1:2])
    A = subm(B, range(0,2), range(0,1));
    // or equivalently
    A = subm(B, rectangle(0,0,1,2));


    // MATLAB: A = B([1:3], [1:2:4])
    A = subm(B, range(0,2), range(0,2,3));

    // MATLAB: B(:,:) = 5
    B = 5;
    // or equivalently
    set_all_elements(B,5);


    // MATLAB: B([1:2],[1,2]) = 7
    set_subm(B,range(0,1), range(0,1)) = 7;

    // MATLAB: B([1:3],[2:3]) = A
    set_subm(B,range(0,2), range(1,2)) = A;

    // MATLAB: B(:,1) = 4
    set_colm(B,0) = 4;

    // MATLAB: B(:,[1:2]) = 4
    set_colm(B,range(0,1)) = 4;

    // MATLAB: B(:,1) = B(:,2)
    set_colm(B,0) = colm(B,1);

    // MATLAB: B(1,:) = 4
    set_rowm(B,0) = 4;

    // MATLAB: B(1,:) = B(2,:)
    set_rowm(B,0) = rowm(B,1);

    // MATLAB: var = det(E' * E)
    var = det(trans(E)*E);

    // MATLAB: C = pinv(E)
    C = pinv(E);

    // MATLAB: C = inv(E)
    C = inv(E);

    // MATLAB: [A,B,C] = svd(E)
    svd(E,A,B,C);

    // MATLAB: A = chol(E,'lower') 
    A = chol(E);

    // MATLAB: var = min(min(A))
    var = min(A);
}

// ----------------------------------------------------------------------------------------


"""

To add on to this, I defined my PKG_CONFIG_PATH as shown below:

ENV["PKG_CONFIG_PATH"]="/home/hp/OpenCV_installation/installation/OpenCV-3.4.1/lib/pkgconfig:/home/hp/cv2faces/cv4faces/dlib/build/dlib"

When I call my code as, @cxx main(), I get this:

0
x: 
-0.211558 
 -13.9097 
  9.74368 

M*x - y: 
2.84217e-14 
9.99201e-15 
1.77636e-15 

7.4
1.2
sum of all elements in M is 86.65
sum of all elements in M is 86.65
54.2  7.4 12.1 
   1    2    3 
 5.9 0.05    1 

54.2, 7.4, 12.1
1, 2, 3
5.9, 0.05, 1

It’s weird that I am not getting any error this time.

I will appreciate any help regarding this.
Thanks
Vishwesh


#5

So I thought that maybe the error is in the OpenCV libraries, I removed the dlib libraries part, and tested a code with pure OpenCV libraries (highgui to be specific) and it worked perfectly fine!
TLDR:
Using only OpenCV or dlib libraries, the codes are working, but when I am using both together, they are not working, how? :confused:


#6

Can you post one example with an error, it will be much easier to help? So both the code and the error.


#7

Hi @Tero_Frondelius!

Here is the code and the error:

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.4 (2018-07-09 19:09 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-pc-linux-gnu

julia> ENV["PKG_CONFIG_PATH"]="/home/hp/OpenCV_installation/installation/OpenCV-3.4.1/lib/pkgconfig:/home/hp/cv2faces/cv4faces/dlib/build/dlib"
"/home/hp/OpenCV_installation/installation/OpenCV-3.4.1/lib/pkgconfig:/home/hp/cv2faces/cv4faces/dlib/build/dlib"

julia> using OpenCV
WARNING: requiring "OpenCV" in module "Main" did not define a corresponding module.

julia> using Cxx

julia> pwd()
"/home/hp/workfolder/julia"

julia> # C++ test Code
       cxx"""
       // The contents of this file are in the public domain. See LICENSE_FOR_EXAMPLE_PROGRAMS.txt
       /*
           This is an example illustrating the use of the dlib machine learning tools for 
           learning to solve the assignment problem. 
           
           Many tasks in computer vision or natural language processing can be thought of 
           as assignment problems.  For example, in a computer vision application where 
           you are trying to track objects moving around in video, you likely need to solve
           an association problem every time you get a new video frame.  That is, each new 
           frame will contain objects (e.g. people, cars, etc.) and you will want to 
           determine which of these objects are actually things you have seen in previous 
           frames.  
          
           The assignment problem can be optimally solved using the well known Hungarian 
           algorithm.  However, this algorithm requires the user to supply some function 
           which measures the "goodness" of an individual association.  In many cases the 
           best way to measure this goodness isn't obvious and therefore machine learning 
           methods are used.  
           
           The remainder of this example will show you how to learn a goodness function
           which is optimal, in a certain sense, for use with the Hungarian algorithm.  To
           do this, we will make a simple dataset of example associations and use them to
           train a supervised machine learning method. 
           Finally, note that there is a whole example program dedicated to assignment
           learning problems where you are trying to make an object tracker.  So if that is
           what you are interested in then take a look at the learning_to_track_ex.cpp
           example program.
       */


       #include <iostream>
       #include <dlib/svm_threaded.h>

       using namespace std;
       using namespace dlib;


       // ----------------------------------------------------------------------------------------

       /*
           In an association problem, we will talk about the "Left Hand Set" (LHS) and the
           "Right Hand Set" (RHS).  The task will be to learn to map all elements of LHS to 
           unique elements of RHS.  If an element of LHS can't be mapped to a unique element of 
           RHS for some reason (e.g. LHS is bigger than RHS) then it can also be mapped to the 
           special -1 output, indicating no mapping to RHS.
           So the first step is to define the type of elements in each of these sets.  In the
           code below we will use column vectors in both LHS and RHS.  However, in general,
           they can each contain any type you like.  LHS can even contain a different type 
           than RHS.
       */

       typedef dlib::matrix<double,0,1> column_vector;

       // This type represents a pair of LHS and RHS.  That is, sample_type::first
       // contains a left hand set and sample_type::second contains a right hand set.
       typedef std::pair<std::vector<column_vector>, std::vector<column_vector> > sample_type;

       // This type will contain the association information between LHS and RHS.  That is,
       // it will determine which elements of LHS map to which elements of RHS.
       typedef std::vector<long> label_type;

       // In this example, all our LHS and RHS elements will be 3-dimensional vectors.
       const unsigned long num_dims = 3;

       void make_data (
           std::vector<sample_type>& samples,
           std::vector<label_type>& labels
       );
       /*!
           ensures
               - This function creates a training dataset of 5 example associations.  
               - #samples.size() == 5
               - #labels.size() == 5
               - for all valid i:
                   - #samples[i].first == a left hand set
                   - #samples[i].second == a right hand set
                   - #labels[i] == a set of integers indicating how to map LHS to RHS.  To be
                     precise:  
                       - #samples[i].first.size() == #labels[i].size()
                       - for all valid j:
                           -1 <= #labels[i][j] < #samples[i].second.size()
                           (A value of -1 indicates that #samples[i].first[j] isn't associated with anything.
                           All other values indicate the associating element of #samples[i].second)
                       - All elements of #labels[i] which are not equal to -1 are unique.  That is,
                         multiple elements of #samples[i].first can't associate to the same element
                         in #samples[i].second.
       !*/

       // ----------------------------------------------------------------------------------------

       struct feature_extractor
       {
           /*!
               Recall that our task is to learn the "goodness of assignment" function for
               use with the Hungarian algorithm.  The dlib tools assume this function
               can be written as:
                   match_score(l,r) == dot(w, PSI(l,r)) + bias
               where l is an element of LHS, r is an element of RHS, w is a parameter vector,
               bias is a scalar value, and PSI() is a user supplied feature extractor.
               This feature_extractor is where we implement PSI().  How you implement this
               is highly problem dependent.  
           !*/

           // The type of feature vector returned from get_features().  This must be either
           // a dlib::matrix or a sparse vector.
           typedef column_vector feature_vector_type;

           // The types of elements in the LHS and RHS sets
           typedef column_vector lhs_element;
           typedef column_vector rhs_element;


           unsigned long num_features() const
           {
               // Return the dimensionality of feature vectors produced by get_features()
               return num_dims;
           }

           void get_features (
               const lhs_element& left,
               const rhs_element& right,
               feature_vector_type& feats
           ) const
           /*!
               ensures
                   - #feats == PSI(left,right)
                     (i.e. This function computes a feature vector which, in some sense, 
                     captures information useful for deciding if matching left to right 
                     is "good").
           !*/
           {
               // Let's just use the squared difference between each vector as our features.
               // However, it should be emphasized that how to compute the features here is very
               // problem dependent.  
               feats = squared(left - right);
           }

       };

       // We need to define serialize() and deserialize() for our feature extractor if we want 
       // to be able to serialize and deserialize our learned models.  In this case the 
       // implementation is empty since our feature_extractor doesn't have any state.  But you 
       // might define more complex feature extractors which have state that needs to be saved.
       void serialize   (const feature_extractor& , std::ostream& ) {}
       void deserialize (feature_extractor&       , std::istream& ) {}

       // ----------------------------------------------------------------------------------------

       int main()
       {
           try
           {
               // Get a small bit of training data.
               std::vector<sample_type> samples;
               std::vector<label_type> labels;
               make_data(samples, labels);


               structural_assignment_trainer<feature_extractor> trainer;
               // This is the common SVM C parameter.  Larger values encourage the
               // trainer to attempt to fit the data exactly but might overfit. 
               // In general, you determine this parameter by cross-validation.
               trainer.set_c(10);
               // This trainer can use multiple CPU cores to speed up the training.  
               // So set this to the number of available CPU cores. 
               trainer.set_num_threads(4);

               // Do the training and save the results in assigner.
               assignment_function<feature_extractor> assigner = trainer.train(samples, labels);


               // Test the assigner on our data.  The output will indicate that it makes the
               // correct associations on all samples.
               cout << "Test the learned assignment function: " << endl;
               for (unsigned long i = 0; i < samples.size(); ++i)
               {
                   // Predict the assignments for the LHS and RHS in samples[i].   
                   std::vector<long> predicted_assignments = assigner(samples[i]);
                   cout << "true labels:      " << trans(mat(labels[i]));
                   cout << "predicted labels: " << trans(mat(predicted_assignments)) << endl;
               }

               // We can also use this tool to compute the percentage of assignments predicted correctly.
               cout << "training accuracy: " << test_assignment_function(assigner, samples, labels) << endl;


               // Since testing on your training data is a really bad idea, we can also do 5-fold cross validation.
               // Happily, this also indicates that all associations were made correctly.
               randomize_samples(samples, labels);
               cout << "cv accuracy:       " << cross_validate_assignment_trainer(trainer, samples, labels, 5) << endl;



               // Finally, the assigner can be serialized to disk just like most dlib objects.
               serialize("assigner.dat") << assigner;

               // recall from disk
               deserialize("assigner.dat") >> assigner;
           }
           catch (std::exception& e)
           {
               cout << "EXCEPTION THROWN" << endl;
               cout << e.what() << endl;
           }
       }

       // ----------------------------------------------------------------------------------------

       void make_data (
           std::vector<sample_type>& samples,
           std::vector<label_type>& labels
       )
       {
           // Make four different vectors.  We will use them to make example assignments.
           column_vector A(num_dims), B(num_dims), C(num_dims), D(num_dims);
           A = 1,0,0;
           B = 0,1,0;
           C = 0,0,1;
           D = 0,1,1;

           std::vector<column_vector> lhs;
           std::vector<column_vector> rhs;
           label_type mapping;

           // In all the assignments to follow, we will only say an element of the LHS 
           // matches an element of the RHS if the two are equal.  So A matches with A, 
           // B with B, etc.  But never A with C, for example.
           // ------------------------

           lhs.resize(3);
           lhs[0] = A;
           lhs[1] = B;
           lhs[2] = C;

           rhs.resize(3);
           rhs[0] = B;
           rhs[1] = A;
           rhs[2] = C;

           mapping.resize(3);
           mapping[0] = 1;  // lhs[0] matches rhs[1]
           mapping[1] = 0;  // lhs[1] matches rhs[0]
           mapping[2] = 2;  // lhs[2] matches rhs[2]

           samples.push_back(make_pair(lhs,rhs));
           labels.push_back(mapping);

           // ------------------------

           lhs[0] = C;
           lhs[1] = A;
           lhs[2] = B;

           rhs[0] = A;
           rhs[1] = B;
           rhs[2] = D;

           mapping[0] = -1;  // The -1 indicates that lhs[0] doesn't match anything in rhs.
           mapping[1] = 0;   // lhs[1] matches rhs[0]
           mapping[2] = 1;   // lhs[2] matches rhs[1]

           samples.push_back(make_pair(lhs,rhs));
           labels.push_back(mapping);

           // ------------------------

           lhs[0] = A;
           lhs[1] = B;
           lhs[2] = C;

           rhs.resize(4);
           rhs[0] = C;
           rhs[1] = B;
           rhs[2] = A;
           rhs[3] = D;

           mapping[0] = 2;
           mapping[1] = 1;
           mapping[2] = 0;

           samples.push_back(make_pair(lhs,rhs));
           labels.push_back(mapping);

           // ------------------------

           lhs.resize(2);
           lhs[0] = B;
           lhs[1] = C;

           rhs.resize(3);
           rhs[0] = C;
           rhs[1] = A;
           rhs[2] = D;

           mapping.resize(2);
           mapping[0] = -1;
           mapping[1] = 0;

julia> # C++ test Code
       cxx"""
       // The contents of this file are in the public domain. See LICENSE_FOR_EXAMPLE_PROGRAMS.txt
       /*
           This is an example illustrating the use of the dlib machine learning tools for 
           learning to solve the assignment problem. 
           
           Many tasks in computer vision or natural language processing can be thought of 
           as assignment problems.  For example, in a computer vision application where 
           you are trying to track objects moving around in video, you likely need to solve
           an association problem every time you get a new video frame.  That is, each new 
           frame will contain objects (e.g. people, cars, etc.) and you will want to 
           determine which of these objects are actually things you have seen in previous 
           frames.  
          
           The assignment problem can be optimally solved using the well known Hungarian 
           algorithm.  However, this algorithm requires the user to supply some function 
           which measures the "goodness" of an individual association.  In many cases the 
           best way to measure this goodness isn't obvious and therefore machine learning 
           methods are used.  
           
           The remainder of this example will show you how to learn a goodness function
           which is optimal, in a certain sense, for use with the Hungarian algorithm.  To
           do this, we will make a simple dataset of example associations and use them to
           train a supervised machine learning method. 
           Finally, note that there is a whole example program dedicated to assignment
           learning problems where you are trying to make an object tracker.  So if that is
           what you are interested in then take a look at the learning_to_track_ex.cpp
           example program.
       */


       #include <iostream>
       #include <dlib/svm_threaded.h>

       using namespace std;
       using namespace dlib;


       // ----------------------------------------------------------------------------------------

       /*
           In an association problem, we will talk about the "Left Hand Set" (LHS) and the
           "Right Hand Set" (RHS).  The task will be to learn to map all elements of LHS to 
           unique elements of RHS.  If an element of LHS can't be mapped to a unique element of 
           RHS for some reason (e.g. LHS is bigger than RHS) then it can also be mapped to the 
           special -1 output, indicating no mapping to RHS.
           So the first step is to define the type of elements in each of these sets.  In the
           code below we will use column vectors in both LHS and RHS.  However, in general,
           they can each contain any type you like.  LHS can even contain a different type 
           than RHS.
       */

       typedef dlib::matrix<double,0,1> column_vector;

       // This type represents a pair of LHS and RHS.  That is, sample_type::first
       // contains a left hand set and sample_type::second contains a right hand set.
       typedef std::pair<std::vector<column_vector>, std::vector<column_vector> > sample_type;

       // This type will contain the association information between LHS and RHS.  That is,
       // it will determine which elements of LHS map to which elements of RHS.
       typedef std::vector<long> label_type;

       // In this example, all our LHS and RHS elements will be 3-dimensional vectors.
       const unsigned long num_dims = 3;

       void make_data (
           std::vector<sample_type>& samples,
           std::vector<label_type>& labels
       );
       /*!
           ensures
               - This function creates a training dataset of 5 example associations.  
               - #samples.size() == 5
               - #labels.size() == 5
               - for all valid i:
                   - #samples[i].first == a left hand set
                   - #samples[i].second == a right hand set
                   - #labels[i] == a set of integers indicating how to map LHS to RHS.  To be
                     precise:  
                       - #samples[i].first.size() == #labels[i].size()
                       - for all valid j:
                           -1 <= #labels[i][j] < #samples[i].second.size()
                           (A value of -1 indicates that #samples[i].first[j] isn't associated with anything.
                           All other values indicate the associating element of #samples[i].second)
                       - All elements of #labels[i] which are not equal to -1 are unique.  That is,
                         multiple elements of #samples[i].first can't associate to the same element
                         in #samples[i].second.
       !*/

       // ----------------------------------------------------------------------------------------

       struct feature_extractor
       {
           /*!
               Recall that our task is to learn the "goodness of assignment" function for
               use with the Hungarian algorithm.  The dlib tools assume this function
               can be written as:
                   match_score(l,r) == dot(w, PSI(l,r)) + bias
               where l is an element of LHS, r is an element of RHS, w is a parameter vector,
               bias is a scalar value, and PSI() is a user supplied feature extractor.
               This feature_extractor is where we implement PSI().  How you implement this
               is highly problem dependent.  
           !*/

           // The type of feature vector returned from get_features().  This must be either
           // a dlib::matrix or a sparse vector.
           typedef column_vector feature_vector_type;

           // The types of elements in the LHS and RHS sets
           typedef column_vector lhs_element;
           typedef column_vector rhs_element;


           unsigned long num_features() const
           {
               // Return the dimensionality of feature vectors produced by get_features()
               return num_dims;
           }

           void get_features (
               const lhs_element& left,
               const rhs_element& right,
               feature_vector_type& feats
           ) const
           /*!
               ensures
                   - #feats == PSI(left,right)
                     (i.e. This function computes a feature vector which, in some sense, 
                     captures information useful for deciding if matching left to right 
                     is "good").
           !*/
           {
               // Let's just use the squared difference between each vector as our features.
               // However, it should be emphasized that how to compute the features here is very
               // problem dependent.  
               feats = squared(left - right);
           }

       };

       // We need to define serialize() and deserialize() for our feature extractor if we want 
       // to be able to serialize and deserialize our learned models.  In this case the 
       // implementation is empty since our feature_extractor doesn't have any state.  But you 
       // might define more complex feature extractors which have state that needs to be saved.
       void serialize   (const feature_extractor& , std::ostream& ) {}
       void deserialize (feature_extractor&       , std::istream& ) {}

       // ----------------------------------------------------------------------------------------

       int main()
       {
           try
           {
               // Get a small bit of training data.
               std::vector<sample_type> samples;
               std::vector<label_type> labels;
               make_data(samples, labels);


               structural_assignment_trainer<feature_extractor> trainer;
               // This is the common SVM C parameter.  Larger values encourage the
               // trainer to attempt to fit the data exactly but might overfit. 
               // In general, you determine this parameter by cross-validation.
               trainer.set_c(10);
               // This trainer can use multiple CPU cores to speed up the training.  
               // So set this to the number of available CPU cores. 
               trainer.set_num_threads(4);

               // Do the training and save the results in assigner.
               assignment_function<feature_extractor> assigner = trainer.train(samples, labels);


               // Test the assigner on our data.  The output will indicate that it makes the
               // correct associations on all samples.
               cout << "Test the learned assignment function: " << endl;
               for (unsigned long i = 0; i < samples.size(); ++i)
               {
                   // Predict the assignments for the LHS and RHS in samples[i].   
                   std::vector<long> predicted_assignments = assigner(samples[i]);
                   cout << "true labels:      " << trans(mat(labels[i]));
                   cout << "predicted labels: " << trans(mat(predicted_assignments)) << endl;
               }

               // We can also use this tool to compute the percentage of assignments predicted correctly.
               cout << "training accuracy: " << test_assignment_function(assigner, samples, labels) << endl;


               // Since testing on your training data is a really bad idea, we can also do 5-fold cross validation.
               // Happily, this also indicates that all associations were made correctly.
               randomize_samples(samples, labels);
               cout << "cv accuracy:       " << cross_validate_assignment_trainer(trainer, samples, labels, 5) << endl;



               // Finally, the assigner can be serialized to disk just like most dlib objects.
               serialize("assigner.dat") << assigner;

               // recall from disk
               deserialize("assigner.dat") >> assigner;
           }
           catch (std::exception& e)
           {
               cout << "EXCEPTION THROWN" << endl;
               cout << e.what() << endl;
           }
       }

       // ----------------------------------------------------------------------------------------

       void make_data (
           std::vector<sample_type>& samples,
           std::vector<label_type>& labels
       )
       {
           // Make four different vectors.  We will use them to make example assignments.
           column_vector A(num_dims), B(num_dims), C(num_dims), D(num_dims);
           A = 1,0,0;
           B = 0,1,0;
           C = 0,0,1;
           D = 0,1,1;

           std::vector<column_vector> lhs;
           std::vector<column_vector> rhs;
           label_type mapping;

           // In all the assignments to follow, we will only say an element of the LHS 
           // matches an element of the RHS if the two are equal.  So A matches with A, 
           // B with B, etc.  But never A with C, for example.
           // ------------------------

           lhs.resize(3);
           lhs[0] = A;
           lhs[1] = B;
           lhs[2] = C;

           rhs.resize(3);
           rhs[0] = B;
           rhs[1] = A;
           rhs[2] = C;

           mapping.resize(3);
           mapping[0] = 1;  // lhs[0] matches rhs[1]
           mapping[1] = 0;  // lhs[1] matches rhs[0]
           mapping[2] = 2;  // lhs[2] matches rhs[2]

           samples.push_back(make_pair(lhs,rhs));
           labels.push_back(mapping);

           // ------------------------

           lhs[0] = C;
           lhs[1] = A;
           lhs[2] = B;

           rhs[0] = A;
           rhs[1] = B;
           rhs[2] = D;

           mapping[0] = -1;  // The -1 indicates that lhs[0] doesn't match anything in rhs.
           mapping[1] = 0;   // lhs[1] matches rhs[0]
           mapping[2] = 1;   // lhs[2] matches rhs[1]

           samples.push_back(make_pair(lhs,rhs));
           labels.push_back(mapping);

           // ------------------------

           lhs[0] = A;
           lhs[1] = B;
           lhs[2] = C;

           rhs.resize(4);
           rhs[0] = C;
           rhs[1] = B;
           rhs[2] = A;
           rhs[3] = D;

           mapping[0] = 2;
           mapping[1] = 1;
           mapping[2] = 0;

           samples.push_back(make_pair(lhs,rhs));
           labels.push_back(mapping);

           // ------------------------

           lhs.resize(2);
           lhs[0] = B;
           lhs[1] = C;

           rhs.resize(3);
           rhs[0] = C;
           rhs[1] = A;
           rhs[2] = D;

           mapping.resize(2);
           mapping[0] = -1;
           mapping[1] = 0;

           samples.push_back(make_pair(lhs,rhs));
           labels.push_back(mapping);

           // ------------------------

           lhs.resize(3);
           lhs[0] = D;
           lhs[1] = B;
           lhs[2] = C;

           // rhs will be empty.  So none of the items in lhs can match anything.
           rhs.resize(0);

           mapping.resize(3);
           mapping[0] = -1;
           mapping[1] = -1;
           mapping[2] = -1;

           samples.push_back(make_pair(lhs,rhs));
           labels.push_back(mapping);

       }
       """
true

julia> @cxx main()
LLVM ERROR: Program used external function '_ZN4dlib26thread_pool_implementationD1Ev' which could not be resolved!

This is one of the examples for dlib on github repo.

Thanks
Vishwesh


#8

My best guess of the problem is C++ name mangling. Could it be that opencv include dlib and your dlib and opencv is compiled with different compilers?

These links might help you:


#9

Did you link to the dlib using Libdl.dlopen? Where is the addHeaderDir command so you can reference the dlib headers? You should post all of the working code and all of the non-working code with the complete error message, and then it will be easier to diagnose.


#10

Hi @vvjn!
Thank you for the reply. I am sorry but I haven’t carried out any of those commands. This is what I have done so far regarding dlib:
I basically just built the dlib and then did sudo make install and sudo ldconfig so that the .so file and other files can go to /usr/local/ and the correct folders.
Can you help me out please by suggesting how to use the commands that you mentioned?
I have used the commands that I have mentioned in my last comment, but now I realize that they were not enough :frowning_face:
Thanks
Vishwesh


#11

Hi!

Thanks for the reply. I was able to run the same script using g++ directly on command line. I don’t think we have to compile OpenCV libraries and dlib library separately. But yes, this is a sample CMakeLists.txt file that I use for compiling codes like these from command line. So maybe I do have to link the libraries. But, I don’t want to use any separate compilation step to link the libraries. Is it possible to do all this with Julia?


cmake_minimum_required(VERSION 2.8.12)

PROJECT(CV4Faces)


######################## EDIT IF REQUIRED ####################
# ###Uncomment the line below and specify the path to OpenCV directory i.e. the path to the OpenCVConfig.cmake file. Check the examples given below.
SET(OpenCV_DIR /home/hp/OpenCV_installation/installation/OpenCV-3.4.1/share/OpenCV)

################### OpenCV_DIR Examples  #####################

### MACOS : /usr/local/Cellar/opencv/3.3.1_1/share/OpenCV/

### UBUNTU : /usr/local/share/OpenCV/

### WINDOWS : C:\Users\yourname\Documents\opencv-3.3.1\build\install

##############################################################




###################   ***DO NOT EDIT***  #####################

#############  Common Instructions for all Users  ############
find_package( OpenCV REQUIRED )

include_directories( ${OpenCV_INCLUDE_DIRS})
add_subdirectory(/home/hp/cv2faces/cv4faces/dlib dlib_build)

MACRO(add_example name)
  ADD_EXECUTABLE(${name} ${name}.cpp)
  TARGET_LINK_LIBRARIES(${name} ${OpenCV_LIBS} dlib::dlib)
ENDMACRO()

add_example(OpenCV_trackMultipleObjects)

Thanks
Vishwesh


#12

Try following Keno’s example from the README:

Example 8: Using C++ with shared libraries
ArrayMaker.h

#ifndef ARRAYMAKER_H
#define ARRAYMAKER_H

class ArrayMaker
{
    private:
        int iNumber;
        float fNumber;
        float* fArr;
    public:
        ArrayMaker(int, float);
        float* fillArr();
};

#endif

ArrayMaker.cpp

#include "ArrayMaker.h"
#include <iostream>

using namespace std;

ArrayMaker::ArrayMaker(int iNum, float fNum) {
    cout << "Got arguments: " << iNum << ", and " << fNum << endl;
    iNumber = iNum;
    fNumber = fNum;
    fArr = new float[iNumber];
}

float* ArrayMaker::fillArr() {
    cout << "Filling the array" << endl;
    for (int i=0; i < iNumber; i++) {
        fArr[i] = fNumber;
        fNumber *= 2;
    } 
    return fArr;
}

Compiling into shared library

>> g++ -shared -fPIC ArrayMaker.cpp -o libArrayMaker.so

Using in Julia

julia> using Cxx

# Importing shared library and header file
julia> const path_to_lib = pwd()
julia> addHeaderDir(path_to_lib, kind=C_System)
julia> Libdl.dlopen(path_to_lib * "/libArrayMaker.so", Libdl.RTLD_GLOBAL)
Ptr{Void} @0x000000000728e6d0
julia> cxxinclude("ArrayMaker.h")

# Creating class object
julia> maker = @cxxnew ArrayMaker(5, 2.0)
Got arguments: 5, and 2
Cxx.CppPtr{Cxx.CppValue{Cxx.CppBaseType{:ArrayMaker},(false,false,false)},(false,false,false)}(Ptr{Void} @0x00000000060ab570)

julia> arr = @cxx maker->fillArr()
Filling the array

julia> pointer_to_array(arr, 5)
5-element Array{Float32,1}:
  2.0
  4.0
  8.0
 16.0
 32.0

My C++ skills are very limited and I cannot help you more than this.


#13

Thanks @Tero_Frondelius for the reply. Actually I have gone through these examples but haven’t tried them yet. I guess I should be following the ArrayMaker.h example. The issue is this:

I have a lot of C++ codes which use OpenCV and dlib modules. I was able to run all (or most) of the C++ codes which used OpenCV modules using Cxx.jl and OpenCV.jl. It didn’t require any of the steps mentioned in the ArrayMaker.h example.

I want to keep the process as clean and simple to understand for a user. And that’s why I want to avoid the compiling into shared library step.

Is it possible to do these steps:

  1. If I am right, the addHeaderDir will contain all the *.h files for dlib?
  2. Libdl.dlopen if I supply the *.so files made from dlib, will it work? Will I have to run this part separately for all the so files?
  3. cxxinclude: Can this be automated? If this can be done, I can merge this and the above two steps in a single julia script and call it every time when I am running a script.

I know these look like shots in the dark, but it would be amazing for me if this is possible.

Thanks

Vishwesh


#14

I encourage you to go through all eight examples. If you are lucky, you will reproduce your error message.


#15

Thanks @Tero_Frondelius! I will give it a shot. Hopefully it will work. Please let me know in case if you find something that can help me.
Thanks