Problem with the positivedefinitivenesse of a kernel matrix

Specific Domains Statistics
1

Hi guys,

I am trying to generate a positive definite kernel matrix. Below is the minimalistic code:


julia> using MultivariateStats, ForwardDiff, StatsBase, MLKernels, LinearAlgebra, Optim, Plots, Distributions, ForwardDiff, StatsBase, MLKernels

julia> ν = 0.2
0.2

julia> n = 550
550

julia> Nf = 2
2

julia> lowerBound = 2
2

julia> upperBound = 7
7

julia> leftBound = 2
2

julia> rightBound = 7
7

julia> lowerLeft = [leftBound, lowerBound]
2-element Array{Int64,1}:
 2
 2

julia> upperRight = [rightBound, upperBound]
2-element Array{Int64,1}:
 7
 7

julia> polyConst = 1
1

julia> polyDegree = 2
2

julia> abstract type AbstractKernel <: Function end

julia> struct PolynomialKernel{T₁ <: Real, T₂ <: Real} <: AbstractKernel
           c::T₁
           degree::T₂
       end

julia> function (kernel::PolynomialKernel)(xᵢ::AbstractVecOrMat{<: Number}, xⱼ::AbstractVecOrMat{<: Number})
           c = kernel.c
           degree = kernel.degree
           (c + dot(xᵢ, xⱼ))^degree
       end

julia> function generate_frame_data(N, Nf, lowerLeft, upperRight, ν)
           N = 4 * floor(Int,N/4)
           d = Array{Float64,2}(undef, Nf, N)
           x = rand(Uniform(lowerLeft[1], upperRight[1]), floor(Int,N/2))
           y = rand(Uniform(lowerLeft[2], upperRight[2]), floor(Int,N/2))
           ξ = rand(Normal(0, ν), N)
           for n = 1:4:N
               d[:,n] = [x[trunc(Int, ceil(n/2))],  lowerLeft[2] + ξ[n]]
               d[:,n+1] = [x[trunc(Int, ceil(n/2))+1],  upperRight[2] + ξ[n+1]]
               d[:,n+2] = [lowerLeft[1] + ξ[n+2],  y[trunc(Int, ceil(n/2))]]
               d[:,n+3] = [upperRight[1] + ξ[n+3],  y[trunc(Int, ceil(n/2))+1]]
           end
           d
       end
generate_frame_data (generic function with 1 method)

julia> data = generate_frame_data(n, Nf, lowerLeft, upperRight, ν)
2×548 Array{Float64,2}:
 2.31205  3.14641  1.89181  6.91895  2.3989   3.19527  1.90201  6.82866  2.50081  4.44663  …  2.32862  7.038    4.6699   5.29233  2.22841  6.94015  5.80691  5.66697  2.07714  6.8466 
 1.82655  7.05153  2.64666  6.88816  2.18666  7.44543  3.99939  2.91274  1.91899  6.97779     5.48881  6.07231  1.99929  6.67752  5.86748  3.64929  1.57316  7.38312  6.76614  5.66168

julia> fig = plot(data[1,:], data[2,:], seriestype = :scatter, color = :green, legend = false)

julia> tempNormData = fit(ZScoreTransform, data)
ZScoreTransform{Float64}(2, [4.4492, 4.50386], [2.04428, 2.08999])

julia> normalizedData = StatsBase.transform(tempNormData, data)
2×548 Array{Float64,2}:
 -1.04543  -0.637285  -1.251     1.20813  -1.00294  -0.613382  -1.24601    1.16396   -0.953093  …   0.107961  0.412433  -1.08634    1.21849   0.66415  0.595694  -1.16034  1.17273 
 -1.28101   1.21898   -0.888618  1.14082  -1.10871   1.40745   -0.241375  -0.761307  -1.23679      -1.19836   1.04003    0.652453  -0.40889  -1.40226  1.37764    1.08243  0.553983

julia> fig = plot(normalizedData[1,:], normalizedData[2,:], seriestype = :scatter, color = :green, legend = false)

julia> function kernelMatrixConstruct(X::AbstractArray{T, 2};
                                       kernel::PolynomialKernel) where {T<:Real}
               dims = size(X) # the dimensions are: no. of features x number of datapoints
               kernelMatrix = Array{T,2}(undef, dims[2], dims[2]) # dims[2] is the number of datapoints
               for i in 1:dims[2]
                   for j in 1:dims[2]
                       kernelMatrix[i, j] = kernel(X[:, i], X[:, j])
                   end
               end
               kernelMatrix
       end
kernelMatrixConstruct (generic function with 1 method)

julia> function centreKernelMatrix(kernelMatrix::AbstractVecOrMat{T}) where {T<:Real}
               dims = size(kernelMatrix)
               kernelMatrix = (I - ones(dims[1], dims[2])) * kernelMatrix * (I - ones(dims[1], dims[2]))
       end
centreKernelMatrix (generic function with 1 method)

julia> kernelMatrix = kernelMatrixConstruct(normalizedData; kernel = PolynomialKernel(polyConst, polyDegree))
548×548 Array{Float64,2}:
 13.9421     0.0109622   11.876        2.97359    12.0324       0.0261532   6.82157     …  0.582871   1.68971     0.062473     4.41836     1.92524      0.683005    0.87547   
  0.0109622  8.36396      0.509841     2.62672     0.0827482    9.65072     2.24949        4.01979    6.18834     0.0756017    1.28274     5.28859      9.35707     0.861056  
 11.876      0.509841    11.2536       2.32598    10.497        0.26693     7.69088        0.193725   3.16566     0.0259172    2.00285     0.939752     2.21925     0.920383  
  2.97359    2.62672      2.32598     14.1454      2.18011      3.47674     0.609493       7.2079     0.18653     4.02255      0.0410717  10.8327       0.693917    9.2952    
 12.0324     0.0827482   10.497        2.18011    10.4661       0.00299528  6.33675        0.321193   1.86638     0.0534809    3.56678     1.26529      0.928624    0.624719  
  0.0261532  9.65072      0.26693      3.47674     0.00299528  11.2705      2.02936     …  4.88769    6.68035     0.104262     1.90712     6.62329     10.4665      1.12439   
  6.82157    2.24949      7.69088      0.609493    6.33675      2.02936     6.81626        0.0552572  4.82287     0.176029     0.261052    0.00559012   4.77213     0.353968  
  0.575185   0.448624     0.0485757    2.36452     0.4579       0.616941    0.0710453      0.473721   0.579385    7.45055      8.06896     0.415454     1.37981     3.77627   
 12.8216     0.00995384  10.833        2.4411     11.0698       0.0243699   6.18064        0.461559   1.50906     0.118591     4.41544     1.61696      0.588557    0.644619  
  0.265247   5.97173      0.00252855   5.51717     0.0967962    7.11168     0.512443       4.97542    3.14592     0.264675     0.436506    6.91671      5.21087     2.73663   
  4.25603    2.85923      5.2252       0.0555727   4.09009      2.78902     5.26498     …  0.350586   4.6104      0.0782942    0.0767945   0.166615     4.98898     0.044695  
  0.03509    0.0475523    0.336443     6.03209     0.028491     0.0530199   0.537037       1.78562    0.524245    8.14512      5.29925     2.29966      0.877546    6.50328   
 12.8107     0.155979    11.4577       2.59507    11.1789       0.0264606   7.15347        0.359547   2.31207     0.00416521   3.14762     1.38024      1.2911      0.868469  
  1.95156    4.24591      1.11493     11.168       1.3178       5.36424     0.0557114      6.8634     1.02096     1.94745      0.0737499  10.1286       2.24183     6.63282   
  9.57633    0.577663     9.10699      1.35258     8.51617      0.349781    6.35808        0.0537468  2.7959      1.48241e-6   1.82277     0.468882     2.03197     0.462273  
  ⋮                                                             ⋮                       ⋱                                                  ⋮                                  
  7.19816    2.64586      8.32455      0.769231    6.7109       2.42223     7.55804        0.052584   5.53546     0.338487     0.139357    0.00779258   5.60708     0.506335  
  0.635776   0.373064     0.852286     8.74462     0.464906     0.501425    0.473535    …  3.60218    0.02789     5.87114      1.84696     5.07578      0.0121682   7.31326   
  6.83792    0.101767     4.79788      0.225785    5.81204      0.29082     2.08584        0.0639736  0.154787    1.71192      6.3833      0.443875     0.00845457  0.0422178 
  1.30888    4.22842      0.64688      9.46469     0.836924     5.25987     0.00132596     6.13887    1.24569     1.56598      0.0658918   8.92289      2.49518     5.57742   
  2.19259    4.99764      3.53029      0.080893    2.30393      5.288       4.74698        1.1285     5.92614     0.208546     0.122345    1.06361      7.36451     0.00198693
  1.65182    1.22713      1.56521     11.4654      1.21468      1.63703     0.576139       5.30282    0.0129812   5.0006       0.622088    7.77418      0.117583    8.4149    
  5.86732    0.280461     3.72424      0.0560184   4.93         0.566799    1.33341     …  0.0407258  0.0101693   2.62942      7.57413     0.344101     0.178436    0.214125  
  0.582871   4.01979      0.193725     7.2079      0.321193     4.88769     0.0552572      5.07043    1.5142      1.16055      0.0340292   7.1742       2.71326     4.24291   
  1.68971    6.18834      3.16566      0.18653     1.86638      6.68035     4.82287        1.5142     6.79035     0.348668     0.405004    1.5668       8.80166     0.00764925
  0.062473   0.0756017    0.0259172    4.02255     0.0534809    0.104262    0.176029       1.16055    0.348668    7.03268      5.67693     1.35152      0.73353     4.85079   
  4.41836    1.28274      2.00285      0.0410717   3.56678      1.90712     0.261052       0.0340292  0.405004    5.67693     11.6105      0.287482     1.6602      1.00409   
  1.92524    5.28859      0.939752    10.8327      1.26529      6.62329     0.00559012  …  7.1742     1.5668      1.35152      0.287482   10.5803       3.23998     6.06036   
  0.683005   9.35707      2.21925      0.693917    0.928624    10.4665      4.77213        2.71326    8.80166     0.73353      1.6602      3.23998     12.3767      0.0570631 
  0.87547    0.861056     0.920383     9.2952      0.624719     1.12439     0.353968       4.24291    0.00764925  4.85079      1.00409     6.06036      0.0570631   7.19419   

julia> kernelMatrix = centreKernelMatrix(kernelMatrix)
548×548 Array{Float64,2}:
      8.954e5         8.95935e5  895615.0             8.95373e5       8.95681e5       8.95684e5  …       8.96054e5       8.95664e5       8.95673e5       8.95613e5  895991.0      
      8.95935e5       8.96491e5  896152.0             8.95921e5       8.96217e5       8.96242e5          8.96603e5       8.96209e5       8.96225e5       8.9617e5        8.96539e5
 895615.0        896152.0             8.95831e5  895589.0             8.95896e5       8.95901e5          8.96271e5  895878.0             8.95889e5       8.95831e5  896208.0      
      8.95373e5       8.95921e5  895589.0             8.95367e5       8.95654e5       8.95671e5          8.96042e5       8.95643e5       8.95665e5       8.95596e5  895983.0      
      8.95681e5       8.96217e5       8.95896e5       8.95654e5       8.95961e5  895966.0                8.96337e5       8.95945e5       8.95954e5       8.95895e5  896273.0      
      8.95684e5       8.96242e5       8.95901e5       8.95671e5  895966.0             8.95993e5  …  896352.0             8.95959e5       8.95975e5       8.9592e5        8.96289e5
      8.9605e5        8.96594e5       8.96268e5       8.96028e5       8.96332e5       8.96343e5     896712.0             8.96317e5       8.96328e5       8.96274e5  896648.0      
      8.95917e5       8.96465e5       8.96133e5       8.95902e5       8.96199e5       8.96214e5          8.96592e5       8.96197e5       8.96201e5       8.96143e5       8.96524e5
      8.95568e5       8.96103e5       8.95782e5       8.95541e5       8.95848e5       8.95852e5     896223.0        895832.0             8.95841e5       8.95781e5       8.96159e5
      8.96171e5       8.96725e5       8.96388e5  896160.0             8.96453e5       8.96476e5          8.96839e5       8.96444e5       8.96463e5  896402.0             8.96778e5
 896347.0        896894.0             8.96565e5       8.96326e5       8.96629e5       8.96643e5  …  897011.0             8.96616e5       8.96628e5       8.96574e5       8.96947e5
      8.95787e5  896335.0             8.96004e5       8.95776e5       8.96069e5       8.96084e5          8.96463e5       8.96065e5       8.96074e5       8.96013e5  896397.0      
      8.95548e5       8.96084e5       8.95764e5       8.95522e5       8.95829e5  895833.0                8.96204e5       8.95811e5       8.95822e5       8.95763e5       8.9614e5 
      8.95725e5       8.96275e5       8.95941e5       8.95717e5       8.96006e5       8.96025e5          8.96393e5       8.95996e5       8.96017e5       8.95951e5       8.96333e5
      8.95967e5       8.96507e5       8.96184e5       8.95942e5       8.96248e5       8.96256e5          8.96626e5       8.96232e5       8.96243e5       8.96185e5  896562.0      
      ⋮                                                                               ⋮          ⋱                                       ⋮                                        
      8.95892e5       8.96436e5  896110.0        895869.0             8.96174e5       8.96185e5          8.96553e5       8.96158e5       8.96169e5       8.96116e5       8.96489e5
      8.95946e5       8.96494e5       8.96163e5       8.95937e5       8.96228e5  896243.0        …  896619.0             8.9622e5        8.96235e5       8.96171e5       8.96556e5
      8.96202e5  896744.0             8.96417e5       8.96179e5       8.96484e5       8.96493e5          8.96865e5       8.96475e5       8.96481e5       8.96421e5       8.96799e5
      8.95946e5       8.96498e5       8.96163e5  895938.0             8.96228e5       8.96248e5          8.96615e5  896218.0             8.96239e5       8.96173e5       8.96555e5
      8.9625e5        8.96801e5       8.96468e5       8.96231e5       8.96532e5       8.9655e5           8.96916e5       8.96521e5       8.96533e5       8.96481e5       8.96851e5
      8.95714e5       8.96262e5  895931.0             8.95708e5  895996.0             8.96012e5          8.96386e5  895986.0        896005.0             8.95939e5       8.96325e5
      8.96157e5       8.96699e5       8.96371e5       8.96134e5       8.96438e5       8.96449e5  …       8.96822e5       8.96431e5       8.96436e5       8.96377e5       8.96755e5
      8.96235e5       8.96787e5       8.96452e5       8.96225e5       8.96517e5       8.96537e5          8.96904e5       8.96508e5       8.96527e5       8.96463e5       8.96843e5
 896090.0             8.96643e5       8.96308e5  896072.0             8.96372e5       8.96393e5          8.96757e5       8.96362e5       8.96375e5  896323.0             8.96692e5
      8.96054e5       8.96603e5       8.96271e5       8.96042e5       8.96337e5  896352.0                8.9673e5        8.96333e5       8.9634e5        8.96281e5       8.96663e5
      8.95664e5       8.96209e5  895878.0             8.95643e5       8.95945e5       8.95959e5          8.96333e5       8.95944e5       8.95944e5       8.95887e5  896264.0      
      8.95673e5       8.96225e5       8.95889e5       8.95665e5       8.95954e5       8.95975e5  …       8.9634e5        8.95944e5       8.95966e5  895900.0             8.96281e5
      8.95613e5       8.9617e5        8.95831e5       8.95596e5       8.95895e5       8.9592e5           8.96281e5       8.95887e5  895900.0             8.9585e5   896216.0      
 895991.0             8.96539e5  896208.0        895983.0        896273.0             8.96289e5          8.96663e5  896264.0             8.96281e5  896216.0             8.96601e5

julia> eigenValVec = eigen(kernelMatrix)
Eigen{Complex{Float64},Complex{Float64},Array{Complex{Float64},2},Array{Complex{Float64},1}}
eigenvalues:
548-element Array{Complex{Float64},1}:
     4.911747351880314e8 + 0.0im                   
      1137.8202188752973 + 0.0im                   
      1054.0479761456127 + 0.0im                   
       838.9836296970793 + 0.0im                   
       312.5398081227351 + 0.0im                   
      21.633901822518677 + 0.0im                   
    1.664423035365322e-8 + 3.584221841941111e-8im  
    1.664423035365322e-8 - 3.584221841941111e-8im  
  -2.7162862774100785e-8 + 0.0im                   
    2.502946340220498e-8 + 0.0im                   
  -2.3080504883470032e-8 + 0.0im                   
  -1.3999997646844292e-8 + 1.70315313702118e-8im   
  -1.3999997646844292e-8 - 1.70315313702118e-8im   
   4.7988399655563274e-9 + 1.66361824146358e-8im   
   4.7988399655563274e-9 - 1.66361824146358e-8im   
                         ⋮                         
    5.46022460876261e-11 + 3.7263223949169517e-10im
    5.46022460876261e-11 - 3.7263223949169517e-10im
   -4.46532142270148e-10 + 4.704761128069045e-11im 
   -4.46532142270148e-10 - 4.704761128069045e-11im 
  1.1263945579014663e-10 + 0.0im                   
  -4.289819408540298e-10 + 2.041056526897707e-10im 
  -4.289819408540298e-10 - 2.041056526897707e-10im 
  2.6214349981218933e-10 + 6.798206427991077e-11im 
  2.6214349981218933e-10 - 6.798206427991077e-11im 
 -1.8915121536809485e-10 + 2.3206075081781673e-10im
 -1.8915121536809485e-10 - 2.3206075081781673e-10im
 -1.5150420311178568e-10 + 8.831852238755217e-11im 
 -1.5150420311178568e-10 - 8.831852238755217e-11im 
  -7.075526147730452e-11 + 0.0im                   
eigenvectors:
548×548 Array{Complex{Float64},2}:
 0.0426961+0.0im   -0.0720917+0.0im   0.00940534+0.0im    0.0579544+0.0im   -0.0207216+0.0im  …    0.0524645+0.0154957im      0.0524645-0.0154957im      -0.0703808+0.0im
 0.0427222+0.0im    0.0175642+0.0im   -0.0593175+0.0im   -0.0375602+0.0im   -0.0378221+0.0im     0.000343896+0.00127586im   0.000343896-0.00127586im    -0.00479335+0.0im
 0.0427064+0.0im   -0.0651934+0.0im  -0.00826281+0.0im    0.0492872+0.0im     0.033543+0.0im       0.0102564+0.00158704im     0.0102564-0.00158704im     -0.0121957+0.0im
 0.0426953+0.0im     0.066625+0.0im   0.00328951+0.0im    0.0693362+0.0im  -0.00310441+0.0im      0.00502525-0.000974774im   0.00502525+0.000974774im    0.00678188+0.0im
 0.0427095+0.0im   -0.0650449+0.0im   0.00523479+0.0im    0.0479469+0.0im  -0.00746049+0.0im       0.0349251-0.00511892im     0.0349251+0.00511892im     -0.0364507+0.0im
 0.0427103+0.0im    0.0236062+0.0im   -0.0649393+0.0im   -0.0418898+0.0im   -0.0584519+0.0im  …   -0.0216943+0.00290838im    -0.0216943-0.00290838im     0.00877036+0.0im
 0.0427274+0.0im    -0.043739+0.0im   -0.0288086+0.0im    0.0127114+0.0im    0.0643621+0.0im      -0.0066288+0.000305589im   -0.0066288-0.000305589im     0.0308194+0.0im
 0.0427213+0.0im    0.0152994+0.0im     0.058535+0.0im   -0.0455283+0.0im    0.0285042+0.0im     0.000655473-0.00384114im   0.000655473+0.00384114im       0.027396+0.0im
 0.0427041+0.0im    -0.067759+0.0im    0.0105112+0.0im    0.0503989+0.0im   -0.0236829+0.0im      -0.0295279+0.00965549im    -0.0295279-0.00965549im      0.0179927+0.0im
 0.0427334+0.0im    0.0334583+0.0im   -0.0382319+0.0im  0.000454383+0.0im   -0.0560928+0.0im       0.0325476+0.0151013im      0.0325476-0.0151013im      -0.0135687+0.0im
 0.0427416+0.0im   -0.0296302+0.0im   -0.0316837+0.0im  -0.00226301+0.0im    0.0479902+0.0im  …   -0.0278107-0.00399492im    -0.0278107+0.00399492im     0.00704777+0.0im
 0.0427152+0.0im    0.0383635+0.0im    0.0526997+0.0im    -0.016897+0.0im    0.0748643+0.0im      -0.0224047-0.00566027im    -0.0224047+0.00566027im     0.00813065+0.0im
 0.0427032+0.0im   -0.0683428+0.0im   0.00113766+0.0im    0.0540347+0.0im   0.00568535+0.0im        0.012311-0.0126567im       0.012311+0.0126567im      -0.0319987+0.0im
 0.0427121+0.0im    0.0563294+0.0im   -0.0161644+0.0im    0.0472602+0.0im   -0.0473515+0.0im      -0.0116512+0.000494527im   -0.0116512-0.000494527im  -0.000966125+0.0im
 0.0427233+0.0im   -0.0556427+0.0im  -0.00760667+0.0im    0.0354221+0.0im    0.0255586+0.0im      -0.0194237+0.00641922im    -0.0194237-0.00641922im     0.00717196+0.0im
          ⋮                                                                                   ⋱             ⋮                                                            
 0.0427198+0.0im   -0.0457168+0.0im   -0.0334265+0.0im    0.0115639+0.0im    0.0782429+0.0im        0.105972-0.00589204im      0.105972+0.00589204im    -0.00543094+0.0im
 0.0427227+0.0im    0.0490814+0.0im    0.0299769+0.0im    0.0238459+0.0im    0.0538103+0.0im  …   -0.0628663+0.0162232im     -0.0628663-0.0162232im      -0.0326258+0.0im
 0.0427346+0.0im   -0.0390823+0.0im    0.0306605+0.0im   0.00118507+0.0im   -0.0521176+0.0im       0.0338543+0.0105197im      0.0338543-0.0105197im       -0.011491+0.0im
 0.0427227+0.0im    0.0501676+0.0im   -0.0184792+0.0im     0.035861+0.0im   -0.0462094+0.0im       0.0064186+0.0201039im      0.0064186-0.0201039im      -0.0632509+0.0im
 0.0427371+0.0im   -0.0154009+0.0im   -0.0462731+0.0im   -0.0239529+0.0im    0.0401102+0.0im       0.0325889-0.00914366im     0.0325889+0.00914366im      0.0149876+0.0im
 0.0427116+0.0im     0.058597+0.0im    0.0173045+0.0im    0.0485008+0.0im    0.0329596+0.0im       0.0508233-0.00471094im     0.0508233+0.00471094im     -0.0283691+0.0im
 0.0427325+0.0im   -0.0326595+0.0im    0.0389971+0.0im   -0.0126914+0.0im   -0.0569178+0.0im  …     0.046509-0.0202266im       0.046509+0.0202266im      0.00637999+0.0im
 0.0427365+0.0im    0.0410269+0.0im   -0.0202196+0.0im    0.0218283+0.0im    -0.039804+0.0im      -0.0404817+0.0218601im     -0.0404817-0.0218601im       0.0674474+0.0im
 0.0427295+0.0im   -0.0111131+0.0im   -0.0537179+0.0im   -0.0343847+0.0im    0.0371366+0.0im      -0.0570779+0.00649837im    -0.0570779-0.00649837im      0.0526048+0.0im
 0.0427279+0.0im    0.0267622+0.0im    0.0501617+0.0im   -0.0249457+0.0im    0.0485728+0.0im      -0.0377565+0.0195414im     -0.0377565-0.0195414im       0.0869055+0.0im
 0.0427091+0.0im   -0.0201301+0.0im    0.0612899+0.0im   -0.0523741+0.0im   -0.0606421+0.0im        0.046732-0.0170236im       0.046732+0.0170236im      -0.0248631+0.0im
 0.0427097+0.0im    0.0550659+0.0im   -0.0250049+0.0im    0.0410069+0.0im   -0.0666905+0.0im  …   -0.0331885+0.000852603im   -0.0331885-0.000852603im     0.0587821+0.0im
 0.0427069+0.0im  0.000123438+0.0im   -0.0703981+0.0im   -0.0607409+0.0im    0.0163231+0.0im      -0.0700966+0.00680126im    -0.0700966-0.00680126im      0.0249807+0.0im
 0.0427248+0.0im    0.0507146+0.0im    0.0201118+0.0im    0.0334471+0.0im    0.0352572+0.0im      -0.0175624-0.0153147im     -0.0175624+0.0153147im      -0.0247107+0.0im

julia>

As you can see from the result of the values of eigenvalues and eigenvectors it fails to do so.

I suspect it has to do with the overflow.

Your feedback would be very appreciated.

Cheers.

Ergnoor

2

Hi Ergnoor—I have not run your code, but from looking at the matrix there are a few things that stand out. For one, you can avoid the complex-valued eigenvalues by wrapping your matrix in a Symmetric or Hermitian structure. Running eigvals(Symmetric(kernelMatrix)) would return real numbers.

Secondly, and more importantly, your matrix seems incredibly poorly conditioned. I don’t know what’s going on in the dots of the printed output, but every entry we’re seeing looks to be almost exactly the same. If you look at the magnitude of the eigenvalues, your first is is on the order of 10^8 and the second one is on the order of 10^3. So in some sense your matrix is almost a rank one matrix and is probably better thought of as being positive semi-definite from a numerical perspective.

If this is the exact matrix you need to work with, then you’ll probably need to think about some relatively specialized methods for working with such a numerically difficult matrix. Otherwise, it might pay to pick a different kernel or parameterization.

1 Like
3

@cgeoga Hi Chris and thank you very much for your suggestions. I investigated a little bit further the problem of the matrix that you pointed out and in one paper I found this:

If, in the feature space, the origin is far away from the convex hull of the data, then the elements of K (the Gram matrix) have about the same value and, as a result, the matrix is ill-conditioned.

Hence I believe I need to center the data in feature space in order to avoid problems with, as you portrayed them, numerically difficult matrices.

Once more thank you very much for your support.

Cheers.

Ergnoor

1 Like
4

It looks like you center your data, so I don’t think that this is the issue. Your kernelMatrix is also effectively singular and is not positive definite. Are you sure that the kernel is positive definite? I’m not familiar with this particular kernel. It looks similar to the rational quadratic but with a different parameterization?

1 Like
5

@jkbest2 Hi John, and thank you very much for your support. Yes the kernel is positive definite, at least theoretically.

I found the solution using the MLKernels package, it generates much better eigenvalues. It seems the problem is a numerical one.

Thank you again for your support.

Cheers.

Ergnoor

6

Glad you figured it out. I’m interested to see how MLKernels implements this kernel.

7

@jkbest2 Hi John, I think I found out what was wrong with my code. At the centering function I forgot to use 1/n in the formula that does matrix centering. When I did that correction I got the same results as MLKernels package.

function centreKernelMatrix(kernelMatrix::AbstractVecOrMat{T}) where {T<:Real}
        dims = size(kernelMatrix)
        kernelMatrix = (I - 1/dims[2]*ones(dims[1], dims[2])) * kernelMatrix * (I - 1/dims[2]*ones(dims[1], dims[2]))
end

as you can see I use now 1/dims[2] that is to say 1/n.

Again thank you for your support.

Cheers.

Ergnoor

8

Great, glad you figured it out!

9

Just to clarify properties of the polynomial kernel:

The polynomial kernel is actually only positive-semidefinite. For any kernel matrix generated by the polynomial kernel, the highest rank is degree. Once the number of data points exceeds degree, you end up with a rank-deficient and non-definite matrix. Therefore, your problem is not just numerical.

1 Like