How to use OpenCV.jl's calibrateCamera correctly?

General Usage
1

Hi,
I’m trying to reproduce :OpenCV: Camera Calibration using OpenCV.jl.
I made some adjustments but could not figure out how to properly pass the arguments to cv.calibrateCamera. 1st and 2nd arguments I passed were Vector{Array{Float32, 3}}s but those were not detected as Vector{AbstractArray{Float32, 3}}. Also, I need the 4th and 5th arguments that we don’t need in Python version.
Could anyone suggest changes to make it work?

using OpenCV
const cv = OpenCV

## file
filename = "calib_radial.jpg"
img = cv.imread(filename, -1)
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)

# termination criteria
criteria = cv.TermCriteria(cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)

# Arrays to store object points and image points from all the images.
objpoints = Vector{Array{Float32, 3}}(undef, 0) # 3d point in real world space
imgpoints = Vector{Array{Float32, 3}}(undef, 0) # 2d points in image plane.

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = zeros(Float32, 2, 1, 7 * 6)
n = 1
for i in 1:7, j in 1:6
	objp[1, 1, n] = i - 1
	objp[2, 1, n] = j - 1
	n += 1
end

# Find the chess board corners
isValid, corners = cv.findChessboardCorners(gray, cv.Size{Int32}(7, 6))

# If found, add object points, image points (after refining them)
if isValid
	push!(objpoints, objp)
end

corners2 = cv.cornerSubPix(gray, corners, cv.Size{Int32}(11, 11), cv.Size{Int32}(-1, -1), criteria)
push!(imgpoints, corners2)

## camera cal
ret, mtx, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, cv.Size{Int32}(size(gray)[[3, 2]]...))

The error I got.

ERROR: MethodError: no method matching calibrateCamera(::Vector{Array{Float32, 3}}, ::Vector{Array{Float32, 3}}, ::OpenCV.Size{Int32})

Closest candidates are:
  calibrateCamera(::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::OpenCV.Size{Int32}, ::Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}, ::Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}; rvecs, tvecs, flags, criteria)
   @ OpenCV C:\Users\---\.julia\artifacts\fb232e63f59cec37f3a904b5decdaa33513a13b1\OpenCV\src\cv_cxx_wrap.jl:1786
  calibrateCamera(::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::OpenCV.Size{Int32}, ::Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}, ::Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}, ::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::Int64, ::OpenCV.TermCriteria)
   @ OpenCV C:\Users\---\.julia\artifacts\fb232e63f59cec37f3a904b5decdaa33513a13b1\OpenCV\src\cv_cxx_wrap.jl:1783
1 Like
2

I ran into the exact same issues (described OpenCV function signatures).

Did you get anywhere regarding this?

3

Those are some very strange types indeed. Note for example that

julia> Vector{Array{Float32, 3}} <: Vector{AbstractArray{Float32, 3}}
false

julia> Vector{Array{Float32, 3}} <: Vector{Array{T, 3} where T <: Union{Float32, Float64}}
false

julia> Vector{Array{Float32, 3}} <: Vector{Array{T, 3}} where T <: Union{Float32, Float64}
true

You could, if you really want, convert from Vector{Array{Float32, 3}} to the type in the signature via

objpoints = Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T <: Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}(objpoints)
imgpoints = Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T <: Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}(imgpoints)

but I have no idea why the fourth argument (CameraMatrix, cf. OpenCV docs, a 3 \times 3 matrix) would require three axes (Union{OpenCV.CxxMat, AbstractArray{T, 3} where T <: (...)}}) (nor why

julia> OpenCV.Mat <: AbstractArray{T, 2} where T
false

julia> OpenCV.Mat <: AbstractArray{T, 3} where T
true

for that matter.)

I think you’ll have better luck PythonCalling the OpenCV Python bindings.

4

Concretely,

using Pkg
Pkg.add(["PythonCall", "CondaPkg"])
using CondaPkg
CondaPkg.add(["opencv", "numpy"])

using PythonCall
cv = pyimport("cv2")
np = pyimport("numpy")

## file
filename = "calib_radial.jpg"
img = cv.imread(filename, -1)  # (This is a Python object)
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)

# termination criteria
criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)  # (This is a Julia Tuple, where the first entry is a Python int)

# Python Lists to store object points and image points from all the images.
objpoints = PyList() # 3d points in real world space
imgpoints = PyList() # 2d points in image plane.

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = zeros(Float32, 7 * 6, 3)  # Note: changed compared to original code
let n = 1
	for i in 1:7, j in 1:6
		objp[n, 1] = i - 1
		objp[n, 2] = j - 1
		n += 1
	end
end

# Find the chess board corners
isValid, corners = cv.findChessboardCorners(gray, (7, 6))

# If found, add object points, image points (after refining them)
if pytruth(isValid)  # (or pyconvert(Bool, isValid))
	push!(objpoints, np.asarray(objp))  # objp is Julia, but we need Python for cv, so convert to NumPy
end

corners2 = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
push!(imgpoints, corners2)

## camera cal
ret, mtx, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, reverse(Tuple(gray.shape)), pybuiltins.None, pybuiltins.None)
# Python:
# (82.84610812959355, 
#  array([[1.31550595e+03, 0.00000000e+00, 1.89256059e+02],
#        [0.00000000e+00, 2.72905446e+02, 2.15993621e+02],
#        [0.00000000e+00, 0.00000000e+00, 1.00000000e+00]]),  
#  array([[-9.12214600e+00, -9.69316233e+02, -2.64952158e+00, -1.37557924e+00,  3.36561597e+04]]), 
#  (array([[-1.21794817],
#          [ 0.59696311],
#          [-0.58325645]]),), 
#  (array([[-2.31090673],
#        [ 1.20981955],
#        [22.52187791]]),))

pyconvert(Array, mtx)
# 3×3 Matrix{Float64}:
#  1315.51    0.0    189.256
#     0.0   272.905  215.994
#     0.0     0.0      1.0
5

I’ve now managed to figure out most of my previous issues, and managed to make use of the calibration functions from OpenCV.jl, see here for details in my CameraCalibrations.jl package.

Basically, you need to use OpenCV.InputArray on vectors of 3D arrays, and OpenCV.Mat on 3D arrays… A bit confusing, but that solved it for me.

See here for more details: opencv_contrib/modules/julia at 4.x · opencv/opencv_contrib · GitHub

6

Thank you for sharing your solution. I’m sorry for asking but could you give us some code examples? That will be very helpful. I’m still straggling with this problem after using OpenCV.InputArray and OpenCV.Mat types.

I run the following code and got an error.

const cv = OpenCV

gray # checkerbord pattern image: 1x398x454 OpenCV.Mat{UInt8}
T = Float32 # floating point precision
pattern = (7, 6) # pattern size
imgSize = cv.Size{Int32}(size(gray)[[3, 2]]...)
criteria = cv.TermCriteria(cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001) # termination criteria

# Arrays to store object points and image points from all the images.
objpoints = cv.InputArray[] # 3d point in real world space
imgpoints = cv.InputArray[] # 2d points in image plane.

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ...., (6,5,0)
objp = zeros(T, 2, 1, prod(pattern))
n = 1
for i in 1:7, j in 1:6
	objp[1, 1, n] = i - 1
	objp[2, 1, n] = j - 1
	n += 1
end

# Find the chess board corners
isValid, corners = cv.findChessboardCorners(gray, cv.Size{Int32}(pattern...))

# If found, add object points, image points (after refining them)
if isValid
	push!(objpoints, objp)
end

corners2 = cv.cornerSubPix(gray, corners, cv.Size{Int32}(11, 11), cv.Size{Int32}(-1, -1), criteria)
push!(imgpoints, corners2)

# camera cal
fake_mtx = cv.Mat(Array{T}(undef, 1, 3, 3))
fake_dist = cv.Mat(Array{T}(undef, 1, 1, 5))
fake_r = cv.InputArray[Array{T}(undef, 3, 1, prod(pattern)) for _ in 1:1]
fake_t = cv.InputArray[Array{T}(undef, 3, 1, prod(pattern)) for _ in 1:1]
flags = cv.CALIB_ZERO_TANGENT_DIST + cv.CALIB_FIX_K3 + cv.CALIB_FIX_K2 + cv.CALIB_FIX_ASPECT_RATIO

ret, mtx, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, imgSize, fake_mtx, fake_dist, fake_r, fake_t, flags, criteria)

This returns error as follows.

ERROR: OpenCV(4.10.0) /workspace/srcdir/opencv/modules/calib3d/src/calibration.cpp:3515: error: (-210:Unsupported format or combination of formats) objectPoints should contain vector of vectors of points of type Point3f in function 'collectCalibrationData'

Stacktrace:
 [1] jlopencv_cv_cv_calibrateCamera(arg1::CxxWrap.StdLib.StdVectorAllocated{…}, arg2::CxxWrap.StdLib.StdVectorAllocated{…}, arg3::OpenCV.Size{…}, arg4::OpenCV.CxxMatAllocated, arg5::OpenCV.CxxMatAllocated, arg6::CxxWrap.StdLib.StdVectorAllocated{…}, arg7::CxxWrap.StdLib.StdVectorAllocated{…}, arg8::Int64, arg9::OpenCV.TermCriteria)
   @ OpenCV C:\Users\\.julia\packages\CxxWrap\eWADG\src\CxxWrap.jl:668
 [2] calibrateCamera(objectPoints::Vector{…}, imagePoints::Vector{…}, imageSize::OpenCV.Size{…}, cameraMatrix::OpenCV.Mat{…}, distCoeffs::OpenCV.Mat{…}, rvecs::Vector{…}, tvecs::Vector{…}, flags::Int64, criteria::OpenCV.TermCriteria)
   @ OpenCV C:\Users\\.julia\artifacts\ca7a8cea18e4e4448959e8e169004bad36338b4f\OpenCV\src\cv_cxx_wrap.jl:2276
 [3] top-level scope
   @ c:\Users\\OneDrive\julia_dev\julia_discourse\OpenCV_calibrateCamera\fixed.jl:108
Some type information was truncated. Use `show(err)` to see complete types.
1 Like
7

Sorry. It was my stupid mistake. The objp in my code has to have a shape of (3,1,42). Also, I changed flags to 0 for now. This code should work now. Thanks for @yakir12.

gray # checkerbord pattern image: 1x398x454 OpenCV.Mat{UInt8}
T = Float32 # floating point precision
pattern = (7, 6) # pattern size
imgSize = cv.Size{Int32}(size(gray)[[3, 2]]...)
criteria = cv.TermCriteria(cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001) # termination criteria

# Arrays to store object points and image points from all the images.
objpoints = cv.InputArray[] # 3d point in real world space
imgpoints = cv.InputArray[] # 2d points in image plane.

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ...., (6,5,0)
objp = [T[i-1, j-1, 0] for i in 1:pattern[1], j in 1:pattern[2]]
objp = reshape(stack(objp[:]), 3, 1, prod(pattern))

# Find the chess board corners
isValid, corners = cv.findChessboardCorners(gray, cv.Size{Int32}(pattern...))

# If found, add object points, image points (after refining them)
if isValid
	corners2 = cv.cornerSubPix(gray, corners, cv.Size{Int32}(11, 11), cv.Size{Int32}(-1, -1), criteria)
	push!(objpoints, objp)
	push!(imgpoints, corners2)
end

# camera cal
fake_mtx = cv.Mat(Array{T}(undef, 1, 3, 3))
fake_dist = cv.Mat(Array{T}(undef, 1, 1, 5))
fake_r = cv.InputArray[Array{T}(undef, 3, 1, prod(pattern)) for _ in 1:1]
fake_t = cv.InputArray[Array{T}(undef, 3, 1, prod(pattern)) for _ in 1:1]
flags = 0

ret, mtx, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, imgSize, fake_mtx, fake_dist, fake_r, fake_t, flags, criteria)
1 Like