# How to use OpenCV.jl's calibrateCamera correctly?

**URL:** <https://discourse.julialang.org/t/how-to-use-opencv-jls-calibratecamera-correctly/113765>\
**Category:** General Usage\
**Tags:** question\
**Created:** [May 2, 2024, 10:16pm UTC](https://discourse.julialang.org/t/how-to-use-opencv-jls-calibratecamera-correctly/113765 "2024-05-02T22:16:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kadii](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@Kadii](https://discourse.julialang.org/u/Kadii)\
**Post date:** [May 2, 2024, 10:16pm UTC](https://discourse.julialang.org/t/how-to-use-opencv-jls-calibratecamera-correctly/113765/1 "2024-05-02T22:16:26Z")

</div>

Hi,  
I’m trying to reproduce :[OpenCV: Camera Calibration](https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html) using OpenCV.jl.  
I made some adjustments but could not figure out how to properly pass the arguments to `cv.calibrateCamera`. 1st and 2nd arguments I passed were `Vector{Array{Float32, 3}}`s but those were not detected as `Vector{AbstractArray{Float32, 3}}`. Also, I need the 4th and 5th arguments that we don’t need in Python version.  
Could anyone suggest changes to make it work?

```julia
using OpenCV
const cv = OpenCV

## file
filename = "calib_radial.jpg"
img = cv.imread(filename, -1)
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)

# termination criteria
criteria = cv.TermCriteria(cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001)

# Arrays to store object points and image points from all the images.
objpoints = Vector{Array{Float32, 3}}(undef, 0) # 3d point in real world space
imgpoints = Vector{Array{Float32, 3}}(undef, 0) # 2d points in image plane.

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = zeros(Float32, 2, 1, 7 * 6)
n = 1
for i in 1:7, j in 1:6
	objp[1, 1, n] = i - 1
	objp[2, 1, n] = j - 1
	n += 1
end

# Find the chess board corners
isValid, corners = cv.findChessboardCorners(gray, cv.Size{Int32}(7, 6))

# If found, add object points, image points (after refining them)
if isValid
	push!(objpoints, objp)
end

corners2 = cv.cornerSubPix(gray, corners, cv.Size{Int32}(11, 11), cv.Size{Int32}(-1, -1), criteria)
push!(imgpoints, corners2)

## camera cal
ret, mtx, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, cv.Size{Int32}(size(gray)[[3, 2]]...))

```

The error I got.

```julia
ERROR: MethodError: no method matching calibrateCamera(::Vector{Array{Float32, 3}}, ::Vector{Array{Float32, 3}}, ::OpenCV.Size{Int32})

Closest candidates are:
  calibrateCamera(::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::OpenCV.Size{Int32}, ::Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}, ::Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}; rvecs, tvecs, flags, criteria)
   @ OpenCV C:\Users\---\.julia\artifacts\fb232e63f59cec37f3a904b5decdaa33513a13b1\OpenCV\src\cv_cxx_wrap.jl:1786
  calibrateCamera(::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::OpenCV.Size{Int32}, ::Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}, ::Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}, ::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T<:Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}, ::Int64, ::OpenCV.TermCriteria)
   @ OpenCV C:\Users\---\.julia\artifacts\fb232e63f59cec37f3a904b5decdaa33513a13b1\OpenCV\src\cv_cxx_wrap.jl:1783

```

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [November 28, 2024, 10:45am UTC](https://discourse.julialang.org/t/how-to-use-opencv-jls-calibratecamera-correctly/113765/2 "2024-11-28T10:45:29Z")

</div>

I ran into the exact same issues (described [OpenCV function signatures](https://discourse.julialang.org/t/opencv-function-signatures/123145)).

Did you get anywhere regarding this?

---

<div class="post-metadata">

**Author:** ![eldee](https://avatars.discourse-cdn.com/v4/letter/e/b5a626/32.png) [@eldee](https://discourse.julialang.org/u/eldee)\
**Post date:** [November 29, 2024, 8:22pm UTC](https://discourse.julialang.org/t/how-to-use-opencv-jls-calibratecamera-correctly/113765/3 "2024-11-29T20:22:23Z")

</div>

Those are some very strange types indeed. Note for example that

```julia-repl
julia> Vector{Array{Float32, 3}} <: Vector{AbstractArray{Float32, 3}}
false

julia> Vector{Array{Float32, 3}} <: Vector{Array{T, 3} where T <: Union{Float32, Float64}}
false

julia> Vector{Array{Float32, 3}} <: Vector{Array{T, 3}} where T <: Union{Float32, Float64}
true

```

You could, if you really want, convert from `Vector{Array{Float32, 3}}` to the type in the signature via

```julia
objpoints = Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T <: Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}(objpoints)
imgpoints = Vector{Union{OpenCV.CxxMat, AbstractArray{T, 3} where T <: Union{Float32, Float64, Int16, Int32, Int8, UInt16, UInt8}}}(imgpoints)

```

but I have no idea why the fourth argument (`CameraMatrix`, cf. [OpenCV docs](https://docs.opencv.org/4.x/d9/d0c/group__calib3d.html#ga3207604e4b1a1758aa66acb6ed5aa65d), a 3 \times 3 matrix) would require three axes (`Union{OpenCV.CxxMat, AbstractArray{T, 3} where T <: (...)}}`) (nor why

```julia-repl
julia> OpenCV.Mat <: AbstractArray{T, 2} where T
false

julia> OpenCV.Mat <: AbstractArray{T, 3} where T
true

```

for that matter.)

I think you’ll have better luck `PythonCall`ing the OpenCV Python bindings.

---

<div class="post-metadata">

**Author:** ![eldee](https://avatars.discourse-cdn.com/v4/letter/e/b5a626/32.png) [@eldee](https://discourse.julialang.org/u/eldee)\
**Post date:** [November 30, 2024, 3:56pm UTC](https://discourse.julialang.org/t/how-to-use-opencv-jls-calibratecamera-correctly/113765/4 "2024-11-30T15:56:11Z")

</div>

> [@eldee](#):
>
> I think you’ll have better luck `PythonCall`ing the OpenCV Python bindings.

Concretely,

```julia
using Pkg
Pkg.add(["PythonCall", "CondaPkg"])
using CondaPkg
CondaPkg.add(["opencv", "numpy"])

using PythonCall
cv = pyimport("cv2")
np = pyimport("numpy")

## file
filename = "calib_radial.jpg"
img = cv.imread(filename, -1) # (This is a Python object)
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)

# termination criteria
criteria = (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001) # (This is a Julia Tuple, where the first entry is a Python int)

# Python Lists to store object points and image points from all the images.
objpoints = PyList() # 3d points in real world space
imgpoints = PyList() # 2d points in image plane.

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = zeros(Float32, 7 * 6, 3) # Note: changed compared to original code
let n = 1
	for i in 1:7, j in 1:6
		objp[n, 1] = i - 1
		objp[n, 2] = j - 1
		n += 1
	end
end

# Find the chess board corners
isValid, corners = cv.findChessboardCorners(gray, (7, 6))

# If found, add object points, image points (after refining them)
if pytruth(isValid) # (or pyconvert(Bool, isValid))
	push!(objpoints, np.asarray(objp)) # objp is Julia, but we need Python for cv, so convert to NumPy
end

corners2 = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
push!(imgpoints, corners2)

## camera cal
ret, mtx, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, reverse(Tuple(gray.shape)), pybuiltins.None, pybuiltins.None)
# Python:
# (82.84610812959355, 
# array([[1.31550595e+03, 0.00000000e+00, 1.89256059e+02],
# [0.00000000e+00, 2.72905446e+02, 2.15993621e+02],
# [0.00000000e+00, 0.00000000e+00, 1.00000000e+00]]),  
# array([[-9.12214600e+00, -9.69316233e+02, -2.64952158e+00, -1.37557924e+00, 3.36561597e+04]]), 
# (array([[-1.21794817],
# [0.59696311],
# [-0.58325645]]),), 
# (array([[-2.31090673],
# [1.20981955],
# [22.52187791]]),))

pyconvert(Array, mtx)
# 3×3 Matrix{Float64}:
# 1315.51 0.0 189.256
# 0.0 272.905 215.994
# 0.0 0.0 1.0

```

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [December 3, 2024, 12:05pm UTC](https://discourse.julialang.org/t/how-to-use-opencv-jls-calibratecamera-correctly/113765/5 "2024-12-03T12:05:32Z")

</div>

I’ve now managed to figure out most of my previous issues, and managed to make use of the calibration functions from OpenCV.jl, see [here](https://github.com/yakir12/CameraCalibrations/blob/master/src/detect_fit.jl) for details in my CameraCalibrations.jl package.

Basically, you need to use `OpenCV.InputArray` on vectors of 3D arrays, and `OpenCV.Mat` on 3D arrays… A bit confusing, but that solved it for me.

See here for more details: [opencv\_contrib/modules/julia at 4.x · opencv/opencv\_contrib · GitHub](https://github.com/opencv/opencv_contrib/tree/4.x/modules/julia#usage)

---

<div class="post-metadata">

**Author:** ![Kadii](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@Kadii](https://discourse.julialang.org/u/Kadii)\
**Post date:** [December 4, 2024, 12:23am UTC](https://discourse.julialang.org/t/how-to-use-opencv-jls-calibratecamera-correctly/113765/6 "2024-12-04T00:23:50Z")

</div>

Thank you for sharing your solution. I’m sorry for asking but could you give us some code examples? That will be very helpful. I’m still straggling with this problem after using `OpenCV.InputArray` and `OpenCV.Mat` types.

I run the following code and got an error.

```julia
const cv = OpenCV

gray # checkerbord pattern image: 1x398x454 OpenCV.Mat{UInt8}
T = Float32 # floating point precision
pattern = (7, 6) # pattern size
imgSize = cv.Size{Int32}(size(gray)[[3, 2]]...)
criteria = cv.TermCriteria(cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001) # termination criteria

# Arrays to store object points and image points from all the images.
objpoints = cv.InputArray[] # 3d point in real world space
imgpoints = cv.InputArray[] # 2d points in image plane.

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ...., (6,5,0)
objp = zeros(T, 2, 1, prod(pattern))
n = 1
for i in 1:7, j in 1:6
	objp[1, 1, n] = i - 1
	objp[2, 1, n] = j - 1
	n += 1
end

# Find the chess board corners
isValid, corners = cv.findChessboardCorners(gray, cv.Size{Int32}(pattern...))

# If found, add object points, image points (after refining them)
if isValid
	push!(objpoints, objp)
end

corners2 = cv.cornerSubPix(gray, corners, cv.Size{Int32}(11, 11), cv.Size{Int32}(-1, -1), criteria)
push!(imgpoints, corners2)

# camera cal
fake_mtx = cv.Mat(Array{T}(undef, 1, 3, 3))
fake_dist = cv.Mat(Array{T}(undef, 1, 1, 5))
fake_r = cv.InputArray[Array{T}(undef, 3, 1, prod(pattern)) for _ in 1:1]
fake_t = cv.InputArray[Array{T}(undef, 3, 1, prod(pattern)) for _ in 1:1]
flags = cv.CALIB_ZERO_TANGENT_DIST + cv.CALIB_FIX_K3 + cv.CALIB_FIX_K2 + cv.CALIB_FIX_ASPECT_RATIO

ret, mtx, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, imgSize, fake_mtx, fake_dist, fake_r, fake_t, flags, criteria)

```

This returns error as follows.

```julia
ERROR: OpenCV(4.10.0) /workspace/srcdir/opencv/modules/calib3d/src/calibration.cpp:3515: error: (-210:Unsupported format or combination of formats) objectPoints should contain vector of vectors of points of type Point3f in function 'collectCalibrationData'

Stacktrace:
 [1] jlopencv_cv_cv_calibrateCamera(arg1::CxxWrap.StdLib.StdVectorAllocated{…}, arg2::CxxWrap.StdLib.StdVectorAllocated{…}, arg3::OpenCV.Size{…}, arg4::OpenCV.CxxMatAllocated, arg5::OpenCV.CxxMatAllocated, arg6::CxxWrap.StdLib.StdVectorAllocated{…}, arg7::CxxWrap.StdLib.StdVectorAllocated{…}, arg8::Int64, arg9::OpenCV.TermCriteria)
   @ OpenCV C:\Users\\.julia\packages\CxxWrap\eWADG\src\CxxWrap.jl:668
 [2] calibrateCamera(objectPoints::Vector{…}, imagePoints::Vector{…}, imageSize::OpenCV.Size{…}, cameraMatrix::OpenCV.Mat{…}, distCoeffs::OpenCV.Mat{…}, rvecs::Vector{…}, tvecs::Vector{…}, flags::Int64, criteria::OpenCV.TermCriteria)
   @ OpenCV C:\Users\\.julia\artifacts\ca7a8cea18e4e4448959e8e169004bad36338b4f\OpenCV\src\cv_cxx_wrap.jl:2276
 [3] top-level scope
   @ c:\Users\\OneDrive\julia_dev\julia_discourse\OpenCV_calibrateCamera\fixed.jl:108
Some type information was truncated. Use `show(err)` to see complete types.

```

---

<div class="post-metadata">

**Author:** ![Kadii](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@Kadii](https://discourse.julialang.org/u/Kadii)\
**Post date:** [December 4, 2024, 11:46pm UTC](https://discourse.julialang.org/t/how-to-use-opencv-jls-calibratecamera-correctly/113765/7 "2024-12-04T23:46:50Z")

</div>

Sorry. It was my stupid mistake. The `objp` in my code has to have a shape of `(3,1,42)`. Also, I changed `flags` to 0 for now. This code should work now. Thanks for @yakir12.

```julia
gray # checkerbord pattern image: 1x398x454 OpenCV.Mat{UInt8}
T = Float32 # floating point precision
pattern = (7, 6) # pattern size
imgSize = cv.Size{Int32}(size(gray)[[3, 2]]...)
criteria = cv.TermCriteria(cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001) # termination criteria

# Arrays to store object points and image points from all the images.
objpoints = cv.InputArray[] # 3d point in real world space
imgpoints = cv.InputArray[] # 2d points in image plane.

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ...., (6,5,0)
objp = [T[i-1, j-1, 0] for i in 1:pattern[1], j in 1:pattern[2]]
objp = reshape(stack(objp[:]), 3, 1, prod(pattern))

# Find the chess board corners
isValid, corners = cv.findChessboardCorners(gray, cv.Size{Int32}(pattern...))

# If found, add object points, image points (after refining them)
if isValid
	corners2 = cv.cornerSubPix(gray, corners, cv.Size{Int32}(11, 11), cv.Size{Int32}(-1, -1), criteria)
	push!(objpoints, objp)
	push!(imgpoints, corners2)
end

# camera cal
fake_mtx = cv.Mat(Array{T}(undef, 1, 3, 3))
fake_dist = cv.Mat(Array{T}(undef, 1, 1, 5))
fake_r = cv.InputArray[Array{T}(undef, 3, 1, prod(pattern)) for _ in 1:1]
fake_t = cv.InputArray[Array{T}(undef, 3, 1, prod(pattern)) for _ in 1:1]
flags = 0

ret, mtx, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, imgSize, fake_mtx, fake_dist, fake_r, fake_t, flags, criteria)

```