Global Positioning from a Single Image of a Rectangle in Conical Perspective

Abstract

This article presents a method to obtain the overall positioning of the focus of a camera from an image that includes a rectangle in a fixed reference with known position and dimension. This technique uses basic principles of descriptive geometry introduced in engineering courses. The document will first show how to obtain the dihedral projections of a rectangle after three turns and one translation. Secondly, we will proceed to obtain the image of the rectangle rotated in a conical perspective, taking the elevation plane as the drawing plane and a specific point in space as the view point, and represented in the dihedral system. Thirdly, we proceed with the inverse perspective transformation; we will expose a method to obtain the coordinates in the space of a rectangle obtained from an image. Finally, we check the method experimentally by taking an image of the rectangle with a camera in which the coordinates in the drawing plane (center of the image) are the only available position information. Then, the positioning and orientation of the camera in 3D will be obtained.

1. Introduction

Pose determination is to estimate the position and orientation of one calibrated camera using a set of correspondences between 3D control points and 2D image points [1]. Determination of surface orientation has important applications such robotics, object recognition, 3D measurement or tracking of moving objects. Magee [2] was the first to present a procedure for determining the unique position of a robot in a three dimensional space. That method has been continuously improved in different areas as large non-cooperative satellites [3] or Unmanned Aerial Vehicle (UAV) Control [4,5]. Different methods for monocular pose estimation have been studied in the past [6,7,8,9]. More recently, marker-based positioning systems as ArUco, Chilitags, ApriTags, or ArToolKit, among others, have been introduced to estimate quantitative changes in distances and orientations in many technological applications, such as autonomous robots [10,11,12], unmanned vehicles [13,14,15,16], or virtual assistants [17,18,19,20].

The calibration and orientation of a camera from its images has been obtained through different approaches in the past with good precision through techniques such as using a single image with four coplanar control lines [21], three coplanar circles [22], using parallelogrammatic grid points [23], or even using only three points in the world coordinate system when a multiple camera system is used [24]. Becker [25] introduced a new technique using an iterative method which solves the parameters that minimize vanishing point dispersion to solve for radial and decentering lens distortion directly from the results of vanishing point estimation, precluding the need for special calibration templates. Single image based reconstruction has been deeply studied by many authors such as Delage [26], Wilczkowiak et al. [27], Sturm and Maybank [28] or Micusik et al. [29], assuming perpendicularity and parallelism to recover the lack of information. Other authors such as Penna [30] showed that there is sufficient information in the two-dimensional perspective projection of an arbitrary quadrilateral of known shape and size in three-space to determine the exact three-dimensional coordinates of its vertices, generalizing known results for rectangles. Duan [1] used the projection of a trapezium for pose estimation and plane measurement in a very simple way. An iterative algorithm was used by Hong & Yang [31] to establish the relationship between parameters and the world coordinates of a given 3D calibration point. Nevertheless, additional studies via rectangular structures as in Haralick [8] or Wefelscheid [32] use similar concepts with a different approach. In contrast, our research used the information provided by the dihedral projections of a rectangle to determine the image of the rectangle rotated in a conical perspective.

Computer vision has been used in areas, such as unmanned vehicles, to estimate relative 3D position and altitude using algorithms based on four feature points, such as square and parallel relations, to avoid complicated calculations [33]. An algorithm for pose estimation based on volume measurement of tetrahedra composed of target points and the lens center of the vision system was proposed by Abidi [6]. 3D model reconstruction from a single image calibrating a camera and recovering the geometry and the photometry of objects was part of Guillou’s [34] research and a novel method to find the initial solutions for iterative camera pose estimation using coplanar points was provided by Zhou [35]. A general photogrammetric method for determining object position and orientation was presented by Yuan [36]. Recently Wang et al. [37] studied active relocalization of a 3D camera pose from a single reference image; a recent and challenging problem in computer vision and robotics. Pose estimation of smooth metal parts is an important task in intelligent manufacturing. Ulrich [38], Sakcak [39], Han [40] and He [41] proposed a solution using a monocular camera and corresponding practical algorithms.

The adjustment of tools in machining centers is usually the slowest and most critical operation in the positioning of the machined parts. The provision of a tool that includes machine displacements and images with edge detection can be adjusted at micrometric scales without the need for lasers or probes. Other possible application could be the metrology by vision, since in the case of characteristics to be measured in the same plane of a rectangle of known dimensions, the dihedral perspective of the aforementioned characteristic can be obtained and non-contact metrological checks can be performed immediately (in real time) compensating many of the existing errors. This is an essential aspect to achieve the efficiency and flexibility required by controls in production systems in Industry 4.0.

In this work a new method to obtain the camera coordinates of a rectangle from its image is proposed. This method is based in the principles of descriptive geometry as developed by Monge [42], which is studied in basic engineering courses. In order to explain the method a remembrance of the construction of a rectangle in conical perspective is described, and an inverse path is proposed. Finally, an experiment has been designed to check the precision of the method.

2. Dihedral Projection of a Rectangle. Rotations and Translations

In this case the problem input data is the dihedral projection of a rectangle in which the length of one side L is known. Therefore, it is represented by its coordinates $x^{*}$ and $z^{*}$. This rectangle is rotated by three angles $\varphi$, $\xi$, and $\theta$. The transformation matrices are applied to obtain a global rotation matrix and the translation is made to the point $X_0$, the coordinates of the vertices are then obtained and presented in a table of dihedral information. The Top View of the dihedral would be represented by the $xy$ plane, and the elevation of the dihedral is the $xz$ plane. The projections of the rectangle on both planes will be its dihedral representation [42,43].

3. Conical Projection

With the point of view with coordinates $(V_x,V_y,0)$ and represented in the same dihedral system as the rectangle, where the Front View coincide with the image plane and $V_y$ is the focal distance, the vertices coordinates $(x^{*},z^{*})$ in the rectangle in the conical perspective are obtained. The method used consists in creating, from the Top View, a line that passes through $(V_x,V_y,0)$ and the Top Projection of the point P$(P_x,P_y,0)$ obtaining the intersection with the image plane which will be the coordinate $x_p^{*}$. This coordinate $x_p^{*}$ is calculated by drawing the line that passes through $(V_x,V_y,0)$ and the Front View of the point P$(P_x,P_y,0)$ and obtaining the intersection with the vertical line that starts at $x^{*}$. Consequently, the conical projection of the point in the image plane with coordinates $(x^{*},z^{*})$ is calculated. When this operation is performed Figure 1 with the four points in the rectangle, the rectangle in conical perspective is obtained.

4. Obtaining the Possible Front View and Top View of Dihedral Projection of the Rectangle

Using the coordinates in the conical perspective and knowing the projection of the point of view in the Front and Top planes, the coordinates of the edges of the rectangle are calculated (Figure 2).

To do this, we use part of the geometric method described by Wefelscheid et al. [32] obtaining auxiliary points that help us calculate the dihedral projection from the conical projection. These auxiliary points are:

• Point $M^{*}$. Intersection between the lines joining $P_1^{*}{P}_3^{*}$ and $P_2^{*}{P}_4^{*}$.
• Vanishing point $V_1^{*}$ as intersection between the lines $P_1^{*}{P}_2^{*}$ and $P_3^{*}{P}_4^{*}$.
• Vanishing point $V_2^{*}$ as intersection between the lines $P_1^{*}{P}_4^{*}$ and $P_2^{*}{P}_3^{*}$.
• Midpoint of edges $P_{12}^{*}$, $P_{23}^{*}$, $P_{34}^{*}$, $P_{14}^{*}$ as intersection of the lines that are drawn from the vanishing point to $M^{*}$ with the respective edges.

The auxiliary points are represented in Figure 3.

These operations can be done graphically by drawing on paper, so it is computationally reduced to intersections between lines that are defined each by two points as in

Table 1. Obtaining auxiliary points as intersections of lines that go through two points.

Support Point	Line 1		Line 2
$M^{*}$	$P_1^{*}$	$P_3^{*}$	$P_2^{*}$	$P_4^{*}$
$V_1^{*}$	$P_1^{*}$	$P_2^{*}$	$P_3^{*}$	$P_4^{*}$
$V_2^{*}$	$P_1^{*}$	$P_4^{*}$	$P_2^{*}$	$P_3^{*}$
$P_{12}^{*}$	$P_1^{*}$	$P_2^{*}$	$M^{*}$	$V_2^{*}$
$P_{23}^{*}$	$P_2^{*}$	$P_3^{*}$	$M^{*}$	$V_1^{*}$
$P_{34}^{*}$	$P_3^{*}$	$P_4^{*}$	$M^{*}$	$V_2^{*}$
$P_{14}^{*}$	$P_1^{*}$	$P_4^{*}$	$M^{*}$	$V_1^{*}$

as represented in Figure 3.

After obtaining these points, a proposal of the Front View of the rectangle is based on two graphic properties:

• The points of the Front View projection are in the lines that start from the center point V whose coordinates are ($V_x,0,V_z$) and go though the point of the image $P_1^{*}$, $P_2^{*}$, $P_3^{*}$, $P_4^{*}$, $M^{*}$, $P_{12}^{*}$, $P_{23}^{*}$, $P_{34}^{*}$, $P_{14}^{*}$.
• In the dihedral projection the center points are in the geometric center of the segment of the side, dividing this side in two. For example, $P_{12}$ is in the center point of the segment that joins $P_1$ and $P_2$.
• Opposite sides are parallel in the dihedral projection.

By taking advantage of these two properties and a trigonometric interrelation, a first proposal of a rectangle in Front View can be obtained by the following procedure:

• In the triangle $P_1^{*}{P}_2^{*}{V}^{*}$ which is divided by the segment $V^{*}{P}_{12}^{*}$, a line that starts at $P_{12}$ and its intersection with the lines $V^{*}{P}_1^{*}$ and $V^{*}{P}_2^{*}$ is equidistant, in a way that a possible point $P_{12}$ in Front View can be obtained as shown in Figure 4.
• An arbitrary distance d to obtain $P_{12}$ is taken.
• The normal vector of the line $P_1{P}_2$ in dihedral will be found by a rotation of the vector $V^{*}{P}_{12}$ an angle $\omega$ reached using the trigonometric relation (1):

  $$tan\omega =\frac{2}{-cot\alpha +cot\beta .}$$

  (1)

  The deduction of this expression is detailed in Appendix A, where $\alpha$ is the angle between $\overrightarrow{V^{*}{P}_2^{*}}$ and $\overrightarrow{V^{*}{P}_{12}^{*}}$; and $\beta$ is the angle between $\overrightarrow{V^{*}{P}_1^{*}}$ and $\overrightarrow{V^{*}{P}_{12}^{*}}$ represented in the Figure 4 and expressed by the Equations (2) and (3).

  $$\alpha =arccos\frac{\overrightarrow{V^{*}{P}_2^{*}}·\overrightarrow{V^{*}{P}_{12}^{*}}}{|\overrightarrow{V^{*}{P}_2^{*}}\left|\right|\overrightarrow{V^{*}{P}_{12}^{*}}|}$$

  (2)

  $$\beta =arccos\frac{\overrightarrow{V^{*}{P}_1^{*}}·\overrightarrow{V^{*}{P}_{12}^{*}}}{|\overrightarrow{V^{*}{P}_1^{*}}\left|\right|\overrightarrow{V^{*}{P}_{12}^{*}}|.}$$

  (3)
• Points $P_1$ and $P_2$ are obtained from the intersection of the line defined by the point $P_{12}$ and the vector $\overrightarrow{V^{*}{P}_{12}}$ rotated an angle $\omega$. Once the orientations are calculated, starting from a point in the line $V^{*}{P}_1^{*}$ and drawing a line that intersects the line $V^{*}{P}_2^{*}$ gets the hypothetical side $P_1{P}_2$ already in the Front View of the dihedral projection.
• With the presumed points $P_1$ and $P_2$ of the Front View in the dihedral projection, it is possible to calculate, with the central point V in the Top view (which is at a distance equal to the focal distance from the drawing plane), the projection in Top view of points $P_1$ and $P_2$ as shown in Figure 5.
  To get the Top View of the rectangle from the hypothetical Front projection of the side $P_1{P}_2$ it is possible to obtain its Top projection from the Top projection of V which is at a focal distance from the drawing plane. As an example, the y coordinate of the point $P_1$ will be obtained by the intersection of the line that joins V in the Top View with the point $x_{P1}^{*}$ and the coordinate $x_{P1}$ as shown in Figure 6 and in Equation (4). In the same way we proceed to obtain the y coordinate of the point $P_2$.

  $$y_{p1}=V_y+\frac{V_y}{V_x-x_{P1}^{*}}$$

  (4)
• With the Top and Front projection of the points $P_1$ y $P_2$, the length in pixel units of the segment is calculated, and the real distance d that means the length $P_1{P}_2$ matched with the length of the edge of the rectangle is obtained.
  With the complete coordinates of the points $P_1$ y $P_2$ the distance between both points is calculated.
  Being a proportional geometric problem, the solution is found in a single step from the application of Thales’ theorem.
• A similar procedure is done with the triangle $P_2^{*}{P}_3^{*}{V}^{*}$ divided by $V^{*}{P}_{23}$. Consequently, the two orientations of the edges which will have the projection of the rectangle in the dihedral system are calculated.
  Getting the rest of the points is direct as we have the orientations in Top View, finding the points $P_3$ y $P_4$ using a correlative method.

Once the three-dimensional coordinates of the rectangle are found, it is possible to perform any operation related to positioning and orientation of the camera or distance calculation and angle modification. The full method is represented in Figure 7.

5. Comments on the Described Method and Comparison with Previous Ones

The new method has many advantages over the methods of Haralik [8] and Wefelscheid [32] which are the most used:

• It has the advantage of working with points and lines as it works in descriptive geometry science, making the calculations much more intuitive, based on simple sequences.
• The method makes little use of trigonometric functions. The only trigonometric relation used is the tangent angle function between two vectors which induces very few floating point errors. In addition to this, a rotation is applied on the vectors to redraw the rectangle edges in dihedral.
• It is a direct method without iterations or matrix inversions.
• As it is sequential, we can perform checks and easily determine where an error may have occurred. Once the calculations have been verified, the equations in mega formulas that save the calculation times can be exposed. The algebraic operations to obtain the points barely exceed one hundred which equates to less than thousandths of a second of computer time.

With the results, several verification can be performed since it provides data which can already be calculated such as:

• The length of the second edge of the rectangle, since it has not been used for the calculation of the inverse perspective.
• The spatial lines that join the point V with the vanishing points $V_1$ and $V_2$ in the drawing plane, are parallel to the sides of the rectangle so they are perpendicular to each other. Consequently, the scalar product must be zero, which means that the starting data (the focal length) can actually be determined from the vanishing points [34].

6. Positioning of the Camera in Coordinate System Defined in the Rectangle

Object tracking systems in space through images, navigation systems or calculation of distances and angles in images can be easily made from the coordinates of known rectangles that serve as a reference (Figure 8). Therefore, they can be used for tracker systems in positioning of parts in specific coordinate systems.

In global coordinates (5):

$$\overrightarrow{i}={\displaystyle \frac{\overrightarrow{P_1{P}_2}}{|\overrightarrow{P_1{P}_2}|}}\overrightarrow{j}={\displaystyle \frac{\overrightarrow{P_4{P}_1}}{|\overrightarrow{P_4{P}_1}|}}\overrightarrow{k}={\displaystyle \frac{\overrightarrow{P_1{P}_2}\wedge \overrightarrow{P_4{P}_1}}{|\overrightarrow{P_1{P}_2}\wedge \overrightarrow{P_4{P}_1}|}}=\overrightarrow{i}\wedge \overrightarrow{j}.$$

(5)

The position in global coordinates would be given by (6):

$$\overrightarrow{X_v}=\left\{\begin{array}{c}{x}_v-x_M\\ y_v-y_M\\ z_v-z_M\end{array}\right\},$$

(6)

and the check in local coordinates would be obtained according to (7):

$$\overrightarrow{X_v}=\left\{\begin{array}{c}\overrightarrow{MV}·\overrightarrow{i}\\ \overrightarrow{MV}·\overrightarrow{j}\\ \overrightarrow{MV}·\overrightarrow{k}\end{array}\right\}.$$

(7)

In global coordinates the orientation of the camera follows the ${\overrightarrow{j}}_v$ vector (8):

$$\left\{\begin{array}{c}0\\ 1\\ 0\end{array}\right\}{\overrightarrow{j}}_v.$$

(8)

In rectangle coordinates (9) is obtained, which is the projection of j on each of the three axes that coincides with the component in y of the three vectors $\overrightarrow{i}$, $\overrightarrow{j}$, and $\overrightarrow{k}$ expressed in global coordinates.

$$\left\{\begin{array}{c}{j}_{vx}\\ j_{vy}\\ j_{vz}\end{array}\right\}=\left\{\begin{array}{c}\overrightarrow{i}·{\overrightarrow{j}}_v\\ \overrightarrow{j}·{\overrightarrow{j}}_v\\ \overrightarrow{k}·{\overrightarrow{j}}_v\end{array}\right\}.$$

(9)

7. Experimental Tests

For the test of the method all the images are taken with a CASIO Exilim EX-ZR200 digital camera with a resolution of 4608 × 3456 (16MPixels) and a sensor dimension of 6.16 × 4.62 mm (1/2.3”). After the calibration the focal distance is 4.6 mm and the central point is not in the middle of the image but at coordinates (2186, 1991). The camera is calibrated by standard method of Computer Vision by images taken of a chessboard. To analyze the position a Coordinate Measuring Machine (CMM) model Pioneer DEA 03.10.06 with measuring strokes 600 × 1000 × 600 mm has been used as seen in Figure 9. The Maximum Permissible Error of the DEA in the measurements is $2.8+4.0$ L/1000 $\mathsf{\mu }$m. The software for the measurements was PC-DMIS.

The procedure followed was as follows:

• Place a DIN A4 size paper on the granite table.
• Position the camera on a tripod.
• Take photo of DIN A4 paper remotely so as not to influence the captured image.
• Take six points of the camera housing according to the 3-2-1 method [44].
• Calculate the position of the camera focus with respect to the center of the A4 sheet from the palpated points.
• Contrast this position with that obtained by image analysis.
• Repeat several times the steps 3–6, while varying the position and angles of the camera.

The parameters used for the test are summarized in

Table 2. Parameters used for the test.

Camera Parameters			A4 Sheet Parameters			Focus Position in Camera Coordinates
Sensor size	1/2.3”	6.16 × 4.62 mm	width	210	mm	x	30	mm
Width	4608	pixels	height	297	mm	y	30	mm
Height	3456	pixels				z	3.5	mm
Focus	3433	pix (4.6 mm)
$V_x$	2186	pixels
$V_y$	1991	pixels
1 pixel	0.0013368	mm

. The coordinates and distances of the paper center obtained by CMM and the image analysis including the differences of the coordinates and distances between both, where subindex e refers to experimental data calculated with the CMM and subindex t refers to theoretical values calculated by the image analysis algorithm, are shown in the

Table 3. Coordinates, distances, and errors obtained by CMM and image analysis where subindex e refers to experimental data calculated with the CMM and subindex t refers to theoretical values calculated by the image analysis algorithm. Coordinates x, y, z and distances d are expressed in mm. Differences of the coordinates and distances (errors) are presented in percentages.

	CMM				Image
Image	${\mathit{x}}_{\mathit{e}}$	${\mathit{y}}_{\mathit{e}}$	${\mathit{z}}_{\mathit{e}}$	${\mathit{d}}_{\mathit{e}}$	${\mathit{x}}_{\mathit{t}}$	${\mathit{y}}_{\mathit{t}}$	${\mathit{z}}_{\mathit{t}}$	${\mathit{d}}_{\mathit{t}}$	${\mathit{ϵ}}_{\mathit{x}}$ (%)	${\mathit{ϵ}}_{\mathit{y}}$ (%)	${\mathit{ϵ}}_{\mathit{z}}$ (%)	${\mathit{ϵ}}_{\mathit{d}}$ (%)
1	201.22	222.21	367.44	474.21	210.09	200.44	365.64	466.91	4.41	−9.79	−0.49	−1.54
2	−195.04	218.71	367.43	469.98	−180.47	215.48	374.60	468.32	−7.47	−1.48	1.95	−0.35
3	88.39	469.44	378.18	609.27	111.54	473.32	380.00	617.15	26.19	0.83	0.48	1.29
4	157.90	225.84	374.79	465.19	148.48	212.45	381.40	461.13	−5.97	−5.93	1.76	−0.87
5	−113.73	672.02	390.02	785.27	−120.61	687.26	391.43	800.06	6.06	2.27	0.36	1.88
6	128.21	190.00	366.23	432.05	102.81	185.02	370.68	426.85	−19.82	−2.62	1.21	−1.20
7	184.55	592.80	384.91	730.50	197.23	600.11	389.99	742.38	6.88	1.23	1.32	1.63
8	−80.07	460.32	374.07	598.53	−60.11	462.76	378.33	600.75	−24.93	0.53	1.14	0.37
9	−113.76	128.64	352.65	392.24	−89.32	120.79	354.29	384.83	−21.48	−6.10	0.47	−1.89

.

Result comments:

• The differences in measured distances are less than 2%, e.g., less than 2 cm in 1 m distance.
• The errors in the x coordinate are due to the parallelism of the lines with the image plane, but they hardly affect the distance calculation since the contribution of the x coordinate is small in the global calculation.

As can be seen in Table 3 and Figure 10, the difference between the distance from the camera to the center of the folio, measured by the image analysis and by the CMM, is less than 2%. This indicates that, visually, very close accuracies of the actual distances can be achieved. Nevertheless, analyzing each coordinate, a very high error in the x coordinates is observed in some points. These errors occur when the camera is facing the folio, being the x-axis parallel to the sensor plane. In consequence, the vanishing points in this direction are far apart, and the intersection between the lines is more imprecise. As in these cases, the camera is located at a small value of the x coordinate, the influence of this value on the global error is reduced. This indicates a limitation since the method works best, the closer the vanishing points remain.

Figure 11 shows the images obtained with the camera and used to verify the method presented in this article.

8. Conclusions

A new method has been proposed for rectangle reconstruction using elements of descriptive geometry, as used by Monge in 1847 [42], and of extensive knowledge by engineering users since it is taught in the early stages of such studies. The method presented is mainly based on the intersection between lines, as their calculations are fast and stable in computing and, therefore, minimize errors and optimize computation. The proposed process uses very few trigonometric functions of small angles that are the main source of errors in other methods, so very few floating-point errors are introduced. Additionally, the trigonometric functions are mainly used for the rotation of vectors to align the edges in dihedral projections, which also reduces the errors.

In addition, a procedure was carried out to experimentally test the calculations. The proposed technique was tested in a CMM by locating the camera through the palpation using the 3-2-1 method and the position given by the CMM was compared with the calculation from the image taken by the camera. The proposed method provides maximum errors of 2% in the measured distances. The big errors detected in individual coordinates are due to the parallelism of two sides with the image plane since, in this case, the vanishing point is distance in space and its determination by the intersection of two almost parallel lines has more variability.