Augmented actuality with Python and OpenCV (phase 1)

recordsdata image

You might also (or can also no longer) beget heard of or seen the augmented actuality Invizimals on-line game or the Topps 3D baseball cards. The predominant belief is to render in the conceal conceal of a tablet, PC or smartphone a 3D mannequin of a particular figure on high of a card consistent with the keep of abode and orientation of the card. 


Resolve 1: Invizimal augmented actuality cards. Provide: 

Smartly, this past semester I took a course in Pc Imaginative and prescient where we studied some sides of projective geometry and opinion it shall be an moving venture to blueprint my possess implementation of a card basically based augmented actuality utility. I warn you that we’ll need a bit of algebra to build it work however I’ll strive to steal it as light as in all probability. To build basically the most out of it it’s in all probability you’ll perchance also composed be jubilant working with diversified coordinate systems and transformation matrices.

First, this put up would no longer pretend to be an tutorial, a comprehensive recordsdata or an rationalization of the Pc Imaginative and prescient ways eager and I will appropriate level out the requirements required to apply the put up. On the alternative hand, I assist you to dig deeper in the concepts that can seem along the reach.

Secondly, build no longer inquire of some expert taking a seek results. I did this appropriate for fun and there are quite lots of choices I made that would had been accomplished better. The predominant belief is to blueprint a proof of belief utility.


With that said, right here it goes my pick on it.

The keep build we originate?

Having a seek on the venture as a total can also build it seem more no longer easy than it in truth is. Fortunately for us, we’ll be in a local to divide it into smaller functions that, when mixed one on high of one other, will enable us to beget our augmented actuality utility working. The query now is, which are these smaller chunks that we desire? 

Let’s pick a wiser seek into what we are attempting to develop. As acknowledged earlier than, we are attempting to venture in a conceal conceal a 3D mannequin of a figure whose keep of abode and orientation matches the keep of abode and orientation of some predefined flat surface. Furthermore, we are attempting to build it in true time, so that if the surface changes its keep of abode or orientation the projected mannequin does so accordingly.

To develop this we first can also composed be in a local to name the flat surface of reference in a image or video physique. As soon as identified, we are in a position to simply pick the transformation from the reference surface image (2D) to the target image (2D). This transformation is opinion as homography. On the alternative hand, if what we desire is to venture a 3D mannequin positioned on high of the reference surface to the target image we desire to increase the earlier transformation to deal with cases had been the discontinuance of the direct venture in the reference surface coordinate system is diversified than zero. That is also carried out with a bit of algebra. In a roundabout diagram, we are in a position to also composed observe this transformation to our 3D mannequin and plan it on the conceal conceal. Bearing the earlier functions in mind our venture will even be divided into:

          1.  Look the reference flat surface.

          2.  Estimate the homography.

          Three.  Derive from the homography the transformation from the reference surface coordinate system to the target image coordinate system.

          Four.  Mission our 3D mannequin in the image (pixel dwelling) and plan it.

AR - Web page 1(1)

Resolve 2: Overview of the total course of that brings to existence our augmented actuality utility.

The predominant instruments we’ll utilize are Python and OpenCV on memoir of they are every originate supply, straightforward to plan up and utilize and it’s instant to make prototypes with them. For the significant algebra bit I steadily is the usage of numpy.

Recognizing the target surface

From the many in all probability ways that exist to assign object recognition I made up my mind to form out the challenge with a characteristic basically based recognition manner. This roughly suggestions, with out going into worthy detail, consist in three predominant steps: characteristic detection or extraction, characteristic description and characteristic matching.

Feature extraction

Roughly talking, this step consists in first taking a seek in every the reference and target photos for functions that stand out and, in some reach, picture phase the object to be known. This functions will even be later outdated to search out the reference object in the target image. We’re going to be in a position to recall we’ve stumbled on the object when a licensed collection of certain characteristic matches are stumbled on between the target and reference photos. For this to work it’s extreme to beget a reference image where the ideally satisfactory component seen is the object (or surface, in this case) to be stumbled on.  We don’t are attempting to detect functions which will no longer be phase of the surface. And, though we’ll address this later, we’ll utilize the measurement of the reference image when estimating the pose of the surface in a scene.

For a plan or level of a image to be labeled as characteristic it will also composed fulfill two significant properties: first of all, it will also composed newest some strong level at least in the neighborhood. Lawful examples of this can also very well be corners or edges. Secondly, since we don’t know beforehand which will in all probability be, for instance, the orientation, scale or brightness prerequisites of this identical object in the image where we are attempting to understand it a characteristic can also composed, ideally, be invariant to transformations; i.e, invariant in opposition to scale, rotation or brightness changes. As a rule of thumb, the more invariant the simpler.

Smooth Design - Web page 1

Resolve Three: On the left, functions extracted from the mannequin of the surface I steadily is the usage of. On the honest, functions extracted from a sample scene. Remark how corners had been detected as pastime functions in the rightmost image.

Feature description

As soon as functions had been stumbled on we are in a position to also composed acquire a appropriate illustration of the records they supply. This would perchance perchance enable us to look for them in other photos and likewise to make a measure of how identical two detected functions are when being in contrast. That is had been descriptors roll in.  A descriptor supplies a illustration of the records given by a characteristic and its atmosphere. As soon as the descriptors had been computed the object to be known can then be abstracted to a characteristic vector,  which is a vector that includes the descriptors of the keypoints stumbled on in the image with the reference object.

That is certain a fine belief, however how can it in truth be accomplished? There are just a few algorithms that extract image functions and compute its descriptors and, since I won’t gallop into worthy more detail (a total put up can also very well be devoted handiest to this) even as you happen to is susceptible to be drawn to incandescent more pick a seek at SIFT, SURF, or Harris. The one we steadily is the usage of used to be developed on the OpenCV Lab and it’s known as ORB (Oriented FAST and Circled BRIEF). The form and values of the descriptor depend on the algorithm outdated and, in our case,  the descriptors bought will in all probability be binary strings.

With OpenCV, extracting functions and its descriptors via the ORB detector is as straightforward as:

img = cv2.imread('scene.jpg',zero)

# Launch ORB detector
orb = cv2.ORB_create()

# acquire the keypoints with ORB
kp = orb.detect(img, None)

# compute the descriptors with ORB
kp, des = orb.compute(img, kp)

# plan handiest keypoints plan,no longer measurement and orientation
img2 = cv2.drawKeypoints(img, kp, img, coloration=(zero,255,zero), flags=zero)

Feature matching

As soon as we’ve stumbled on the functions of every the object and the scene had been the object is to be stumbled on and computed its descriptors it’s far time to look for matches between them. The most efficient reach of doing right here is to select the descriptor of every characteristic in the first plan, compute the space to the total descriptors in the 2d plan and return the closest one as basically the most efficient match (I will even composed assert right here that it’s extreme to procure a reach of measuring distances appropriate with the descriptors being outdated. Since our descriptors will in all probability be binary strings we’ll utilize Hamming distance). That is a brute pressure reach, and more subtle suggestions exist.

As an illustration, and right here is what we’ll be also the usage of, we would test that the match stumbled on as explained earlier than shall be basically the most efficient match when computing matches the alternative reach spherical, from functions in the 2d plan to functions in the first plan. This means that every functions match every other. As soon as the matching has accomplished in every directions we’ll pick as honorable matches handiest these that fulfilled the earlier condition. Resolve Four items basically the most efficient 15 matches stumbled on the usage of this diagram.

One other technique to diminish the collection of unsuitable positives shall be to study if the space to the 2d to simplest match is below a licensed threshold.  If it’s, then the match is opinion to be honorable.


Resolve Four: Closest 15 brute pressure matches stumbled on between the reference surface and the scene

In a roundabout diagram, after matches had been stumbled on, we are in a position to also composed give an explanation for some criteria to procure if the object has been stumbled on or no longer. For this I defined a threshold on the minimal collection of matches that can also composed be stumbled on. If the collection of matches is above the brink, then we recall the object has been stumbled on. Otherwise we pick into memoir that there isn’t enough proof to notify that the recognition used to be triumphant.

With OpenCV all this recognition course of will even be accomplished in about a traces of code:

cap = cv2.imread('scene.jpg', zero)    
mannequin = cv2.imread('mannequin.jpg', zero)
# ORB keypoint detector
orb = cv2.ORB_create()              
# produce brute pressure  matcher object
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=Legal)  
# Compute mannequin keypoints and its descriptors
kp_model, des_model = orb.detectAndCompute(mannequin, None)  
# Compute scene keypoints and its descriptors
kp_frame, des_frame = orb.detectAndCompute(cap, None)
# Match physique descriptors with mannequin descriptors
matches = bf.match(des_model, des_frame)
# Form them in the list of their distance
matches = sorted(matches, key=lambda x: x.distance)

if len(matches) > MIN_MATCHES:
    # plan first 15 matches.
    cap = cv2.drawMatches(mannequin, kp_model, cap, kp_frame,
                          matches[:MIN_MATCHES], zero, flags=2)
    # expose end result
    cv2.imshow('physique', cap)
    print "No longer enough matches had been stumbled on - %d/%d" % (len(matches),


On a closing conceal and earlier than entering into the subsequent step of the course of I must level out that, since we desire a true time utility, it would had been better to enforce a monitoring methodology and never appropriate gross recognition. That is attributable to the reality that object recognition will in all probability be performed in every physique independently with out taking into memoir earlier frames that would add precious details about the positioning of the reference object. One other component to beget in mind is that, the more uncomplicated to stumbled on the reference surface the more sturdy detection will in all probability be. In this particular sense, the reference surface I’m the usage of would perchance perchance no longer be basically the most efficient option, however it helps to cherish the course of.

Homography estimation

As soon as we’ve identified the reference surface in the most contemporary physique and beget a plan of honorable matches we are in a position to proceed to estimate the homography between every photos. As explained earlier than, we are attempting to search out the transformation that maps functions from the surface plane to the image plane (seek Resolve 5). This transformation will can also composed be updated every original physique we course of.


Resolve 5: Homography between a plane and a image. Provide: F. Moreno.

How will we acquire the kind of transformation? Since we’ve already stumbled on a plan of matches between every photos we are in a position to undoubtedly acquire straight away by any of the prevailing suggestions (I reach we steadily is the usage of RANSAC) an homogeneous transformation that performs the mapping, however let’s gain some insight into what we are doing right here (seek Resolve 6). You might perchance be in a local to skip the following phase (and proceed learning after Resolve 10) if desired, since I will handiest point out the reasoning in the assist of the transformation we’ll estimate.

What we’ve is an object (a plane in this case) with known coordinates in the, let’s negate, World coordinate system and we pick a image of it with a camera positioned at a licensed keep of abode and orientation with recognize to the World coordinate system. We’re going to be in a position to recall the camera works following the pinhole mannequin, which roughly reach that the rays passing through a 3D level p and the corresponding 2D level u intersect at c, the camera middle. A marvelous handy resource even as you happen to is susceptible to be drawn to incandescent more about the pinhole mannequin will even be stumbled on right here.


Resolve 6: Portray formation assuming a camera pinhole mannequin.  Provide: F. Moreno.

Although no longer completely ethical, the pinhole mannequin assumption eases our calculations and works well enough for our capabilities. The u, v coordinates (coordinates in the image plane) of a level p expressed in the Digicam coordinate system if we recall a pinhole camera will even be computed as (the derivation of the equation is left as an teach to the reader):


Resolve 7: Portray formation assuming a pinhole camera mannequin. Provide: F. Moreno.

The keep the focal measurement is the space from the pinhole to the image plane, the projection of the optical middle is the keep of abode of the optical middle in the image plane and k is a scaling component. The earlier equation then tells us how the image is fashioned. On the alternative hand, as acknowledged earlier than, we know the coordinates of the level p in the World coordinate system and never in the Digicam coordinate system, so we’ve so as to add one other transformation that maps functions from the World coordinate system to the Digicam coordinate system. The transformation that tells us the coordinates in the image plane of a level p in the World coordinate system is then:


Resolve 8: Computation of the projection matrix. Provide: F. Moreno.

Fortunately for us, since the functions in the reference surface plane build continually beget its z coordinate equal to zero (seek Resolve 5) we are in a position to simplify the transformation that we stumbled on above. It would perchance perchance even be easily seen that the constructed from the z coordinate and the zero.33 column of the projection matrix will continually be zero so we are in a position to plunge this column and the z coordinate from the earlier equation. By renaming the calibration matrix as A and taking into memoir that the external calibration matrix is an homogeneous transformation:


Resolve 9: Simplification of the projection matrix. Provide: F. Moreno.

From Resolve 9 we are in a position to carry out that the homography between the reference surface and the image plane, which is the matrix we’ll estimate from the earlier matches we stumbled on is:


Resolve 10: Homography between the reference surface plane and the target image plane. Provide: F. Moreno.

There are just a few suggestions that enable us to estimate the values of the homography matrix, and you maight be familiar with some of them. The one we steadily is the usage of is RANdom SAmple Consensus (RANSAC).  RANSAC is an iterative algorithm outdated for mannequin becoming in the presence of a tall collection of outliers, and Resolve 12 ilustrates the predominant give an explanation for of the course of. Since we’ll no longer guarantee that the total matches we’ve stumbled on are in truth honorable matches we’ve to select into memoir that there can also very well be some unsuitable matches (which will in all probability be our outliers) and, hence, we’ve to utilize an estimation manner that is sturdy in opposition to outliers. Resolve 11 illustrates the issues we would beget when estimating the homography if we opinion to be that there had been no outliers.


Resolve 11: Homography estimation in the presence of outliers. Provide: F. Moreno.


Resolve 12: RANSAC algorithm give an explanation for. Provide: F. Moreno.

As an illustration of how RANSAC works and to build issues clearer, recall we had the following plan of functions for which we wished to suit a line the usage of RANSAC:


Resolve 13: Initial plan of functions. Provide: F. Moreno

From the final give an explanation for offered in Resolve 12 we are in a position to acquire the actual course of to suit a line the usage of RANSAC (Resolve 14).


Resolve 14: RANSAC algorithm to suit a line to a plan of functions. Provide: F. Moreno.

A in all probability outcome of working the algorithm offered above will even be seen in Resolve 15. Remark that the first Three steps of the algorithm are handiest shown for the first iteration (indicated by the backside honest quantity), and from that on handiest the scoring step is shown.


Resolve 15: The utilize of RANSAC to suit a line to a plan of functions. Provide: F. Moreno.

Now assist to our utilize case, homography estimation. For homography estimation the algorithm is offered in Resolve sixteen. Because it’s basically math, I won’t gallop into limited print on why Four matches are significant or on estimate H. On the alternative hand, even as you happen to would are attempting to grab why and the diagram in which it’s accomplished, this is a first fee rationalization of it.


Resolve sixteen: RANSAC for homography estimation. Provide: F. Moreno.

Sooner than seeing how OpenCV can deal with this for us we are in a position to also composed  divulge about one closing component of the algorithm, which is what does it imply that a match is consistent with H. What this basically reach is that if after estimating an homography we venture into the target image the matches that had been no longer outdated to estimate it then the projected functions from the reference surface can also composed be shut to its matches in the target image. How shut they can also composed be to be opinion to be consistent is as much as you.

I realize it has been noteworthy to keep this level, however fortunately there might be a reward. In OpenCV estimating the homography with RANSAC is as straightforward as:

# assuming matches stores the matches stumbled on and 
# returned by bf.match(des_model, des_frame)
# differenciate between supply functions and vacation space functions
src_pts = np.float32([kp_model[m.queryIdx].pt for m in matches]).reshape(-1, 1, 2)
dst_pts = np.float32([kp_frame[m.trainIdx].pt for m in matches]).reshape(-1, 1, 2)
# compute Homography
M, conceal = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 5.zero)


The keep 5.zero is the brink distance to search out out if a match is consistent with the estimated homography. If after estimating the homography we venture the Four corners of the reference surface on the target image and attach them with a line we are in a position to also composed inquire of the resulting traces to enclose the reference surface in the target image. We are in a position to build this by:

# Plot a rectangle that marks the stumbled on mannequin in the physique
h, w = mannequin.shape
pts = np.float32([[zero, zero], [zero, h - 1], [w- 1, h - 1], [w- 1, zero]]).reshape(-1, 1, 2)
# venture corners into physique
dst = cv2.perspectiveTransform(pts, M)  
# connect them with traces
img2 = cv2.polylines(img_rgb, [np.int32(dst)], Legal, 255, Three, cv2.LINE_AA) 
cv2.imshow('physique', cap)


which finally ends up in:


Resolve 17: Projected corners of the reference surface with the estimated homography.

I mediate right here is enough for nowadays. On the subsequent put up we’ll seek lengthen the homography we already estimated to venture no longer handiest functions in the reference surface plane however any 3D level from the reference surface coordinate system to the target image. We’re going to be in a position to then utilize this technique to compute in true time, for every video physique, the actual projection matrix and then venture in a video movement a 3D mannequin of our alternative from an .obj file. What it’s in all probability you’ll perchance also inquire of on the tip of the subsequent put up is one thing corresponding to what it’s in all probability you’ll perchance also seek in the gif below:


As continually, I will upload the total code of the venture to boot to some 3D units to GitHub for you to envision when publishing phase 2.

Be taught More

What do you think?

0 points
Upvote Downvote

Total votes: 0

Upvotes: 0

Upvotes percentage: 0.000000%

Downvotes: 0

Downvotes percentage: 0.000000%