With a USB camera, the UNIHIKER device and Python it is possible to create augmented reality applications. In this guide I would like to give you an introduction to this topic. I will show you how to create ArUco markers, how to recognize them in a real-time video stream with Python OpenCV and then how to display specific images instead of the markers.
Now connect the USB camera to the UNIHIKER and then switch on the UNIHIKER device.
I recommend setting up the development environment as follows (but you can adapt the steps according to your needs). A current Python 3 installation is required on your computer/laptop. You should use at least Python 3.7.3 (this version is already installed on the UNIHIKER).
Create a new folder for the project, for example named UnihikerAR. A Python virtualenv environment is created in it.
Here are the command line commands:
# create new directory
$ mkdir UnihikerAR
# change into create directory
$ cd UnihikerAR
# create python virtualenv
$ python3 -m venv .venv
# list directory (optional)
$ ls -la
# activate the virtual environment
$ source .venv/bin/activateThen create a file called requirements.txt with the following content:
setuptools==75.6.0
wheel==0.45.1
numpy==2.1.3
opencv-python==4.10.0.84
opencv-contrib-python==4.10.0.84The requirements.txt files serves as a list of packages/modules to be installed by Python pip.
# create requirements.txt via vim
(.venv) $ vim requirements.txt
# update pip (optional)
(.venv) $ pip3 install -U pip
# install all required packages/modules
(.venv) $ pip3 install -r requirements.txt
# verify installation (optional)
(.venv) $ pip3 freezeThe project and the development environment have already been created. You don't have to carry out these steps on the UNIHIKER because everything is already preinstalled there.
ArUco is integrated into OpenCV and can be used for camera calibration, pose estimation, and augmented reality. Create now a file named generate_aruco_marker.py with following content:
from os import makedirs
from os.path import dirname, abspath, join, isdir
import cv2
ARUCO_DICT_ID: int = cv2.aruco.DICT_4X4_50
ARUCO_MARKER_ID: int = 0
ARUCO_MARKER_SIZE: int = 100
if __name__ == "__main__":
current_file_path = dirname(abspath(__file__))
target_directory = join(current_file_path, "dev/markers")
target_file_path = join(target_directory, f"marker_{ARUCO_MARKER_ID}.jpg")
if not isdir(target_directory):
print(f"[INFO] Create directory: {target_directory}")
makedirs(target_directory)
aruco_dict = cv2.aruco.getPredefinedDictionary(dict=ARUCO_DICT_ID)
marker = cv2.aruco.generateImageMarker(dictionary=aruco_dict, id=ARUCO_MARKER_ID, sidePixels=ARUCO_MARKER_SIZE)
print(f"[INFO] Marker is saved: {target_file_path}")
cv2.imwrite(filename=target_file_path, img=marker)
Take a closer look at the Python script! The constants that are important are: ARUCO_DICT_ID, ARUCO_MARKER_ID and ARUCO_MARKER_SIZE.
DICT_4X4_50 is a pre-made set of 50 markers (0 to 49). 4x4 means the number of marker areas. There are also other marker sets, but for this example it is completely sufficient.
This example creates a marker with ID: 0. So each marker has a fixed ID as an assignment!
The total size of the marker is specified as 100px in the example. You can also specify values between 50 and 200. The smaller the marker, the more difficult it may be to recognize.
Here are the steps and how to create the images (markers):
# create python script via vim
(.venv) $ vim generate_aruco_marker.py
# execute python script
(.venv) $ python3 generate_aruco_marker.py
# list created marker (optional)
(.venv) $ ls -la dev/markersNow change the value of the constant ARUCO_MARKER_ID from 0 to 1 and run the script again. You should now have 2 images (marker_0.jpg and marker_1.jpg) in the directory: UnihikerAR/dev/markers.
Print out these two images and cut them out as needed.
If you want to create more markers later, you can always use the script again.
In this step you can use Python OpenCV to check whether you recognize the markers and the respective ID is displayed. This step is actually just for explanation and can be skipped.
Create a Python script called show_marker_id.py with the following content and run the script to test marker detection.
import cv2
ARUCO_DICT_ID: int = cv2.aruco.DICT_4X4_50
if __name__ == "__main__":
aruco_dict = cv2.aruco.getPredefinedDictionary(ARUCO_DICT_ID)
aruco_params = cv2.aruco.DetectorParameters()
detector = cv2.aruco.ArucoDetector(aruco_dict, aruco_params)
cap = cv2.VideoCapture(0)
while True:
ret, frame = cap.read()
if not ret or (cv2.waitKey(1) & 0xFF == ord('q')):
break
frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
corners, ids, rejected = detector.detectMarkers(frame_gray)
if ids is not None:
frame = cv2.aruco.drawDetectedMarkers(frame, corners, ids)
cv2.imshow("AR Marker Detection", frame)
cap.release()
cv2.destroyAllWindows()
The camera should activate on your computer/laptop, a new window with the stream should appear and when you hold the marker in front of the camera, the marker IDs should be displayed.
# create python script via vim
(.venv) $ vim show_marker_id.py
# execute script
(.venv) $ python3 show_marker_id.pyYou may still need to allow access to your camera.
Pressing the q key closes the window and exits the Python script.
This is the first step into the extremely cool world of augmented reality! In the next step, instead of using drawDetectedMarkers(), the code is extended to display an image for each detected ID.
As already mentioned in the marker creation, you can create additional markers and test for yourself whether they are recognized. Please note that the values for the constants ARUCO_DICT_ID should be the same in all Python scripts.
If everything has worked up to this step, create additional folders (for the images) and the Python script main.py. The images and Python script will later be loaded onto the Unihiker. You can find everything in the ZIP file (at the end of this tutorial) if you don't have your own images.
Copy the images to the folder: UnihikerAR/img/photos.
The content of main.py:
from os.path import dirname, abspath, join
from platform import node
import cv2
import numpy as np
DISPLAY_WIDTH: int = 240
DISPLAY_HEIGHT: int = 320
MARKER_SIZE: float = 0.05
OBJ_POINTS: np.ndarray = np.array(
[
[0, 0, 0],
[MARKER_SIZE, 0, 0],
[MARKER_SIZE, MARKER_SIZE, 0],
[0, MARKER_SIZE, 0]
],
dtype=np.float32
)
def draw_image_on_marker(img: np.ndarray,
rotation_vector: np.ndarray,
translation_vector :np.ndarray,
camera_matrix: np.ndarray,
dist_coefficients: np.ndarray,
image_path: str) -> np.ndarray:
"""
Draws an image onto a specified marker within the given image. The function uses
the provided rotation and translation vectors along with the camera matrix and
distortion coefficients to calculate the projection of the marker to ensure that
the overlay image is drawn accurately over the marker. The overlay image is
read from a specified path and resized to fit the marker's bounding rectangle.
:param img: The image on which to draw the overlay.
:type img: np.ndarray
:param rotation_vector: The rotation vector used for projecting points.
:type rotation_vector: np.ndarray
:param translation_vector: The translation vector used for projecting points.
:type translation_vector: np.ndarray
:param camera_matrix: The camera matrix for the scene.
:type camera_matrix: np.ndarray
:param dist_coefficients: The distortion coefficients for the camera.
:type dist_coefficients: np.ndarray
:param image_path: The file path of the overlay image to be drawn.
:type image_path: str
:return: The image with the overlay drawn on the marker.
:rtype: np.ndarray
"""
overlay_image = cv2.imread(image_path, cv2.IMREAD_UNCHANGED)
img_points, _ = cv2.projectPoints(OBJ_POINTS, rotation_vector, translation_vector, camera_matrix, dist_coefficients)
img_points = np.int32(img_points).reshape(-1, 2)
rect = cv2.boundingRect(img_points)
x, y, w, h = rect
overlay_image_resized = cv2.resize(overlay_image, (w, h))
if overlay_image_resized.shape[2] == 4:
overlay_image_resized_rgb = overlay_image_resized[:, :, :3]
overlay_alpha = overlay_image_resized[:, :, 3:] / 255.0
overlay_image_resized_rgb = (overlay_image_resized_rgb * overlay_alpha).astype(np.uint8)
else:
overlay_image_resized_rgb = overlay_image_resized
for val in range(0, 3):
img[y:y + h, x:x + w, val] = overlay_image_resized_rgb[:, :, val]
return img
if __name__ == "__main__":
current_file_path = dirname(abspath(__file__))
matrix = np.array([[800, 0, 320], [0, 800, 240], [0, 0, 1]], dtype=np.float32)
coefficients = np.zeros(5)
aruco_dict = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_50)
aruco_params = cv2.aruco.DetectorParameters()
detector = cv2.aruco.ArucoDetector(aruco_dict, aruco_params)
cap = cv2.VideoCapture(0)
if node() == 'unihiker':
cap.set(cv2.CAP_PROP_FRAME_WIDTH, DISPLAY_WIDTH)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, DISPLAY_HEIGHT)
cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
cap.set(cv2.CAP_PROP_FPS, 15)
cv2.namedWindow('AR Marker Detection', cv2.WND_PROP_FULLSCREEN)
cv2.setWindowProperty('AR Marker Detection', cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_FULLSCREEN)
while True:
ret, frame = cap.read()
if not ret or (cv2.waitKey(1) & 0xFF == ord('q')):
break
frame_height, frame_width, _ = frame.shape
target_aspect = DISPLAY_HEIGHT / DISPLAY_WIDTH
frame_aspect = frame_height / frame_width
if frame_aspect > target_aspect:
new_height = int(frame_width * target_aspect)
start_y = (frame_height - new_height) // 2
cropped_frame = frame[start_y:start_y + new_height, :]
else:
new_width = int(frame_height / target_aspect)
start_x = (frame_width - new_width) // 2
cropped_frame = frame[:, start_x:start_x + new_width]
frame = cv2.resize(cropped_frame, (DISPLAY_WIDTH, DISPLAY_HEIGHT), interpolation=cv2.INTER_LINEAR)
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
corners, ids, rejected = detector.detectMarkers(gray)
if ids is not None:
img_path = join(current_file_path, f"img/photos/monk_{ids[0][0]}.jpg")
for i in range(len(ids)):
raw_img_points = corners[i][0]
ret, r_vec, t_vec = cv2.solvePnP(OBJ_POINTS, raw_img_points, matrix, coefficients)
if ret:
frame = draw_image_on_marker(frame, r_vec, t_vec, matrix, coefficients, img_path)
flipped_frame = cv2.flip(frame, 1)
cv2.imshow("AR Marker Detection", flipped_frame)
cap.release()
cv2.destroyAllWindows()
The Python script starts the camera again (in the same size/resolution as it will later be on the Unihiker). If you hold the marker in front of the camera, an image should appear.
Note: Only one marker can be recognized in this example. If you hold a marker in front of the camera where there is no image, the script crashes with an error message.
If you want to use other names, search for the line of code: img_path = join(current_file_path, f"img/photos/monk_{ids[0][0]}.jpg") and adapt it to your needs.
# create directories
(.venv) $ mkdir -p img/photos
# create script via vim
(.venv) $ vim main.py
# execute script
(.venv) $ python3 main.py
# stop virtualenv (after you are done)
(.venv) $ deactivateThe result should be similar to this:
In this step I am assuming that you have connected the Unihiker to the USB camera and (via USB) to your computer. After the upload the files and folders on Unihiker should look like this:
# list all files and folder (optional)
$ tree .
.
|-- img
| `-- photos
| |-- monk_0.jpg
| `-- monk_1.jpg
`-- main.pySCP is used for the upload (but you can also choose other options such as SMB). The Secure copy protocol (SCP) makes the transfer of data between systems very easy. For the transmission, it is based on SSH (Secure Shell) and also uses this for authentication.
In both protocols, root is used as the user and dfrobot is used as the password (if you have not changed the basic configuration).
Here are the commands:
# create a target directory (on Unihiker)
$ ssh [email protected] -C 'mkdir UnihikerAR'
# copy main.py (to Unihiker)
$ scp main.py [email protected]:/root/UnihikerAR/
# copy pictures (to Unihiker)
$ scp -r img [email protected]:/root/UnihikerAR/
# verify target files/folders on Unihiker (optional)
$ ssh [email protected] -C 'tree /root/UnihikerAR'Now it's time to do everything on the Unihiker device. You can run the application via the Unihiker touchscreen or via command line.
I show the command line execution here:
# start SSH session
$ ssh [email protected]
# change directory
$ cd /root/UnihikerAR
# run application
$ python3 main.pyAfter a few seconds you should see the stream on the Unihiker display and, if you hold the markers in front of it, the respective images.
With ctrl + c you can stop the application.
You have now learned the basics of AR and can go even deeper into the entire topic. For example, create your own photo book with extra content, build a virtual treasure map or many other things. There is still a lot to discover!!!!
UnihikerAR.zip 
