YOLO Object Detection on the Raspberry Pi AI Hat+ | How to Write Custom Python Code - Tutorial Australia (2024)

In this guide, we will be exploring how to set up YOLO object detection with the Raspberry Pi AI HAT, and more importantly, learning how to apply this in your Python projects. We will be taking a look at how to install the required hardware and firmware as well as how to set up and use the object detection Python pipelines. The result of this guide will have you equipped with an understanding of this whole setup, as well as three different example scripts we have written. One will "do something" when an object is detected, another when a certain number of objects are detected, and the last when an object is detected in a certain location.

Like most of our other computer vision guides this one is a fun one, so let's get into it!

Install the 4 standoffs that came with your AI HAT. Four long and four short screws come with the standoffs and it doesn't matter which ones are used.

The connectors for the camera use a similar tab connectors as the PCIe connector. On both the camera and the Pi's connector slot lift the tab, insert the cable so it sits square, and then push the connector back down to secure it in place.

First things first, we need to install Pi OS onto the micro SD card. Using the Raspberry Pi Imager, select Raspberry PI 5 as the Device, Raspberry Pi OS (64-bit) as the Operating system, and your microSD card as the storage device.

NOTE: INSTALLING PI OS ONTO THE MICROSD CARD WILL WIPE ALL DATA ON IT.

This process may take a few minutes to download the OS and install it. Once the process has finished, insert it into the Pi and boot it up. Your Pi will run through a first-time installation and ensure that you connect it to the internet.

Let's start by installing the required firmware and software to run the AI HAT. Open up a new terminal window and start by updating your Pi with:

sudo apt update && sudo apt full-upgrade

In these steps you may be asked to confirm if you want to install something, just hit "y" and enter.

Now install the HAT firmware with:

sudo apt install hailo-all

This installation may take a good 5 to 10 minutes to complete. Once it has finished restart your Pi. If you want to be a power user you can restart it by typing into the terminal:

reboot

Now we will install Hailo's Python pipeline software and examples, but what is a pipeline?

Communicating with the AI HAT hardware itself is incredibly complicated and the code needed to do that is quite involved. We instead are going to set up and install an object detection pipeline which is just a collection of code and software to allow us to more-easily interact with the hat. It is essentially going to take our simpler and more human-readable code and turn all the gears behind the scenes to get the HAT to run it.

To install the pipeline and required libraries to run it, start by copying their GitHub repository by entering into the terminal:

git clone https://github.com/hailo-ai/hailo-rpi5-examples.git

This will download a folder called "hailo-rpi5-examples" into your Pi's home folder and it is going to be an important location that we will be working in.

Before we install the pipeline we need to tell the terminal to work out of that folder with the change directory command:

cd hailo-rpi5-examples

The blue text with the file location in the terminal shows that you have successfully run this command.Now we will run the shell script installer with:

./install.sh

This installation may take 10 - 20 minutes as it also installs all the YOLO models that we will be using.

Once the installation has finished, restart your Pi once more.

Let's run some demo code! In the previous step, we downloaded some sample pipelines from Hailo as well as a sample Python scripts to use these pipelines. We will be using the object detection pipeline in this tutorial - it's called "detection_pipeline.py" and is located under hailo_rpi5-examples/basic_pipelines.

The easiest way to run these Python scripts is through the terminal. First change the terminal's working location with the change directory command, this is the same as the previous one we used:

cd hailo-rpi5-examples

The installation step also created a virtual environment (also called a Venv). This is essentially an isolated virtual workspace we can use to install packages and experiment without the risk of affecting the rest of our Pi OS. All of the packages we need to use were installed in this Venv and we can tell the terminal to enter by typing in:

source setup_env.sh

You can confirm that you are working in a Venv as you will see the name of it on the left in brackets as seen in the image on the right. If you have this and the blue text from the change directory command, you are now ready to run the Python scripts. If you ever close the terminal or restart your Pi, you will need to run these commands again to return to this state.

We are going to run the Demo Python code called "detection.py" which is located in the "basic_pipelines" folder, so the command for this will be:

python basic_pipelines/detection.py

You should see a new window appear with a video of people crossing the road as well as the YOLO object detection model identifying objects in the frame as shown in the image on the right. Congratulations! You have successfully set up and run computer vision on your AI HAT.

The visual output of this we are seeing is straightforward. YOLO tries to draw a bounding box around where it thinks the object is, then labels it with what it identified it as and rates the confidence of that detection with a percentage. The default YOLO models are trained on the COCO dataset which only contains81 detectable objects. This may not seem like a lot, but many of these are broad categories of objects seen in day-to-day life like "sports ball" and "bottle".

To run the Python code with our camera as the input video source, we will need to specify it as an argument or option. We can get a list of all the available options of the object detection pipeline by entering:

python basic_pipelines/detection.py --help

There are a few helpful options here to explore, and you should sometime, but the one we are interested in is changing the source with the "--input" option. Here we can see that we can specify a file or camera as an input and that we can run the detection script with the camera module with:

python basic_pipelines/detection.py --input rpi

Now that we have our code running and detecting objects from our camera, let's take a quick look under the hood and see what's going on so that we can learn how to apply this object detection in our custom projects. There is a lot of complexity going on here and there are thousands of lines of code being run, but most of that is happening behind the scenes in the pipeline. This is fortunate for us as it means that we only have to interact with the "detection.py" file which is far more streamlined and human-readable (we call this high-level code).

Let's open up detection.py in Thonny and explore what's going on. The code starts by importing all the required packages and libraries to run, 2 of these are Python scripts in the same folder, "detection_pipeline" and "hailo_rpi_common". If you wish to modify a deeper behaviour in this setup that is not in detection.py, it will likely be somewhere in these files. If you wish to import any libraries or packages for your project, whack it up here as usual.

import gigi.require_version('Gst', '1.0')from gi.repository import Gst, GLibimport osimport numpy as npimport cv2import hailofrom hailo_rpi_common import ( get_caps_from_pad, get_numpy_from_buffer, app_callback_class,)from detection_pipeline import GStreamerDetectionApp

class user_app_callback_class(app_callback_class): def __init__(self): super().__init__() self.new_variable = 42 # New variable example def new_function(self): # New function example return "The meaning of life is: "

def app_callback(pad, info, user_data): # Get the GstBuffer from the probe info buffer = info.get_buffer() # Check if the buffer is valid if buffer is None: return Gst.PadProbeReturn.OK # Using the user_data to count the number of frames user_data.increment() string_to_print = f"Frame count: {user_data.get_count()}\n" # Get the caps from the pad format, width, height = get_caps_from_pad(pad) # If the user_data.use_frame is set to True, we can get the video frame from the buffer frame = None if user_data.use_frame and format is not None and width is not None and height is not None: # Get video frame frame = get_numpy_from_buffer(buffer, format, width, height) # Get the detections from the buffer roi = hailo.get_roi_from_buffer(buffer) detections = roi.get_objects_typed(hailo.HAILO_DETECTION)

 detection_count = 0 for detection in detections: label = detection.get_label() bbox = detection.get_bbox() confidence = detection.get_confidence() if label == "person": string_to_print += f"Detection: {label} {confidence:.2f}\n" detection_count += 1

 if user_data.use_frame: # Note: using imshow will not work here, as the callback function is not running in the main thread # Let's print the detection count to the frame cv2.putText(frame, f"Detections: {detection_count}", (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2) # Example of how to use the new_variable and new_function from the user_data # Let's print the new_variable and the result of the new_function to the frame cv2.putText(frame, f"{user_data.new_function()} {user_data.new_variable}", (10, 60), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2) # Convert the frame to BGR frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR) user_data.set_frame(frame) print(string_to_print) return Gst.PadProbeReturn.OK

import gigi.require_version('Gst', '1.0')from gi.repository import Gst, GLibimport osimport numpy as npimport cv2import hailofrom hailo_rpi_common import ( get_caps_from_pad, get_numpy_from_buffer, app_callback_class,)from detection_pipeline import GStreamerDetectionAppfrom gpiozero import AngularServo# Inheritance from the app_callback_classclass user_app_callback_class(app_callback_class): def __init__(self): super().__init__() # Initialize state variables for debouncing self.detection_counter = 0 # Count consecutive frames with detections self.no_detection_counter = 0 # Count consecutive frames without detections # State tracking, is it active or not? self.is_it_active = False self.servo = AngularServo(18, min_pulse_width=0.0006, max_pulse_width=0.0023) def app_callback(pad, info, user_data): # Get the GstBuffer from the probe info buffer = info.get_buffer() # Check if the buffer is valid if buffer is None: return Gst.PadProbeReturn.OK # Using the user_data to count the number of frames user_data.increment() # Get the caps from the pad format, width, height = get_caps_from_pad(pad) # If the user_data.use_frame is set to True, we can get the video frame from the buffer frame = None if user_data.use_frame and format is not None and width is not None and height is not None: frame = get_numpy_from_buffer(buffer, format, width, height) # Get the detections from the buffer roi = hailo.get_roi_from_buffer(buffer) detections = roi.get_objects_typed(hailo.HAILO_DETECTION) # Track if we've seen objects of interest this frame object_detected = False detection_string = "" # Parse the detections for detection in detections: label = detection.get_label() confidence = detection.get_confidence() # Check for objects of interest with confidence threshold if confidence > 0.4: # Adjust confidence threshold as needed if label == "person": object_detected = True detection_string += f"Detection: {label} {confidence:.2f}\n" # Debouncing logic if object_detected: user_data.detection_counter += 1 user_data.no_detection_counter = 0 # Only activate after given amount of consecutive frames with detections if user_data.detection_counter >= 4 and not user_data.is_it_active: # Move the Servo or do what ever you want to do user_data.servo.angle = 90 # Update the is it active variable so this doesnt keep repeating user_data.is_it_active = True print("OBJECT DETECTED!") else: user_data.no_detection_counter += 1 user_data.detection_counter = 0 # Only deactivate after 5 consecutive frames without detections if user_data.no_detection_counter >= 5 and user_data.is_it_active: # Move the Servo or do what ever you want to do user_data.servo.angle = 0 user_data.is_it_active = False print("Object Gone.") # Print detections if any if detection_string: print(detection_string, end='') return Gst.PadProbeReturn.OKif __name__ == "__main__": # Create an instance of the user app callback class user_data = user_app_callback_class() app = GStreamerDetectionApp(app_callback, user_data) app.run()

To run this code, create a new Python file in the same basic_pipelines folder. For this example, we called it "watcher.py". To run the code, it is the same command as before but we must use the new name:

python basic_pipelines/watcher.py --input rpi

This code is currently set up to move a servo if a person is detected and an example of it in action is to solve a real problem in the office. At my desk, I often have my headphones on which makes it easy for people to startle me when they enter the office from behind me. When this code detects a person in the camera pointed at the door behind me, it rotates a servo creating a visual alert that someone has entered!

Let's take a look at some of the essential sections of the code to see what we have added and how you can purpose it for your needs.

from gpiozero import AngularServo

class user_app_callback_class(app_callback_class): def __init__(self): super().__init__() # Initialize state variables for debouncing self.detection_counter = 0 # Count consecutive frames with detections self.no_detection_counter = 0 # Count consecutive frames without detections # State tracking, is it active or not? self.is_it_active = False self.servo = AngularServo(18, min_pulse_width=0.0006, max_pulse_width=0.0023)

 # Track if we've seen objects of interest this frame object_detected = False detection_string = "" # Parse the detections for detection in detections: label = detection.get_label() confidence = detection.get_confidence() # Check for objects of interest with confidence threshold if confidence > 0.4: # Adjust confidence threshold as needed if label == "person": object_detected = True detection_string += f"Detection: {label} {confidence:.2f}\n"

 # Parse the detections for detection in detections: label = detection.get_label() confidence = detection.get_confidence() # Check for objects of interest with confidence threshold if confidence > 0.4: # Adjust confidence threshold as needed if label == "person": object_detected = True detection_string += f"Detection: {label} {confidence:.2f}\n"

This next section is what truly makes this code robust. If we have detected a person, we increase the detection_counter by 1 and set the no_detection_counter back to 0 as this counter is instead counting how many frames we haven't seen a person. Very importantly observe that when we use detection_counter (which was declared in the class before) we have to have the "user_data." prefix. So when we create a variable above in the class section we use "self." - and when we want to use it here in app_callback() we must have "user_data.".

In the next line, we check if detection_counter is above 4 and if it is, we move the servo (again noting that when we move the servo we have to still use "user_data."). Here you can put your custom code to "do something" if your object is detected. You could send an email, turn on a light, spin a motor, whatever you want!

But why do we count 4 frames in a row before we do something? Well, this is because we are adding something similar to debounce on a button. Let's say this code didn't have this debounce and we used the code to watch outside a house to automatically open the dog door when a dog is detected. Let's say the system works great, but one day a cat approaches the door. Yolo will quite reliably identify it as a cat - but computer vision isn't perfect. For 1 frame it might identify it as a dog by mistake which would open up the dog door when we don't want it to. Most of the time this isn't an issue, but we have found in our testing that there are some outlier cases where this can be quite a bit of an issue.

This debounce code helps remove this issue by ensuring that YOLO detects the target object 4 times in a row before it "does something" and you can change this number to fit the needs of your project.

 # Debouncing logic if object_detected: user_data.detection_counter += 1 user_data.no_detection_counter = 0 # Only activate after given amount of consecutive frames with detections if user_data.detection_counter >= 4 and not user_data.is_it_active: # Move the Servo or do what ever you want to do user_data.servo.angle = 90 # Update the is it active variable so this doesnt keep repeating user_data.is_it_active = True print("OBJECT DETECTED!")

We also have the inverse of all of this. In this else branch (if we don't detect a human), we set detection_counter to 0 and increase no_detection_counter by 1. We then check if no_detection_counter is greater than 5 and if it is, we will move the servo back. Here is where you can put your custom code to "do something else" when the target object isn't detected.

This code needs 5 frames where the object isn't detected before it "does something" and we do it for similar reasons as before. Let's say a dog is walking up to the doggie door and as it is walking through YOLO detects it as a horse for a single frame. This would close the door on the poor dog. So we fix this by ensuring we have 5 frames in a row without the target object so that we can be sure it has disappeared.

It is worth noting that this debounce does make the code less responsive as 4 or 5 frames must pass before we react to an object in frame. The code will most likely be running at 30fps so that is about 1/6 of a second delay. You can set these values to 1 if you want to remove this feature.

There is also one more thing in this section of the code to explore. On the end of these debounce checks we also check the is_it_active variable we made:

 if user_data.detection_counter >= 4 and not user_data.is_it_active:

import gigi.require_version('Gst', '1.0')from gi.repository import Gst, GLibimport osimport numpy as npimport cv2import hailofrom hailo_rpi_common import ( get_caps_from_pad, get_numpy_from_buffer, app_callback_class,)from detection_pipeline import GStreamerDetectionAppfrom gpiozero import LEDclass user_app_callback_class(app_callback_class): def __init__(self): super().__init__() # Configuration self.target_object = "cup" # Object type to detect # Debouncing variables self.detection_counter = 0 # Consecutive frames with exact match self.no_detection_counter = 0 # Consecutive frames without match # State tracking, is it active or not? self.is_it_active = False self.green_led = LED(18) self.red_led = LED(14) self.red_led.off() self.green_led.on()def app_callback(pad, info, user_data): # Get the GstBuffer from the probe info buffer = info.get_buffer() # Check if the buffer is valid if buffer is None: return Gst.PadProbeReturn.OK # Using the user_data to count the number of frames user_data.increment() # Get the caps from the pad format, width, height = get_caps_from_pad(pad) # If the user_data.use_frame is set to True, we can get the video frame from the buffer frame = None if user_data.use_frame and format is not None and width is not None and height is not None: frame = get_numpy_from_buffer(buffer, format, width, height) # Get the detections from the buffer roi = hailo.get_roi_from_buffer(buffer) detections = roi.get_objects_typed(hailo.HAILO_DETECTION) # Count objects in this frame object_count = 0 detection_string = "" # Parse the detections for detection in detections: label = detection.get_label() confidence = detection.get_confidence() # Check for target objects with confidence threshold if confidence > 0.4: if label == user_data.target_object: object_count += 1 detection_string += f"{label.capitalize()} detected! Confidence: {confidence:.2f}\n" # Debouncing logic for number of items if object_count >= 3: user_data.detection_counter += 1 user_data.no_detection_counter = 0 # Only activate after sufficient consistent frames if user_data.detection_counter >= 4 and not user_data.is_it_active: # Turn on red led, or do what ever else you want to do user_data.red_led.on() user_data.green_led.off() user_data.is_it_active = True print(f"NUMBER OF OBJECTS DETECTED!") else: user_data.no_detection_counter += 1 user_data.detection_counter = 0 # Only deactivate after sufficient non-matching frames if user_data.no_detection_counter >= 5 and user_data.is_it_active: # Turn on green LED or what ever else you wish to do user_data.red_led.off() user_data.green_led.on() user_data.is_it_active = False print(f"No longer detecting number of objects.") # Print detections if any if detection_string: print(f"Current {user_data.target_object} count: {object_count}") print(detection_string, end='') return Gst.PadProbeReturn.OKif __name__ == "__main__": # Create an instance of the user app callback class user_data = user_app_callback_class() app = GStreamerDetectionApp(app_callback, user_data) app.run()

from gpiozero import LED

class user_app_callback_class(app_callback_class): def __init__(self): super().__init__() # Configuration self.target_object = "cup" # Object type to detect # Debouncing variables self.detection_counter = 0 # Consecutive frames with exact match self.no_detection_counter = 0 # Consecutive frames without match # State tracking, is it active or not? self.is_it_active = False self.green_led = LED(18) self.red_led = LED(14) self.red_led.off() self.green_led.on()

 # Check for target objects with confidence threshold if confidence > 0.4: if label == user_data.target_object: object_count += 1 detection_string += f"{label.capitalize()} detected! Confidence: {confidence:.2f}\n"

 # Debouncing logic for number of items if object_count >= 3: user_data.detection_counter += 1 user_data.no_detection_counter = 0 # Only activate after sufficient consistent frames if user_data.detection_counter >= 4 and not user_data.is_it_active: # Turn on red led, or do what ever else you want to do user_data.red_led.on() user_data.green_led.off() user_data.is_it_active = True print(f"NUMBER OF OBJECTS DETECTED!") else: user_data.no_detection_counter += 1 user_data.detection_counter = 0 # Only deactivate after sufficient non-matching frames if user_data.no_detection_counter >= 5 and user_data.is_it_active: # Turn on green LED or what ever else you wish to do user_data.red_led.off() user_data.green_led.on() user_data.is_it_active = False print(f"No longer detecting number of objects.")

import gigi.require_version('Gst', '1.0')from gi.repository import Gst, GLibimport osimport numpy as npimport cv2import hailofrom hailo_rpi_common import ( get_caps_from_pad, get_numpy_from_buffer, app_callback_class,)from detection_pipeline import GStreamerDetectionAppfrom gpiozero import LEDclass user_app_callback_class(app_callback_class): def __init__(self): super().__init__() # Configuration self.target_object = "person" # Object type to detect # Target zone configuration (normalized coordinates 0-1) self.zone_x_min = 0.4 # Left boundary of target zone self.zone_x_max = 0.6 # Right boundary of target zone self.zone_y_min = 0.3 # Top boundary of target zone self.zone_y_max = 0.7 # Bottom boundary of target zone # Debouncing variables self.in_zone_frames = 0 # Consecutive frames with object in zone self.out_zone_frames = 0 # Consecutive frames without object in zone # State tracking self.is_it_active = False self.green_led = LED(18) self.red_led = LED(14) self.red_led.off() self.green_led.on() def app_callback(pad, info, user_data): # Get the GstBuffer from the probe info buffer = info.get_buffer() if buffer is None: return Gst.PadProbeReturn.OK user_data.increment() # Get the caps from the pad format, width, height = get_caps_from_pad(pad) frame = None if user_data.use_frame and format is not None and width is not None and height is not None: frame = get_numpy_from_buffer(buffer, format, width, height) # Get the detections from the buffer roi = hailo.get_roi_from_buffer(buffer) detections = roi.get_objects_typed(hailo.HAILO_DETECTION) object_in_zone = False detection_string = "" # Parse the detections for detection in detections: label = detection.get_label() confidence = detection.get_confidence() if confidence > 0.4 and label == user_data.target_object: # Get bounding box coordinates bbox = detection.get_bbox() # Call the coordinate methods x_min = bbox.xmin() y_min = bbox.ymin() box_width = bbox.width() box_height = bbox.height() # Calculate max coordinates x_max = x_min + box_width y_max = y_min + box_height # Calculate center point (these are normalized 0-1) center_x = x_min + (box_width / 2) center_y = (y_min + (box_height / 2) - 0.22) * 1.83 # Debug print for coordinates detection_string += (f"{label.capitalize()} detected!\n" f"Position: center=({center_x:.2f}, {center_y:.2f})\n" f"Bounds: xmin={x_min:.2f}, ymin={y_min:.2f}, xmax={x_max:.2f}, ymax={y_max:.2f}\n" f"Confidence: {confidence:.2f}\n") # Check if object's center is in the target zone if (user_data.zone_x_min <= center_x <= user_data.zone_x_max and (user_data.zone_y_min - 0.22) * 1.83 <= center_y <= (user_data.zone_y_max - 0.22) * 1.83): object_in_zone = True detection_string += f"Object is in target zone!\n" # Debouncing logic for zone detection if object_in_zone: user_data.in_zone_frames += 1 user_data.out_zone_frames = 0 if user_data.in_zone_frames >= 4 and not user_data.is_it_active: # Turn on red led, or do what ever else you want to do user_data.red_led.on() user_data.green_led.off() user_data.is_it_active = True print(f"{user_data.target_object.capitalize()} detected in target zone - Servo activated!") else: user_data.out_zone_frames += 1 user_data.in_zone_frames = 0 if user_data.out_zone_frames >= 5 and user_data.is_it_active: user_data.red_led.off() user_data.green_led.on() user_data.is_it_active = False print(f"No {user_data.target_object} in target zone - Servo deactivated!") # Print detections if any if detection_string: print(detection_string, end='') return Gst.PadProbeReturn.OKif __name__ == "__main__": user_data = user_app_callback_class() app = GStreamerDetectionApp(app_callback, user_data) app.run()

We start with the same class setup section as before but with some additional variables to draw a box on the image with minimum and maximum x and y values. If the object is inside of this box we will "do something". The box is defined by decimals with 0 on the x-axis being the left side of the image, 1.0 being the right side of the image, and 0.5 being the middle. The y-axis spans from 0 being the top and 1.0 being the bottom. Below is an example of two boxes that can be drawn and you can change these x and y min and max variables to set your custom detection boxes.

The detection_counter variables have also been changed to in_zone_frames and out_zone_frames. This is just a name change and are used exactly the same.

class user_app_callback_class(app_callback_class): def __init__(self): super().__init__() # Configuration self.target_object = "person" # Object type to detect # Target zone configuration (normalized coordinates 0-1) self.zone_x_min = 0.4 # Left boundary of target zone self.zone_x_max = 0.6 # Right boundary of target zone self.zone_y_min = 0.3 # Top boundary of target zone self.zone_y_max = 0.7 # Bottom boundary of target zone # Debouncing variables self.in_zone_frames = 0 # Consecutive frames with object in zone self.out_zone_frames = 0 # Consecutive frames without object in zone # State tracking self.is_it_active = False self.green_led = LED(18) self.red_led = LED(14) self.red_led.off() self.green_led.on()

 if confidence > 0.4 and label == user_data.target_object: # Get bounding box coordinates bbox = detection.get_bbox() # Call the coordinate methods x_min = bbox.xmin() y_min = bbox.ymin() box_width = bbox.width() box_height = bbox.height() # Calculate max coordinates x_max = x_min + box_width y_max = y_min + box_height # Calculate center point (these are normalized 0-1) center_x = x_min + (box_width / 2) center_y = (y_min + (box_height / 2) - 0.22) * 1.83 # Debug print for coordinates detection_string += (f"{label.capitalize()} detected!\n" f"Position: center=({center_x:.2f}, {center_y:.2f})\n" f"Bounds: xmin={x_min:.2f}, ymin={y_min:.2f}, xmax={x_max:.2f}, ymax={y_max:.2f}\n" f"Confidence: {confidence:.2f}\n") # Check if object's center is in the target zone if (user_data.zone_x_min <= center_x <= user_data.zone_x_max and (user_data.zone_y_min - 0.22) * 1.83 <= center_y <= (user_data.zone_y_max - 0.22) * 1.83): object_in_zone = True detection_string += f"Object is in target zone!\n"