Or to rephrase, “Why do pedestrians need better protection at night?”
This has become such an easy question to answer that even the mainstream press is writing about it. Last December, The New York Times provided stark answers with its article Why Are So Many American Pedestrians Dying at Night? Simply put, data from the Federal Fatality Analysis Reporting System shows that as cars have gotten bigger and drivers have become more distracted over the last 10 years, pedestrian fatalities have greatly climbed, with over 75% of these deaths occurring at night (Figure 1).
Figure 1. NHTSA fatality statistics show how dangerous night-time is for pedestrians. Total deaths were over 7,300 in 2021 and are still climbing. Source: NHTSA
Just days earlier, a Forbes article drew a similar conclusion, but also hinted at a potential solution: Drivers Tend to Kill Pedestrians at Night. Thermal Imaging May Help.
The National Highway Traffic Administration (NHTSA), prompted by insurers and the public, recently concluded that the time has come to address this growing tragedy, proposing a rule that would require light vehicles starting with 2029 models to be able to see pedestrians at all hours and automatically stop to avoid collisions. In this proposed rule, NHTSA acknowledged that technological improvements would be necessary to achieve this goal, presaging the Forbes suggestion that thermal imaging might be the key.
Meeting safety needs with thermal imaging
Thermal imaging can be the right technology at the right time, only when the cost constraints in the vehicle markets and the computing demands necessary to find pedestrians quickly and accurately are both met in real vehicle systems. Fortunately, rapid progress is now being made toward satisfying both requirements.
Most cameras detect light we can see, either emitted, as from traffic lights, or more likely, reflected from various sources by the objects being viewed. The sources are necessary. Without them, there is nothing to detect: the cameras go black. Thermal images, in contrast, are generated by detecting the infrared radiation continuously emitted by all objects above absolute zero. Around 300K, human body temperature, this emission peaks in the 8 to 14 µm band. Thermal cameras can detect this infrared energy to produce images of warm objects without any external illumination.
While thermal imaging has existed for almost 100 years, only recently have methods been developed to produce high-quality images without the need for expensive cryogenic cooling of the detectors. Even so, the uncooled detectors, using a technology designated the microbolometer array, required substantial image processing and periodic recalibration to provide useful images. The resulting image anomalies, bulk and cost have rendered these unsuitable for automotive use. Further, no microbolometer arrays, outside of a few very expensive defense devices, had been built offering sufficient resolution to provide the wide fields of coverage required in automotive safety systems (Figure 2).
Figure 2. Effective automotive safety systems require monitoring a scene as wide as that observed by a driver with special emphasis on extended distances along the road ahead. Source: Owl Autonomous Imaging
Together, the requirements for a thermal sensor suitable for the pedestrian automatic emergency braking (PAEB) function in advanced driver assistance systems (ADAS) that NHTSA has mandated raise formidable challenges, such as:
- Sufficient resolution to view the entire scene with one camera
- Sufficient thermal sensitivity to see pedestrians in all environments
- Elimination of the need for an external calibration shutter
- Incorporation of calibration capabilities inside the sensor
- Straightforward interfacing with automotive systems
- Compliance with automotive safety standards
- Substantial reduction in cost
Analyzing the details of these requirements can produce a complete description of the performance necessary for deployment of effective PAEB systems.
Deploying thermal imaging for ADAS
When thermal imaging systems first became available, defense applications were the primary use, so the Army Night Vision Laboratory conducted studies to determine what performance was required to permit system operators to figure out what they were seeing in the thermal images. From these studies, three levels of perception were defined (Figure 3).
Figure 3. Perceiving objects in a scene requires enough resolution to elucidate defining features. Source: Owl Autonomous Imaging
While these three levels are useful for human observers, ADAS operations rely on computer image processing, which was not a consideration when the studies were done. To make effective decisions, a computer needs to “classify” objects, determining, for instance, if a pedestrian is an adult or a child, but not whether the person is in a uniform or a swimsuit. For reliable classification, resolution between the “recognition” and “identification” levels is needed, nominally 10 pixels per meter (ppm). This leads directly to the need for about 1,200 pixels across the automotive front field of view at the required range.
Sufficient thermal sensitivity to see pedestrians is now commonplace in microbolometer sensors, which provide contrast between objects differing in temperature by as little as 1/20th° C (50 milliKelvins (mK)) or less. Sustaining this sensitivity requires, with most microbolometer sensors, periodic thermal recalibration to compensate for changes in temperature of the sensor itself. Typically, this recalibration requires closing a shutter to provide a uniform thermal calibration image and correction of the data after acquisition for measured non-uniformities.
Shuttering in automotive applications is unacceptable because it interrupts the flow of images, producing momentary system blindness and uncertainty during the recovery when the images return while correction after acquisition reduces the system dynamic range, hampering the capability to reliably classify pedestrians in all environments. Thermal cameras intended for ADAS use must avoid both operations.
For incorporation into an automotive sensor suite a thermal camera must comply with the same requirements as other automotive electronics covering safety, interoperability and reliability. PAEB systems impose additional requirements related to the intended purpose of the thermal camera.
PAEB systems that can operate in total darkness are what the NHTSA is mandating in its new rule proposal. These must detect pedestrians who could be struck by the vehicle and stop the vehicle when such a collision seems likely. This decision requires consideration, for instance, of the trajectories of both the vehicle and the pedestrian, the likelihood that pedestrians will be in the vehicle path at the calculated time of collision, and the area of the vehicle that would be impacted. These considerations of physics and behavior are currently the subject of concentrated research at all of the vehicle companies because the results must be reducible to computer applications to be run on the electronic control unit (ECU).
To make these decisions, the ECU needs accurate, timely raw data describing what the camera sees. Fundamentally, this data must include a label classifying a group of pixels as a pedestrian, and the location in 3D space of the pedestrian relative to the vehicle. Figure 4 shows what is needed.
Figure 4. For a PAEB system to make decisions, the raw thermal image data (lower right) must be segmented to classify and label pedestrians (upper left) and to determine their locations relative to the vehicle (center).
Pedestrians come in a variety of shapes and sizes and assume many poses while moving on streets. As a result, using predetermined rules to find them in an image is destined to fail. Instead, the power of artificial intelligence (AI) is applied. Specifically, structures called convolutional neural networks (CNNs), configured for image processing and already in use in automobiles, can be trained to provide the information a PAEB system needs to make decisions.
One of the goals in using AI with thermal cameras is to eliminate the need for installing stereo pairs. While this technique already determines 3D location of objects using visible-band cameras, it does require at least two carefully mounted cameras, doubling the cost. To avoid this burden, a CNN can be trained to determine distance using a monocular training set. This technique has often been demonstrated using visible images but, with the proper thermal image training set, it is just as useful in the infrared. No additional camera hardware is required; it simply measures disparity, offsets in the image resulting from the characteristics of the lens and sensor, which can be calibrated to indicate true distance.
Finding objects and classifying them is also widely used, generally involving a CNN and a technique called YOLO that can make the necessary determinations from a single frame to produce objects with bounding boxes and labels containing the object classes. Since the distance and object information are developed from the same image, no post-alignment is required, so distance data can instantly be added to the object labels. In vehicles now being designed, the ECUs will have sufficient power to run the CNN software themselves, minimizing the bulk and cost to be added when thermal capabilities are deployed.
Summary
Format the data appropriately and provide the right interface, and information from thermal imaging becomes readily available to automotive systems. This means that within about five years, the technical and cost barriers will be gone, allowing automobile manufacturers to design and build cars that will see pedestrians at night and to greatly reduce nighttime traffic accidents, saving hundreds of thousands, perhaps millions of lives.
About Owl Autonomous Imaging
After several years of development, Owl is working with automotive manufacturers and their Tier 1 suppliers to provide cost-effective thermal imaging modules and software to convert their images into data suitable for new systems destined to protect pedestrians at all hours of the day or night. The combination of the thermal camera and a neural processor constitutes the Thermal Ranger system. developed by Owl Autonomous Imaging. For demonstrations, Owl offers the entire system packaged with additional software and a series of useful interfaces intended to allow potential users to add thermal imaging capabilities to existing vehicles.
About the author
Appelman is the chief business officer for Owl Autonomous Imaging. Previously, he helped launch to market several companies including SensL, which was a leader in single photon detection used in lidar and medical imaging. He also launched an agricultural robotics company, Harvest Automation an early user of LiDAR technology. He has extensive experience holding leadership roles in both private and public sector organizations. He holds a bachelor’s degree from Wagner College in New York City.