Abstract—A night-vision system has been developed to help reduce vehicle-pedestrian accidents occurring at night. High-temperature objects assumed to be pedestrians are detected by processing the images from infrared stereo cameras mounted on the vehicle, and the possibility of a collision is judged by calculating the position and relative moving vectors of the pedestrian. In addition, voice guidance is provided and a highlighted infrared image of the pedestrian is displayed at the bottom of the front windshield using a head-up display (HUD). It was determined that the system could judge the possibility of collisions with pedestrians on the road or crossing the road.
Index Terms—Head-up display, infrared stereo cameras, judgment of collision, pedestrian/night vision.
I. INTRODUCTION
Approximately 10,000 fatal traffic accidents occur in Japan annually, of which some 30% are collision accidents with pedestrians, who are the vulnerable parties in traffic accidents [1].
This paper reports on our examination of systems for providing drivers with advance notice about the presence of pedestrians for preventive safety purposes.
In order to examine the system functions and specifications, we first conducted a detailed investigation of pedestrian accidents (fatalities). Fig. 1 shows the results of our investigation.
1) The majority of accidents involving collisions with pedestrians walking along the road or crossing the road take place when vehicles are traveling straight on straight roads at night [Fig.1 (a)–(c)].
2) The majority of these accidents involve collisions by vehicles cruising at speeds of 40 km/h–80 km/h [Fig.1 (d)].
As a result, we developed a night-vision system that would detect pedestrians walking along the road or crossing the road that are difficult to see from vehicles cruising at night, (see Fig. 2), and would notify the driver by providing voice guidance and displaying infrared camera images.
We also verified the effectiveness of this system by means of road crossing pedestrian detection tests and obstacle avoiding tests.
II. SYSTEM CONCEPT
A. System Functions and Configuration
The following target requirements were established for the purpose of deciding system functions.
1) The information provided will be able to call the driver’s attention to obstacles and will allow judgment of forward conditions.
2) The system will be capable of detecting targets in size ranging from a pedestrian’s head to the full-body (20 cm * 20 cm to 2 m * 2 m) while taking into consideration hiding of targets by oncoming vehicles and other such factors on ordinary roads.
3) The system will allow a time margin (information provision time) of 3.5 s or more after information provision and before collision [2], and will allow the detection range needed for this.
It was decided to satisfy requirement 1 by using voice guidance and head-up display (HUD) of images of nighttime forward conditions from an infrared camera.
It was decided that detection capabilities satisfying requirements 2 and 3 could be obtained by using infrared stereo cameras, calculating the distance to high temperature objects that can be perceived as pedestrians, and thus judging the possibility of a collision [3]–[6].
B. Conditions for Information Provision Operation
The target area for information provision by the system is the region that is difficult to see with headlights and so is subject to the visual enhancement system. This was defined as the region extending outward from the high intensity of illumination area of the headlights (which extends up to 30 m forward).
Based on result 2 from the investigation of pedestrian accidents, it was decided that information provision would operate at cruising speeds of 40 km/h and above. The objective was to satisfy the requirement of 3.5 s or more of information provision time up to the cruising speed of 80 km/h (this is, in other words, the 30–80 m detection area shown in Fig. 3).
C. Detection and Judgment Conditions
The two detection area judgment conditions shown in Fig. 3 were established in order to assure detection of pedestrians subject to likely collision.
1) Detection Area and Judgment of Road Side Pedestrian:
The system established an area which the body width of the vehicle with an additional margin of 1 m to either side for detecting pedestrians walking along the road (toward and away) at a rate of 5 km/h or less within this area.
When the subject vehicle is turning, the system uses the yaw rate and vehicle speed to estimate the predicted course of the vehicle, and changes the shape of the detection area accordingly.
2) Detection Area and Judgment of Crossing Pedestrian:
The condition for a collision between the subject vehicle and an object crossing the road is that, when seen from the subject vehicle, the object azimuth ?(in degrees) is constant [7]. The azimuth?is determined by the speed of one’s own vehicle Vj and the crossing speed Vh of the object according to the following equation:
Assuming that the pedestrian is crossing at the rate of 1 m/s and that the minimum operating speed for carrying out information provision is 40 km/h, then the crossing objects will be the total imaging range of the infrared cameras across an angle of 11 or more.
Objects with a greater crossing speed will only appear for a short time as images in the infrared camera’s angular field of view. Although they can be detected, the time for information provision will be reduced.
III. DETECTION PROCESSING ALGORITHMS
A. Definition of Coordinate System
This system uses two coordinate systems, the three-dimensional (3-D) coordinate system related to the vehicle body as shown in Fig. 4, and the image plane coordinate system.
The origin ? of the 3-D coordinate system is the lens center of the camera on the right. Right-hand rotation of the axis is treated as positive. The image plane coordinate system is the plane in the 3-D coordinate system where Z=f (where is the camera focal length), and the origin is the point of intersection with the Z axis.
B. Pedestrian Isolation and Tracking over Time
Measurement of the difference between atmosphere temperature and pedestrian skin surface temperature in all seasons showed that a temperature difference of about 5 C occurs at atmosphere temperatures up to about 30 C. The measurement results are shown in Fig. 5.
Consequently, high temperature objects such as pedestrians appear in the infrared image with higher brightness than background objects.
We confirmed the possibility of segmenting pedestrian in images by cruising on ordinary roads in every season, collecting image data, and performing thresholding of it.
The system on an actual vehicle generates a histogram of the brightness of the right-hand image from the infrared stereo cameras and sets the intermediate value in the brightness distribution as the threshold value. The pedestrian in the right-hand image is segmented in this way.
Next, the equality of the object extracted from the right-hand image by segmentation is judged at each point in time, and tracked over time.
The judgment of equality is objects with the shortest distance of X direction and varying rate of area under a fixed rate, on the condition that the center of gravity of each object does not cross each other at each point in time.
C. Calculating the 3-D Positional Coordinates of the Object
The left and right images from the infrared stereo cameras display a difference of lateral position (disparity) of dn (in pixels) according to the distance to the object [8]. The area around the object in the right-hand image is extracted and used to perform a correlation calculation with the left-hand image. In this way, the disparity dn is calculated. Next, (2) is used with that calculated disparity to derive the distance to the object Zi (in meters).
The center of gravity position q=(xi,yi)T in the image and the calculated value Zi (in meters) are then used in (3) and (4) to calculate the 3-D position of the object P=(xi,yi,zi)T.
D is the distance (in meters) between cameras, while f and are the focal length (in meters) and p pixel pitch (in m/pixel) of the cameras.
The calculated position of the object along the X axis can be changed significantly by slight changes in the turning angle of the vehicle while it is cruising. Therefore, it becomes difficult for the system when operating from a cruising vehicle to identify the difference in the 3-D position of a crossing pedestrian along the X axis.
Consequently, as shown in Fig. 6, the system uses the vehicle speed and yaw rate sensors to calculate the distance traveled (T) and turning angle (?) of the subject vehicle.
Equations (5) and (6) are then used to compensate for the difference.
Fig. 7 shows the group of object position P[t]=(Xc[t],Yc[t]Zc[t])T during the period from time t=0 (the past) to t=n (the present) with compensation for the turning angle of the subject vehicle using (5) and (6).
From the collision data shown in Fig. 1, pedestrian accidents do not take place when the vehicle’s behavior is unstable. Therefore, principal component analysis (PCA) is adopted to calculate the relative moving vector.
The straight-line directional vector L=(Lx,Ly,Lz)T is obtained by performing principal component analysis [9] on the center of gravity position coordinate Q=(X,Y,Z)T and the matrix of cross-correlation coefficient for the group of object position (P[t]=,t=0,1,….,n) shown in (7). In this way, the relative moving vector for the object is determined as the straight line shown in (8).
The relative velocity (Vs) is calculated from the changes occurring over time in the object position.
E. Condition of Collision Judgment:
1) satisfies the conditions in (9) for the calculated relative velocity (Vs) and the driving velocity V calculated from the speed sensor;
2) Satisfies the detection judgment in Section II-C;
3) An object that is up to 2 m in height above the road plane;
4) An object in size ranging from a pedestrian’s head to the full-body (20 cm * 20 cm to 2 m * 2 m).
F. Processing Flow
In order to realize real-time processing, the system separates processes 1 and 2. These are carried out on independent operation cycles, with process 1 occurring at an National Television System Committee (NTSC) frame rate of 33 ms, and process 2 each 100 ms.
1) Process 1: Using only the right-hand image from the infrared stereo cameras, this process performs segmentation, tracking, and image display of the object.
2) Process 2: This process calculates the 3-D position of the object from the stereo images, estimates the relative vector, and judges the possibility of a collision.
The pedestrian detection processing flow is shown in Fig. 8.
IV. HARDWARE
Fig. 9 shows the layout of the system on an actual vehicle[10].
A. Infrared Stereo Cameras
The infrared stereo cameras system is made up of two 320- pixel * 240-pixel uncooled infrared cameras that have a horizontal field of view of 12 and a vertical field of view of 9. The distance between the cameras is 360 mm.
The characteristics of the infrared stereo cameras must meet the following four conditions in order to measure the distance to the object correctly.
1) The two cameras must have synchronous detection.
2) The two cameras must have equivalent image brightness characteristics.
B. Yaw Rate Sensor
B. Sensor for spatial purpose
The system uses an optical fiber gyroscope with a dynamic range of±50°/s and a minimum resolution of about 0.03 /s to calculate the turning angle of the vehicle in order to estimate the relative moving vector and change the shape of the detection area.
C. Image Processing Electronic Control Unit
The left and right images from the infrared stereo cameras are output as NTSC signals. These are merged in a 640-pixel * 480-pixel image split horizontally into two (left and right camera images in 640-pixel * 240-pixel format) from which 8-b gray-scale images are sampled at a video rate (30 frames/s).
Speed, yaw rate, and other such vehicle information are also read on a 33-ms cycle, along with the images.
D. HUD
The system displays an 11°X 4(horizontal*vertical) field of view image in order to be capable of displaying the detection range for pedestrians crossing the road and a full-body view of a pedestrian 30 m to the ahead.
The HUD visible to the driver has a 10°X 3.5°(horizontal*vertical) display. This size was selected because it is sufficient to allow natural-feeling perception of a pedestrian’s direction and distance when a pedestrian ahead of the vehicle is displayed as described above.
The display is located in the lower front of the driver, so that the angle when looking down is 5. The focus point of the image is approximately 2 m forward, near the front edge of the hood.
The brightness of the visual image is 0–50 cd, and the color of the detected pedestrian’s highlighted image is an easily-recognizable red (see Fig. 10).
V. TEST RESULTS
A. Object Detection Results
Two verification tests were carried out to determine whether the prototype system satisfied the detection performance parameters as they had been set.
Two verification tests were carried out to determine whether the prototype system satisfied the detection performance parameters as they had been set by setting up the actual traffic conditions on the test course.
1) Relative Velocity Calculation Test: Fig. 11 shows the results of relative velocity calculations conducted when driving at 60 km/h toward a stationary object (the horizontal axis shows the distance to the object, and the vertical axis shows the relative velocity error).
Moving at a speed of 60 km/h, the forward distance to the object at 3.5 s before collision is approximately 60 m. Object to be detected should have a relative velocity within 30 km/h, as determined by (9).
The relative velocity calculation for a stationary object at a distance of 60 m has accuracy within 25 km/h, as shown in Fig. 11. Therefore, a pedestrian walking at a velocity within ±5 km/h will have estimated relative velocity calculation accuracy within 30 km/h. This will satisfy the detection condition of a relative velocity within 30 km/h.
Similar tests were also conducted at cruising speeds of 40–80 km/h, and these confirmed that the system could detect a pedestrian 3.5 s before collision.
2) Relative Moving Vector Calculation Test: Fig. 12 presents the results from calculation of the relative moving vectors of a pedestrian crossing the road at approximately 1 m/s and a stationary object (a streetlight) approximately 60 m ahead of a vehicle cruising at 50 km/h.
From calculation of the relative moving vectors, the crossing velocity of the pedestrian was 1.02 m/s, and the velocity of the streetlight was 0.08 m/s. This shows that the lateral moving vectors are being calculated with accuracy within ±0.1m/s.
The calculation of the relative moving vectors at a point 3.5 s before collision and approximately 50m away yielded the result of 40 km/h. This satisfies the conditions of (9), and shows that the system is capable of detection and judgment of pedestrians crossing the road.
There was no erroneous detection or non detection from the 100 times of repeated tests at the test course.
B. Evaluation of Information Provision Methods
Nighttime driving conditions that make obstacles difficult to see were reproduced on the test course, and multiple techniques for providing information about obstacles were implemented.
Comparisons were then done of the collision avoiding timing for each technique.
1) Comparison of information provision methods.
Voice guidance only and voice plus highlighted infrared image.
2) Comparison of image display positions.
LCD on instrument panel and HUD (see Fig. 13).
1) Test Method: Fog lamps were used as the only illumination to limit the nighttime range of vision to 20–30 m, and the test subjects were instructed to cruise at a constant speed of 40 km/h. Information was then provided on the obstacles. Obstacles were placed randomly at two locations, one directly in front of the vehicle and one on the centerline, in order to avoid a simple reaction when information was provided. Information provision was implemented at a distance of 40 m (3.6 s before collision).
2) Test Results: Collision avoidance actions varied with the test subjects. Therefore, the faster of steering avoidance and braking avoidance was selected as the timing for the start of collision avoidance action (response time). Fig. 14 shows the comparing results for different information provision methods and different display position.
The results from the two tests show that collision avoidance could be initiated about 1 s in advance with voice plus HUD. With voice only and with LCD display, however, the timing for the start of collision avoidance action was 1 s or more in many cases. This indicates that voice plus HUD providing a highlighted infrared image is more effective with respect to collision avoidance by the driver.
VI. CONCLUSION
Based on an analysis of nighttime pedestrian accidents, we created a system that uses infrared stereo cameras, an image processor, and HUD to notify drivers who are driving at night to the presence of pedestrians walking along the road and crossing the road. This project yielded the following two results.
1) We created an algorithm to judge the possibility of collision with high temperature obstacles that can be perceived as pedestrians by means of infrared stereo cameras and image processing. We determined that this enables judgment of the possibility of a collision 3.5 s in advance at cruising speeds of 40–80 km/h.
2) We created an interface that enables collision avoidance within 1 s from information provision by using a HUD to display a highlighted infrared image that allows comparison of the virtual image to the real roadway scene observed by the operator.
REFERENCES
[1] “Annual Report on Traffic Accident Statistics 1998,” Institute for Traffic Accident Research and Data Analysis, Tokyo, Japan, 1999.
[2] H. Uno and K. Hiramatsu, “Time margins under emergency conditions and their relationship to driver steering avoidance,” J. Jpn. Ergonomics Soc., vol. 35, no. 4, pp. 219–227, 1999.
[3] M. Hirota, S. Saito, S. Morita, and H. Fukuhara, “Nighttime pedestrian monitoring system and thermal infrared technology,” J. Soc. Auto. Eng. Jpn., vol. 50, no. 11, pp. 58–63, 1996.
[4] S. Yasutomi and H. Mori, “A method for discriminating of pedestrian based on rhythm,” in Proc. IEEE Int. Conf. Robotics and Automation, vol. 2, 1994, pp. 988–995.
[5] T. Adachi, T. Yoshioka, S. Morioka, and S. Matsuoka, “Development of recognition algorithm for crossing pedestrian using laser radar system,” in Proc. Soc. Automotive Engineers Japan, 1996, pp. 363–366.
[6] N. Kitagawa, M. Imanishi, and T. Mizuno, “Development of foreground obstacle detection system,” in Proc. Soc. Automotive Engineers Japan, 1996, pp. 101–104.
[7] N. Uchida, K. Fujita, and K. Katayama, “Concerning crossing collisions in intersections with good visibility,” in Proc. Soc. Automotive Engineers Japan, vol. 30, 1999, pp. 133–138.
[8] T. Matsuyama, Y. Kuno, A. Imiya, and M. Okutomi, Computer Vision: Technology Review and Future Prospects. Tokyo, Japan: New Technology Communications, 1998, pp. 123–137.
[9] Image Analysis Handbook, Univ. Tokyo Press, Tokyo, Japan, 1991, pp. 40–43.
[10] N. Asanuma, M. Kawai, A. Takahashi, and T. Tsuji, “ Introduction of Honda ASV-2 (passenger vehicle),” Honda R&D Tech. Rev., vol. 12, no. 2, pp. 1–8, 2000. |
