Automated synchronization of driving data using vibration and steering events

doi:10.1016/j.patrec.2016.02.011

Pattern Recognition Letters

Volume 75, 1 May 2016, Pages 9-15

https://doi.org/10.1016/j.patrec.2016.02.011 Get rights and content

Highlights

•
A passive synchronization method for driving data is proposed
•
Synchronization of vehicle sensors uses vibration and steering events.
•
Dense optical flow of video is used to capture significant car vibrations events.
•
Cross correlation of vehicle sensor pairs achieves 13.5ms synchronization accuracy.

Abstract

We propose a method for automated synchronization of vehicle sensors useful for the study of multi-modal driver behavior and for the design of advanced driver assistance systems. Multi-sensor decision fusion relies on synchronized data streams in (1) the offline supervised learning context and (2) the online prediction context. In practice, such data streams are often out of sync due to the absence of a real-time clock, use of multiple recording devices, or improper thread scheduling and data buffer management. Cross-correlation of accelerometer, telemetry, audio, and dense optical flow from three video sensors is used to achieve an average synchronization error of 13 milliseconds. The insight underlying the effectiveness of the proposed approach is that the described sensors capture overlapping aspects of vehicle vibrations and vehicle steering allowing the cross-correlation function to serve as a way to compute the delay shift in each sensor. Furthermore, we show the decrease in synchronization error as a function of the duration of the data stream.

Introduction

Large multi-sensor on-road driving datasets offer the promise of helping researchers develop a better understanding of driver behavior in the real world and aid in the design of future advanced driver assistance systems (ADAS) [1], [2]. As an example, the Strategic Highway Research Program (SHRP 2) Naturalistic Driving Study includes over 3400 drivers and vehicles with over 5,400,000 trip records [3] that contains video, telemetry, accelerometer, and other sensor data. The most interesting insights are likely to be discovered not in the individual sensor streams but in their fusion. However, sensor fusion requires accurate sensor synchronization. The practical challenge of fusing “big data”, especially in the driving domain, is that it is often poorly synchronized, especially when individual sensor streams are collected on separate hardware [4]. A synchronization error of 1 s may be deemed acceptable for traditional statistical analyses that focus on data aggregated over a multi-second or multi-minute windows. But in the driving context, given high speed and close proximity to surrounding vehicles, a lot can happen in less than one second. We believe that the study of behavior in relation to situationally relevant cues and the design of an ADAS system that supports driver attention on a moment-to-moment basis requires a maximum synchronization error of 100 ms. For example, events associated with glances (e.g., eye saccades, blinks) often occur on a sub-100-ms timescale [5].

Hundreds of papers are written every year looking at the correlation between two or more aspects of driving (e.g., eye movement and steering behavior). The assumption in many of these analyses is that the underlying data streams are synchronized or aggregated over a long enough window that the synchronization error is not significantly impacting the interpretation of the data. Often, these assumptions are not thoroughly tested. The goal of our work is to motivate the feasibility and the importance of automated synchronization of multi-sensor driving datasets. This includes both “one-factor synchronization” where the passive method is the primary synchronizer and “two-factor synchronization” where the passive method is a validator of a real-time clock based method engineered into the data collection device.

For the passive synchronization process, we use two event types: (1) vehicle vibration and (2) vehicle steering. These event types can be detected by video, audio, telemetry, and accelerometer sensors. Cross-correlation of processed sensor streams is used to compute the time-delay of each sensor pair. We evaluate the automated synchronization framework on a small dataset and achieve an average synchronization error of 13 ms. We also characterize the increase in accuracy with respect to increasing data stream duration which motivates the applicability of this method to online synchronization.

The implementation tutorial and source code for this work is available at: http://lexfridman.com/carsync.

Section snippets

Related work

Sensor synchronization has been studied thoroughly in the domain of sensor networks where, generally, a large number of sensor nodes are densely deployed over a geographic region to observe specific events [6], [7]. The solution is in designing robust synchronization protocols to provide a common notion of time to all the nodes in the sensor network [8]. These protocols rely on the ability to propagate ground truth timing information in a master-slave or peer-to-peer framework. Our paper

Dataset and sensors

In order to validate the proposed synchronization approach we instrumented a 2014 Mercedes CLA with a single-board computer and consumer-level inexpensive sensors: 3 webcams, a shotgun microphone behind the rear tire, GPS, an IMU module, and a CAN controller for vehicle telemetry. The instrumented vehicle is shown in Fig. 1. Details on the positioning of the sensors are provided in the figure’s caption. Through empirical testing we found that small changes in the position of the sensors did not

Synchronization framework

Unless otherwise noted, the figures in this section show sensor traces and cross correlation functions for a single 37 min example run. The two synchronizing event types are vibrations and steering, both densely represented throughout a typical driving session.

Conclusion

Analysis and prediction based on fusion of multi-sensor driving data requires that the data is synchronized. We propose a method for automated synchronization of vehicle sensors based on vibration and steering events. This approach is applicable in both an offline context (i.e., for driver behavior analysis) and an online context (i.e., for real-time intelligent driver assistance). We show that a synchronization error of 13.5 ms can be achieved for a driving session of 35 min.

Acknowledgment

Support for this work was provided by the New England University Transportation Center, and the Toyota Class Action Settlement Safety Research and Education Program. The views and conclusions being expressed are those of the authors, and have not been sponsored, approved, or endorsed by Toyota or plaintiffs class counsel.

References (17)

L. Fridman et al.
Semi-automated annotation of discrete states in large video datasets
AAAI
(2016)
B. Reimer
Driver assistance systems and the transition to automated vehicles: a path to increase older adult safety and mobility?
Public Policy Aging Rep.
(2014)
J.F. Antin
Design of the in-vehicle driving behavior and crash risk study: in support of the SHRP 2 naturalistic driving study
(2011)
W.Q. Meeker et al.
Reliability meets big data: Opportunities and challenges
Qual. Eng.
(2014)
D.K. McGregor et al.
Time on task and blink effects on saccade duration
Ergonomics
(1996)
F. Sivrikaya et al.
Time synchronization in sensor networks: a survey
Netw., IEEE
(2004)
I.-K. Rhee et al.
Clock synchronization in wireless sensor networks: an overview
Sensors
(2009)
J. Elson et al.
Fine-grained network time synchronization using reference broadcasts
ACM SIGOPS Operat. Syst. Rev.
(2002)

There are more references available in the full text version of this article.

Cited by (32)

Automated vehicle data pipeline for accident reconstruction: New insights from LiDAR, camera, and radar data
2023, Accident Analysis and Prevention
As automated vehicles are deployed across the world, it has become critically important to understand how these vehicles interact with each other, as well as with other conventional vehicles on the road. One such method to achieve a deeper understanding of the safety implications for Automated Vehicles (AVs) is to analyze instances where AVs were involved in crashes. Unfortunately, this poses a steep challenge to crash-scene investigators. It is virtually impossible to fully understand the factors that contributed to an AV involved crash without taking into account the vehicle’s perception and decision making. Furthermore, there is a tremendous amount of data that could provide insight into these crashes that is currently unused, as it also requires a deep understanding of the sensors and data management of the vehicle. To alleviate these problems, we propose a data pipeline that takes raw data from all on-board AV sensors such as LiDAR, radar, cameras, IMU’s, and GPS’s. We process this data into visual results that can be analyzed by crash scene investigators with no underlying knowledge of the vehicle’s perception system. To demonstrate the utility of this pipeline, we first analyze the latest information on AV crashes that have occurred in California and then select two crash scenarios that are analyzed in-depth using high-fidelity synthetic data generated from the automated vehicle simulator CARLA. The data visualization procedure is demonstrated on the real-world Kitti dataset by using the YOLO object detector and a monocular depth estimator called AdaBins. Depth from LIDAR is used as ground truth to calibrate and assess the effect of noise and errors in depth estimation. The visualization and data analysis from these scenarios clearly demonstrate the vast improvement in crash investigations that can be obtained from utilizing state-of-the-art sensing and perception systems used on AVs.
Comprehensive driver behaviour review: Taxonomy, issues and challenges, motivations and research direction towards achieving a smart transportation environment
2022, Engineering Applications of Artificial Intelligence
Citation Excerpt :
Analysing the environment surrounding a driver can be used to improve ADAS reaction, hence an enhancement of passenger comfort ability (Klimke et al., 2014). Therefore, the current method of implementing ADAS should also be improved (Fridman et al., 2016). ADAS particularly helps to reduce and avoid collisions, improve road safety, reduce energy consumption and mitigate emissions (Wang et al., 2015a).
The aim of this article is to review and analyse previous academic articles associated with car behaviour analysis for the period of 2010 to June 10, 2021 and understand the benefits of using data collection devices. Articles related to car driver behaviour and sensor utilisation are systematically searched. Three major databases – ScienceDirect, IEEE Xplore and Web of Science – were searched. A set of inclusion and exclusion criteria were developed for the search protocol. All articles were coherently classified via taxonomy. Also. The motives that have led researchers to continue their investigations are explored. The challenges and issues of driver behaviour analysis are illustrated with respect to power consumption, data analysis, detection, cost, security and privacy, sensor usage and individual challenges. The research direction of this review points towards different aspects based on the critical analysis of the different scenarios of driver behaviour studies in real time situations. Here, the critical behaviour analysis of intelligent transportation system development is addressed. The gaps in the reviewed articles include the following: sensors used during experiments, the effect of thresholds on labelling processes or data balancing and classification accuracy, the thresholds in identifying driving styles in the car-following model, insufficient experiment size (large scale or small scale) and limitations in data pre-processing. An implementation map depicting the steps of the case study is provided to give insights into the procedures and the problems they address. This review is expected to offer valid and clear points, contributing to the enhancement of driver behaviour research.
Data synchronization by continuous spatial measurement with Double Ballbar
2021, Measurement: Journal of the International Measurement Confederation
Citation Excerpt :
In respect of the mechanical systems, some researches in recent years have purposed to exploit the vibration in signals to synchronize them. For instance, L. Fridmann et al. determined the time delay in each vehicle sensor by calculating the cross correlation function between data streams [32]. They found out that pairing all sensors with the accelerometer delivers the best results, since the accelerometer captures the vibration signal best.
In this paper, we first present the continuous spatial Double Ballbar (DBB) measurement we purposed in our former work and then suggest two possible solutions to determine the correspondence of the actual machine poses to the DDB measuring values, so as to extend the scope of application of the continuous spatial DBB measuring approach to quantitative tasks such as calibration. For the machines, on which integrated capture of both data is not possible, it is more flexible to capture the data simultaneously in separate devices and adjust them thereafter. We address the issues of asynchrony and inhomogeneity arising therefrom and purpose an approach which could be automated and integrated as a preprocessing module into the data processing program for further use. The technical feasibility of the data capture concept, along with the purposed approach to synchronization and homogenization, was experimentally validated through measurement on a Stewart–Gough platform controlled using Beckoff TwinCAT.
E-Scooter safety: The riding risk analysis based on mobile sensing data
2021, Accident Analysis and Prevention
The emergence of shared electric scooter (E-Scooter) systems offers a new micro-mobility mode in many urban areas worldwide. These systems have rapidly attracted numerous trips on various types of facilities such as sidewalks and bike lanes. After their burst of popularity, there are also growing safety concerns about E-Scooter riding. Consequently, a few cities have banned or temporarily suspended E-Scooters as severe crashes occurred. As an emerging micro-mobility mode, its safety performance is significantly understudied as compared to other travel modes such as cars and bicycles. The lack of crash records further prevents it from understanding the underlying mechanisms that drive the occurrences of E-Scooter crashes. The overarching goal of this paper is to probe the safety risk when riding E-Scooters. Specifically, it aims to study the interactions between e-scooter riding and the environment settings through naturalistic riding experiments. Rather than focusing on the analysis of individual riders’ heterogeneous behavior (e.g., swinging, hard braking, etc.) and rider characteristics (e.g., age, gender, etc.), the naturalistic riding study examines the riding process in different riding circumstances. A mobile sensing system has been developed to collect data for quantifying the surrogate safety metrics in terms of experienced vibrations, speed changes, and proximity to surrounding objects. The results from naturalistic riding experiments show that E-Scooters can experience notable impacts from different riding facilities. Specifically, compared to bicycle riding, more severe vibration events were associated with E-Scooter riding, regardless of the pavement types. Riding on concrete pavements was found to experience a multiple times higher frequency of vibration events when compared to riding on asphalt pavements of the same length. Riding on both sidewalks and vehicle lanes can both encounter high-frequency close contacts in terms of proximity with other objects. These experimental results suggest that E-Scooters are subject to increased safety challenges due to the increased vibrations, speed variations, and constrained riding environments.
Identifying baseflow source areas using remotely sensed and ground-based hydrologic data
2024, Hydrological Processes
Road Condition Detection Techniques: A Comparative Analysis based on Supervised Machine Learning Techniques
2023, 14th International Conference on Advances in Computing, Control, and Telecommunication Technologies, ACT 2023

View all citing articles on Scopus

^☆: This paper has been recommended for acceptance by R. Davies.

View full text

Automated synchronization of driving data using vibration and steering events☆

Highlights

Abstract

Introduction

Section snippets

Related work

Dataset and sensors

Synchronization framework

Conclusion

Acknowledgment

Semi-automated annotation of discrete states in large video datasets

AAAI

Driver assistance systems and the transition to automated vehicles: a path to increase older adult safety and mobility?

Public Policy Aging Rep.

Design of the in-vehicle driving behavior and crash risk study: in support of the SHRP 2 naturalistic driving study

Reliability meets big data: Opportunities and challenges

Qual. Eng.

Time on task and blink effects on saccade duration

Ergonomics

Time synchronization in sensor networks: a survey

Netw., IEEE

Clock synchronization in wireless sensor networks: an overview

Sensors

Fine-grained network time synchronization using reference broadcasts

ACM SIGOPS Operat. Syst. Rev.