Gallery
If you’ve been to Phoenix, San Francisco, Austin, or Los Angeles, you’ve likely encountered a self-driving vehicle from software company Waymo, a subsidiary of Alphabet Inc. Waymo launched its self-driving service in 2020 and has thousands of driverless cars running in these cities.
Watching them in action is unreal. Equipped with a suite of radar, cameras, and LiDAR sensors, the Waymo cars autonomously pick up passengers who hail them with an app and then take them to their destination. The company has an exemplary safety record so far. Data collected by the company from approximately 57 million miles (91,732,608 km) through March 2025 show 92 per cent fewer crashes with pedestrian injuries and 82 per cent fewer crashes with injuries to cyclists and motorcyclists than human drivers. The data also showed 96 per cent fewer crashes resulting in injuries at intersections, which are the leading cause of road harm, according to the National Highway Traffic Safety Administration (NHTSA).
AI Accelerates Development
Thanks to the proliferation of artificial intelligence (AI) and the latest in neural networks, the technology has progressed exponentially, but it’s still far from being adopted by consumer vehicles, regardless of how good Waymo’s safety record is.
Krzysztof Czarnecki, who heads the Waterloo Intelligent Systems and Engineering Lab (WISE) at the University of Waterloo, says that self-driving technology is 99 per cent there, but that’s still not good enough for mainstream adoption.
“We need to add a whole bunch more nines at the end of it, and that’s where the challenge comes in,” he says.
Add Nines
If adding nines is the challenge, what’s the solution? Czarnecki says it’s training a self-driving car to be prepared for any and every scenario that can happen on the road.
These one-off rare occurrences are known as edge cases in software engineering and could be anything from a vehicle driving with a loose wheel to someone hanging off the back of a bus. There are nearly an unlimited number of situations that can happen on the road, things that we might take for granted because we have the power of our mind to help us anticipate what might happen and take action before it does.
We build this “sixth sense” through our experiences and those of others, and we learn to generalize where a computer cannot. Computers must learn each scenario and then how best to respond to it.
Czarnecki and his team at the WISE lab are on the cusp of delivering the largest public dataset on edge cases, which can be used to train self-driving cars.
A Massive Project
In his office, Czarnecki shows me a library of about a thousand driving videos from roads all over North America, many of which are crowd-sourced. There are videos of impaired drivers, people hitching a ride in the back of a pickup truck, cars driving with a three-seater sofa lightly tied to the roof, and people hanging out of a sunroof. All are examples of edge cases that an AI neural network can learn from.
Each video is labelled and categorized with simple English descriptions using a schema that accounts for environmental conditions, road structure, traffic type, and the presence of foreign objects on that road. These labels are then fed into a vision-language model (think ChatGPT) to create the dataset. The initial batch was annotated manually, which was a tedious and time-consuming process, but now the team is using another vision-language model to suggest labels for videos, which then gets reviewed by the team before being accepted. AI can also help discover new examples of edge cases online to continue scaling the dataset.
“We are working intensely on this,” says Czarnecki, who mentions that this public dataset, supported by Transport Canada, will also give other researchers a greatly improved way of handling edge cases.
Winter Driving Adds Many More Edge Cases
Waymo only operates in a few select cities with predominantly warm, sunny weather, and there’s a reason for that. Each city poses new challenges. There are different rules, drivers, driving styles, topographical features, and weather to contend with. It takes Waymo a couple of years and billions of dollars to map and learn all the nuances of each city it moves to.
Snow, ice, and cold temperatures increase the complexity of the data collection and pose many new edge cases unique to winter weather. But Czarnecki says this can eventually be addressed with more training. The other potential issue is reduced traction, which is beyond the perception of AI, so other solutions have to be found. There’s also the problem of sensors getting dirty or covered in snow.
So Close, but We’re Not There Yet
General Motors’ Super Cruise and Tesla’s “Full Self-Driving (FSD)” are currently some of the best examples of self-driving systems on the market. Cadillac allows hands-free driving on over 1.2 million kilometres of LiDAR-mapped roads across North America. FSD takes it a step further because it can handle traffic lights and intersections and doesn’t rely on pre-mapped roads, but rather its sensors and cameras.
While FSD requires you to have your hands on the steering wheel and stay alert, it does all the driving. It’s the closest thing we have to self-driving today, even though it’s technically considered a Level 2 system like Super Cruise. Czarnecki says FSD is even better than Mercedes Drive Pilot, a Level 3 system available in select markets.
Most driver assistance systems today are Level 2. It goes up to 5, and Waymo is Level 4. The levels often get confused, according to Czarnecki, even though they are clearly defined by the Society of Automotive Engineers (SAE).
“There is no such thing as Level 5, and there won’t be anything like that for at least a decade or more,” he says. A Level 5 vehicle is capable of automated driving at all times and in all conditions. It doesn’t require human intervention or driving controls.
Training a learning network with edge case data, like what’s been compiled by Czarnecki and his team, brings autonomous driving one step closer to mass adoption within the industry. It’s not a matter of if, but when.
