Blog

Solving the Autonomous Vehicle Safety Challenges

Posted on by Erin

The future of driving is automated.  The certainty of this future—despite well-publicized delays and technical challenges—derives from the numerous and substantial benefits of autonomous vehicles (AVs).  The potential safety benefits are extraordinary. At least 31% of road deaths happen in DUI crashes, and an additional 10% occur when the driver is either distracted or drowsy; up to 90% of all accidents are caused by driver error. An NHTSA report estimated the total societal harm from vehicle crashes at $836 billion, a number that could be significantly mitigated with autonomous vehicles.  In addition to safety, AVs would provide mobility for the disabled, elderly, and children who cannot drive themselves. And, perhaps most practically, AVs would free most of us up from having to drive ourselves when commuting to and from work.

Given all of the benefits of AVs and the hype about them, why are they not here already?  Yes, there are promotional videos in which AVs glide down the freeway. Yes, pilot programs exist in which a small number of AVs operate in a constrained area and only under optimal conditions (e.g., good lighting, good road conditions).  Yes, many new vehicles come equipped with more autonomy than the simple cruise control of previous decades. And yes, many of us have seen videos of AVs poking along hesitantly in urban environments. Nevertheless, none of us are routinely lounging in an AV that is driving naturally—at normal speeds and without being confounded by four-way stops and pedestrians—through a city.

The potential market is enormous with the latest data predicting that AVs will be the fastest-growing automotive sector with sales crossing more than 11 million units by 2033 with a compound annual growth rate of 76.09 percent to 2033.  Allied Market Research estimates that the AV market will reach $54 billion by the end of 2019 and then will explode to $556 billion by 2026. The anticipated growth is reflected in the flurry of M&A activity in the sector as automakers, suppliers and technology players in the space to continue vertical integration and consolidation.

However, before fully autonomous vehicles can become commonplace in urban environments, there are still safety challenges that need to be overcome.  Solving this issue is the key to making autonomous driving and vehicles a reality. In our next blog, we will dive into autonomous driving and how we’re overcoming the critical technical challenge.

You can also read more about our philosophy in this recent IEEE article.

Share on: Twitter | Facebook | LinkedIn

Autonomous Driving and the Need For Motion Planning

Posted on by Erin

Our last blog outlined why autonomous vehicles are not a passing fad and are the future of transportation. However to realize their full potential motion planning is an essential component that will address a myriad of safety challenges. Before we dive into motion planning let’s look at autonomous driving in more detail.

There are many aspects to autonomous driving, all of which need to perform well. You can think of autonomous driving as a four-level stack of activities, in the following top-down order: route planning, behavior planning, motion planning, and physical control.

  1. Route Planning determines the sequence of roads to get from location A to B.
  2. Behavior Planning is the process of determining specific, concrete waypoints along the planned route.  These goals can vary based on road conditions, traffic, and road signage, among other factors.
  3. Motion Planning computes a path from the vehicle’s current position to a waypoint specified by the driving task planner.  This path should be collision-free and likely achieve other goals, such as staying within the lane boundaries.
  4. Physical Control is the process of converting desired speeds and orientations into actual steering and acceleration of the vehicle.

All four levels rely on accurate perception and this is where the majority of solutions continue to emerge. However, control of the car ultimately boils down to these four control levels, and of these, motion planning is the current technical bottleneck and is the primary obstacle to the adoption of AVs.

The current state-of-the-art for motion planning leverages high-performance commodity GPUs. Yet even with a “500-watt supercomputer in the trunk,” as one of our customers recently described it to us, they could compute only three plans per second. Even given high-performance GPUs, motion planning is too computationally difficult for commodity processors to achieve the required performance.

realtime-reaction-time-300x300

What is the required motion planning performance?  At an absolute minimum, the motion planner must be able to react—that is, create a new motion plan—as fast as an alert human driver. That reaction time is on the order of 250msec, and one can imagine current technology evolving to reach that planning speed, albeit at an exorbitant power budget. But we would like to achieve far more than this bare minimum; one of the attractive features of autonomous vehicles is the potential to achieve far greater safety than that achievable by a human driver.  Significantly faster motion planning would translate to much faster reaction times. The difference between reaction times of 250msec1 and 5msec2, for a vehicle traveling at 40mph, is the difference between 15 feet and 0.3 feet traveled before reacting.

Motion planning speed is clearly beneficial for safety, but it offers other important benefits. When motion planning is slow, an AV cannot react quickly to dynamic, non-deterministic agents in its environment, including pedestrians, bicyclists, and other vehicles. When you can’t react quickly, you must move more slowly and more cautiously. Videos of AVs driving in urban environments reveal that they drive slowly and haltingly, having to compensate for their inability to rapidly re-plan.  While such hesitant driving is frustrating to the passenger, it is also likely to aggravate other drivers who are stuck behind the autonomous vehicle or waiting for it to navigate a four-way stop. Aggravating the driving public is dangerous for business, particularly if the driving public clamors for legislation to restrict current hesitant-based driving AVs.

The need for motion planning is clear and our final blog in this series explains how we are making this possible.

RTR AV-Stack

¹250ms is the average human reaction time to the visual stimulus, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4456887/
²Realtime Robotic’s AV motion planner can plan in 1ms, an additional 4 ms is taken to receive and process sensor data.

Share on: Twitter | Facebook | LinkedIn

Realtime Robotics Introduces RapidPlan and RapidSense Solutions Enabling Faster, More Capable Robots With Real-Time Motion Planning

Posted on by Erin

Makes Robots Do Real Work, Real Fast, in the Real World

AUTOMATE – Chicago – (Apr. 10th, 2019) – Realtime Robotics, Inc. announced today the immediate availability of its collision-free motion planning solutions that make it possible for all robots to navigate dynamic environments smoothly, quickly and intuitively. Its RapidPlan and RapidSense solutions harness cutting-edge computer processing and software to enable industrial and collaborative robots to operate safely and collaboratively with people and other robots within workcells. The company will showcase the solutions as part of the Automate Launch Pad Startup Competition, held on Wed., Apr. 10 at 3 pm CT and in the company’s booth, 9649, at McCormick Place.

Today, robots are programmed to move from Point A to Point B along a single, defined path in order to complete a task. When something unexpected happens, such as an object is presented out of alignment, the path is interrupted, or an obstacle interferes with the programmed trajectory, the robot stops working and cannot resume until a person comes to resolve the issue. The loss of production time due to the event itself and the recovery time required to bring the workcell back online is costly.

With the release of RapidPlan and RapidSense, Realtime Robotics eliminates this constraint by giving robots the ability to recognize and respond to changing environments. The result: manufacturers automate more tasks, more efficiently.

RapidPlan combines a hardware motion planning accelerator (MPA) with a software-based roadmap generation toolkit and is shipping now. It is robot-arm and sensor agnostic and can be integrated into new and existing workcells to drastically improve the safety and performance of both industrial and collaborative robots.

With RapidPlan’s unique combination of hardware and software, automation engineers in manufacturing and distribution operations gain unprecedented flexibility to create workcells for robots that reflect the realities of the environments in which they work. Using a highly intuitive interface, users can load as many as 20 million motions into the MPA. Once stored, RapidPlan can evaluate up to 800,000 motions at 30 frames-per-second in order to ensure that a robot continues to work regardless of interruption.

The company also announced the beta release of RapidSense, an advanced solution that enables robots to “see” everything within a workcell.  The solution’s Calibration tool is a user-friendly interface that can autonomously calibrate as many as 10 depth-sensing cameras, vastly reducing setup and the reconfiguration time required when deploying vision systems.  Once running, RapidSense sends data gathered with the cameras seamlessly to RapidPlan, making it possible for robots to dynamically replan to avoid a person, another robot, or an object that moves into the workspace.

Together, RapidPlan and RapidSense allow people and multiple robots to work collaboratively and cooperatively within the same workcell, without the need for expensive safety systems or time-consuming programming efforts.

“Automation professionals know that more automation is the key to increasing productivity and lowering costs; they also know that today’s robots are limited in terms of speed and safety… Our collision-free motion planning solutions allow robots to perform safely in dynamic, unstructured, and collaborative workspaces, while instantaneously reacting to changes as they occur.”

— Peter Howard, CEO, Realtime Robotics

About Realtime Robotics

Realtime Robotics, Inc.’s responsive motion planning eliminates a huge challenge on the evolutionary path for robots and autonomous vehicles: the constraining trade-off between speed and safety. Its innovative products, RapidPlan and RapidSense, harness next-generation computer processors and software that make it possible for robots and autonomous vehicles to evaluate alternative scenarios and choose the optimal path to avoid collisions – all within milliseconds.

The company was founded in 2016, by Duke University professors Dan Sorin, George Konidaris, and top researchers Sean Murray and Will Floyd-Jones, based on groundbreaking DARPA-funded research in motion planning. Based in Boston, the company’s early investors include Sparx Group Co., Ltd., Toyota AI, and Scrum Ventures. More information about Realtime Robotics, its technology, and the business benefits it offers can be found on our website.

Share on: Twitter | Facebook | Linkedin

Solving the Autonomous Vehicles Motion Planning Conundrum

Posted on by Erin

Our last blog explained the importance of motion planning for the adoption of AV. In this post, we’ll discuss how Realtime Robotics is enabling normal autonomous driving in urban environments.

Electronics Board

Realtime Robotics has developed its own, special-purpose computer processor for AV motion planning to address the critical need for super-fast motion planning.  This custom-designed processor is tailored to perform motion planning at rates of hundreds to thousands of times per second. The speed advantage of our processor, compared to commodity processors, derives from its sole focus on motion planning; all of its hardware resources and all of its power budget are devoted to motion planning.  The concept of special-purpose processors is not new—graphics processors (GPUs) and bitcoin mining processors are two notable examples—but Realtime Robotics has created the first motion planning processor.

The tremendous speed of the Realtime Robotics motion planning processor enables ultra-fast reaction times and is, in fact, faster than needed simply to react.  There is relatively little difference between reaction times of 5msec and 10msec (0.3 vs 0.6 feet traveled at 40mph). Nevertheless, the ability to plan faster than necessary to react enables an exciting new feature: risk-aware motion planning.

Risk-aware motion planning considers multiple possible behaviors of the non-deterministic agents in the environment and determines a motion plan that minimizes risk given these uncertain behaviors.  Consider a reaction time goal of 10msec. Our motion planner can compute multiple motion plans in that time, and each of those plans can consider different possible behaviors of the other agents. Faster planning enables more plans to be computed per reaction time epoch, thus increasing the range of possible behaviors considered.  This risk-aware planning is analogous to how humans drive. When, for example, we see a bicyclist on our right, we drive far enough away from her to accommodate the risk of her veering to her left, and we subconsciously compute this risk based on a mental model of bicyclist behavior that is informed by prior experiences.

If autonomous vehicles are to fulfill their promise of relieving the tedium of driving and drastically improve the safety of our roads, they will need to be able to deal with the complexity of decision-making in the real world. Realtime’s specialized motion planning processor offers a path to defeating the key remaining obstacle to achieving this vision, leading to autonomous vehicles that react instantly, and that can deal with the uncertainty inherent in complex driving scenarios like the busy urban and suburban streets that you and I navigate every day.

RTR AV-Stack-Bicyclist

Share on: Twitter | Facebook | LinkedIn

Realtime “Graduates” from MassRobotics

Posted on by Erin

Realtime Robotics “graduated” from the MassRobotics incubator late last year. Not-for-profit MassRobotics is a Boston-based workspace for robotics companies to collaborate, build and innovate. Realtime Robotics originally headquartered in MassRobotics as it provided a reasonably priced, centrally located office and lab space that offered the flexibility to grow with their project needs and infrastructure requirements.

“MassRobotics is a perfect place for a robotics startup to launch from,” said Peter Howard, CEO of Realtime Robotics. “We had access to a flexible workspace, like-minded people to bounce ideas off of, robots and hardware without consuming our limited funds, and invaluable connections to customers, investors, advisors, researchers, and events that linked us to incredible employees and partnerships. We would definitely not have made the rapid progress we have achieved had we not had the amazing resources that come with being a MassRobotics member.”

Realtime Robotics moved their headquarters to nearby Tower Point in the Fort Point district of Boston. They currently have 30 full-time employees, most of which you can see here in their new space:

Team-600x400

Learn more about MassRobotics.

Share on: Twitter | Facebook | Linkedin

Deciphering Deconfliction So Robots Can Work in Harmony

Posted on by Erin

Many automation tasks are amenable to being performed by multiple robots working together within the same work-cell. The robots can perform similar jobs—for example, painting different portions of an object—or they can perform different jobs, like wrapping and packaging. In the case of heterogenous jobs, you might even want to use different robots that are tailored for each job. Regardless of the task, the goal of multi-robot cells is speed; more robots working together can finish a job sooner than a single robot.

Robot-cooking-300x169

The key challenge in deploying multi-robot work-cells is deconfliction: ensuring that the robots avoid collisions with each other. Imagine a kitchen with one chef preparing a meal—they can move wherever they please. However, if we add more chefs, then the chefs have to be careful to not bump into each other. Now imagine writing software that controls robotic chefs with the goal of optimizing performance: minimizing the time to prepare food and guaranteeing that the chefs never collide.

Typically, to control robots in a multi-robot work-cell, expert robotics software engineers spend a very long time writing software that ensures collisions are avoided. The demand for collision avoidance coupled with the expense of software engineering often leads to suboptimal solutions that do not take full advantage of robots such as exaggerated movements to give plenty of buffer room or only allowing a single robot to move at a time. Worse, if any of the jobs or any of the robots change in any way, then the software engineer must recheck every robot motion and interlock to ensure they are still collision-free. Thus, despite the appeal of multi-robot work-cells, previous solutions for implementing them have been expensive and fragile.

At Realtime Robotics, we have solved this important problem with our ground-breaking technology that solves deconfliction bottlenecks. Our solution enables each robot to account for the motion plans of all other robots in real-time while simultaneously providing each robot with its own collision-free motion plan. The primary burdens of multi-robot work-cells, discovering motion plans for the robots that do not collide and validating the motion plans during deployment, are simplified by our technology. Our solution can be deployed with any robot and adapts to on-the-fly changes in the task and changes in the robots and even robot failures. This saves robotics software engineers from having to constantly re-validate software during deployment, thus reducing the overall time and cost of multi-robot workcell deployment.

Our next post will explore in more detail how we have reconfigured deconfliction!

Share on: Twitter | Facebook | LinkedIn

PickNik Partners with Realtime Robotics

Posted on by Erin

We are thrilled to welcome PickNik Robotics as our latest integration partner! The company recently incorporated RapidPlan into the MoveIt! open source motion planning framework. This allows users to seamlessly integrate our real-time motion planning technology into their existing robotic development process. This will make it easier to deploy our solution delivering reliable real-time planning and collision checking, a well-established constraint to the large-scale adoption of robotic systems.

PickNik Robotics is recognized for its world-class robotics expertise and open source frameworks for motion planning software. It provides a range of solutions including ROS integration, complex inverse kinematics, and motion planning. The plugin it has developed provides full planning and collision checking capabilities via a common motion planning interface and MoveIt! users will now be able to benefit from our innovative solution for collision-free motion planning: RapidPlan.

Our solution enables machines to navigate dynamic environments smoothly and quickly through high-frequency collision checking and planning using roadmaps. With the new MoveIt! plugin and the Realtime Robotics Motion Planning Accelerator (MPA) users will have access to vision sensor data like occupancy and point clouds and be able to compare this data with Robot motions to check for collisions in real-time. The MPA contains user-generated roadmaps which define all of the robot’s movements inside the workcell. RapidPlan comes with an easy-to-use software path planner for generating roadmaps, that together with the MPA enables a robot to plan up to 1000 different motions every second.

We are excited to collaborate with PickNik Robotics and look forward to continuing to work together on future innovations!

Share on: Twitter | Facebook | LinkedIn

From Academic Research to Commercial Solution, Now We’re Onto Our Next Leap

Posted on by Erin

As a co-founder of Realtime Robotics, it gives me great pleasure to share the exciting news about the investments that have been made as part of our Series A round!  We raised $11.7 million from SPARX Asset Management, Mitsubishi Electric Corporation, Hyundai Motor Company, OMRON Ventures, Toyota AI Ventures, Scrum Ventures, and the Duke Angel Network. This investment is both a great show of confidence in our work and a tremendous catalyst for the next stage of our company.  We are delighted to partner with our investors to bring our game-changing robotics technology to the masses.

A startup company like Realtime Robotics needs three things to succeed: a great idea, outstanding people, and investors who share the company’s vision for the future.

Dan

With this Series A investment, the incredible Realtime team can now take the next step in taking our novel technology from limited-release products to market-ready products that will transform industrial automation, logistics, autonomous driving, and several other industries.  As someone who was there from the beginning—when the founders made the first key breakthrough and then produced a demonstration of the initial idea in practice in a university lab—it is thrilling to see where we are now and to imagine where we will be next.

Stay tuned!
– Dan Sorin

Share on: Twitter | Facebook | Linkedin

Deciphering Deconfliction with RapidPlan MPA

Posted on by Erin

Our first blog in this series explained why deconfliction is a critical challenge constraining the deployment of multi-robot work-cells. For robots to achieve their full potential, they must be able to work together safely and efficiently while avoiding collisions with each other. Without solving deconfliction, robots will continue to have slow deployment and will continue to operate inefficiently.

The technology

Realtime Robotics’ groundbreaking solution, RapidPlan MPA, enables seamless integration via motion planning, allowing robots to work in groups at a productive pace without colliding; delivering significant time and cost savings. We’ve developed a custom-designed, proprietary hardware accelerator which seamlessly plugs into your workstation and can check pre-computed motions for possible collisions in milliseconds. With our technology, robots can make intelligent decisions within work cells, and one hierarchical program can de-conflict multiple robots.

Key benefits from RapidPlan MPA include:

  • Continuous collision-free operation: Ability to transform an existing workspace into a collaborative space with multiple robots working together continuously without colliding at speed.
  • Reduced implementation effort: Reduced programming efforts as interference zones, interlocks, and time/movement synchronization, are now redundant.
  • Dynamic task allocation: When a single robot fails, or the target location changes during execution, Realtime’s planning system enables continuous operation.
  • Customer-driven compatibility: Realtime’s technology is robot agnostic, giving the customer the power to choose the right robot for the given task.
  • Sensor-free operation: RapidPlan’s deconfliction features are not dependent on external sensors allowing implementation in all environments.

By integrating the Realtime Robotics RapidPlan, robots can plan collision-free motion plans intelligently within existing workcells allowing organizations to finally realize the long-promised productivity gains.

Get in touch to find out more about how our motion planning technology is transforming the way industrial robots can work together.

Share on: Twitter | Facebook | LinkedIn

Realtime Robotics’ Technology Interfaces with Yaskawa Motoman

Posted on by Erin

Realtime Robotics’ technology allows this Yaskawa Motoman GP7 to operate at full speed while autonomously avoiding unexpected obstacles that enter its workspace. Traditionally, a robot moving at these speeds would only be operated behind a cage in order to ensure safety. Realtime Robotics provides Motion Planning and Spatial Perception solutions that will, when safety certified, enable this Yaskawa Motoman to be fully autonomous and work collaboratively.

Realtime’s RapidPlan technology makes use of hardware and software to create collision-free motion plans for the robot in real-time. Users define the workcell and specify target locations for the robot to travel to within the RapidPlan toolkit. Then the RapidPlan toolkit will precompute and return all possible paths the robot can take without collisions, and the RapidPlan controller will execute these paths driving robots such as the Motoman GP7 to the desired location. When combined with Realtime’s RapidSense technology, the robot can react and replan a subsequent collision-free path based on obstacles and task variations in real-time.

Realtimes’ RapidSense technology uses depth cameras to continuously scan the environment and create a real-time map for the robot to navigate. The green region you see in the video is the resulting map of unknown obstacles. This map is used by RapidPlan to find paths around the table, robot stand, and any unknown obstacles detected by RapidSense.

In industrial settings where processes change rapidly and safety is a paramount, Realtime Robotics’ technology is there to make your life easier. Rather than spending time programming discrete motions you want your robot to take, you can spend more time developing new applications. Current applications can be retrofitted within a few hours and there is repeatability across all deployments. Realtime Robotics manages your motion planning and spatial perception so you can unleash the full potential of your automation.

Get in touch for more information on our technology and how it can enhance your application.

Share on: Twitter | Facebook | Linkedin