Overview

The main goal of the CARLA Autonomous Driving Leaderboard is to evaluate the driving proficiency of autonomous agents in realistic traffic situations. The leaderboard serves as an open platform for the community to perform fair and reproducible evaluations, simplifying the comparison between different approaches.

Sign up!

Index

Task

Autonomous agents have to drive through a set of predefined routes. For each route, agents will be initialized at a starting point and will be directed to drive to a destination point, provided with a description of the route. Routes will happen in a variety of areas, including freeways, urban scenes, and residential districts. Agents will be evaluated in a variety of weather conditions, including daylight scenes, sunset, rain, fog, and night, among others.


Illustration of a route, highlighting the starting point (blue dot) and the target point (red dot) along with the trajectory (green curve).

Agents will face multiple traffic situations based on the NHTSA typology. The full list of traffic situations can be reviewed in this page, but here are some examples.

  • Lane merging.
  • Lane changing.
  • Negotiations at traffic intersections.
  • Negotiations at roundabouts.
  • Handling traffic lights and traffic signs.
  • Coping with pedestrians, cyclists, and other elements.
placeholder image 1 placeholder image 2 placeholder image 2
Illustration of traffic situations present in the CARLA AD leaderboard.

Participation modalities

The leaderboard offers two participation modalities, SENSORS and MAP. These modalities differ in the type of input data that your agent can request from the platform.

Additionally, agents in both modalities will receive a high-level route description indicating the key points that the path has to follow in order to reach the destination. The route is represented as a list of tuples and has two variations.

For the first case, the first element of the tuple contains a waypoint, expressed as a latitude, a longitude, and a z component.

[({'z': 0.0, 'lat': 48.99822669411668, 'lon': 8.002271601998707}, RoadOption.LEFT),
 ({'z': 0.0, 'lat': 48.99822669411668, 'lon': 8.002709765148996}, RoadOption.RIGHT),
 ...
 ({'z': 0.0, 'lat': 48.99822679980298, 'lon': 8.002735250105061}, RoadOption.STRAIGHT)]

The second case is very similar to the previous one, but instead of using GPS coordinates, the route is expressed in world coordinates instead.

[({'x': 153.7, 'y': 15.6, 'z': 0.0}, RoadOption.LEFT),
 ({'x': 148.9, 'y': 67.8, 'z': 0.0}, RoadOption.RIGHT),
 ...
 ({'x': 180.7, 'y': 45.1, 'z': 1.2}, RoadOption.STRAIGHT)]

The distance between two consecutive waypoints could be of up to hundreds of meters. Do not rely on these as your main mechanism to navigate the environment.

The second element contains a high-level command. The set of available high-level commands is:

  • RoadOption.CHANGELANELEFT: Move one lane to the left.
  • RoadOption.CHANGELANERIGHT: Move one lane to the right.
  • RoadOption.LANEFOLLOW: Continue in the current lane.
  • RoadOption.LEFT: Turn left at the intersection.
  • RoadOption.RIGHT: Turn right at the intersection.
  • RoadOption.STRAIGHT: Keep straight at the intersection.

There might be cases where the semantics of left and right is ambiguous. In order to disambiguate these situations, you could consider the GPS position of the next waypoints.

Important: You are not allowed to make use of any privilege information offered by the CARLA simulator, including planners or any type of ground truth. Submissions using these features will be rejected and teams will be banned from the platform.

SENSORS track

On this track agents can request access to the following sensors.


GNSS

IMU

LIDAR

RADAR

RGB camera

Speedometer
sensor.other.gnss sensor.other.imu sensor.lidar.ray_cast sensor.other.radar sensor.camera.rgb sensor.other.speedometer
0-1 units 0-1 units 0-1 units 0-2 units 0-4 units 0-1 units
GPS sensor returning geo location data. 6-axis Inertial Measurement Unit. Velodyne 64 LIDAR. Long-range RADAR (up to 100 meters). Regular camera that captures images. Pseudosensor that provides an approximation of your linear velocity.

Units of each sensor are limited to keep the computational budget under control.

MAP track

Provides the same set of sensor as the SENSORS track does. Additionally, agents can request to access an HD map, which is provided as an OpenDRIVE file parsed as a string.

You are fully responsible to parse or convert this file into a representation that can be useful to your agent.


GNSS

IMU

LIDAR

RADAR

RGB camera

Speedometer
sensor.other.gnss sensor.other.imu sensor.lidar.ray_cast sensor.other.radar sensor.camera.rgb sensor.other.speedometer
0-1 units 0-1 units 0-1 units 0-2 units 0-4 units 0-1 units
GPS sensor returning geo location data. 6-axis Inertial Measurement Unit. Velodyne 64 LIDAR. Long-range RADAR (up to 100 meters). Regular camera that captures images. Pseudosensor that provides an approximation of your linear velocity.

OpenDRIVE map
sensor.opendrive_map
0-1 unit
Pseudosensor that exposes the HD map in OpenDRIVE format parsed as a string.

Evaluation and metrics

The driving proficiency of an agent can be characterized by multiple metrics. For this leaderboard we have selected a set of metrics that help understand different aspects of driving. While all routes have the same type of metrics, their respective values are calculated separately. The specific metrics are as follows:

  • Driving score: \({R_i P_i},\) — Main metric of the leaderboard, serving as the product between the route completion and the infractions penalty. Here \(Ri\) is the percentage of completion of the \(i−th\) route, and \(Pi\), the infraction penalty of the \(i−th\) route.

  • Route completion: Percentage of the route distance completed by an agent.

  • Infraction penalty: \(\prod_j^{\text{ped., ..., stop}} ({p_i^j})^{\text{#infractions}_j}.\) — The leaderboard tracks several types of infractions and this metric aggregates all of these infractions triggered by an agent as a geometric series. Agents start with an ideal 1.0 base score, which is reduced each type an infraction is commited.

When all routes have been completed, a global metric for each of the previous three types is also generated, being the arithmetic mean of all the individual routes combined. The global driving score is the main metric on which you will be classified with respect to other participants.

Infractions and shutdown events

The CARLA leaderboard offers individual metrics for a series of infractions. Each of these has a penalty coefficient that will be applied everytime it happens. Ordered by severity, the infractions are the following.

  • Collisions with pedestrians0.50.
  • Collisions with other vehicles0.60.
  • Collisions with static elements0.65.
  • Running a red light0.70.
  • Running a stop sign0.80.

Besides these, there is one additional infraction which has no coefficient, and instead affects the computation of the route completion (\(R_i\)).

  • Off-road driving — If an agent drives off-road, that percentage of the route will not be considered towards the computation of the route completion score.

Additionally, some events will interrupt the simulation, preventing the agent to continue. In these cases, the route which is being simulated will be shut down, and the leaderboard will move onto the next one, triggering it normally.

  • Route deviation — If an agent deviates more than 30 meters from the assigned route.
  • Agent blocked — If an agent doesn’t take any actions for 180 simulation seconds.
  • Simulation timeout — If no client-server communication can be established in 60 seconds.
  • Route timeout — If the simulation of a route takes too long to finish.

Each time any of the above happens, several details are recorded, which will be displayed as a list for you to see at the route’s individual metrics. Below is an example of a route where the agent both run a red light and deviated from the route.

"infractions": {
    "collisions_layout": [],
    "collisions_pedestrian": [],
    "collisions_vehicle": [],
    "outside_route_lanes": [],
    "red_light": [
        "Agent ran a red light 203 at (x=341.25, y=209.1, z=0.104)"
    ],
    "route_dev": [
        "Agent deviated from the route at (x=95.92, y=165.673, z=0.138)"
    ],
    "route_timeout": [],
    "stop_infraction": [],
    "vehicle_blocked": []
}

The global infractions compress the individual route’s data into a single value and is given as the number of events per Km.


Get started

In order to get familiar with the leaderboard we recommend you to read carefully through the Get started section. Please, spend enough time making sure everything works as expected locally.

Once you are ready, check the Submit section to learn how to prepare your submission.


Terms and Conditions

The CARLA Autonomous Driving Leaderboard is offered for free as a service to the research community thanks to the generosity of our sponsors and collaborators.

Teams are provided with a time budget (currently 200 hours) to evaluate their submissions. Each submission will be evaluated in AWS using a g3.8xlarge instance. This gives users access to a dedicated node with a modern GPU and CPU.

Teams are provided a number of submissions (currently 20 submissions) for a given a month.

Both budgets are automatically refilled every month. The organization of the CARLA leaderboard reserves the right to assign additional budget to a team. The organization also reserves the right to modify the default values of the monthly time budget and/or the number of submissions.

It is strictly prohibited to misuse or attack the infrastructure of the CARLA leaderboard, including all software and hardware that is used to run the service. Actions that deviate from the spirit of the CARLA leaderboard could result in the termination of a team.

For further instructions, please read the terms and conditions.