The information on this page concerns Leaderboard 2.0, the latest version. If you are using the previous version of leaderboard please consult the Leaderboard 1.0 section.

News

2024 CVPR Autonomous Grand Challenge

This year, we are thrilled to announce that the CARLA AD Leaderboard is part of this year’s CVPR Autonomous Grand Challenge! Showcase your AV development skills and compete for a remarkable prize of $10,000 or an HP workstation! Check out the CVPR Autonomous Grand Challenge website.

Teams participating in the CVPR challenge will be required to complete two qualifying rounds, ensuring their agents are adept at basic navigation. Once qualified, you’ll gain access to submit agents for the complete Leaderboard 2.0 route selection. The Leaderboard 2.0 now allows double the number of sensors compared to previous competitions, including 8 RGB cameras, 2 LIDARs and 4 RADARs.

Please note: In preparation for the competition, we’ll be temporarily closing Leaderboard 1.0. Leaderboard 2.0 will be reopened for submissions on March 25th.

Remember to join our Discord server to be up to date with all the Leaderboard related news, and feel free to ask any questions about the Leaderboard in the CARLA AD Leaderboard channel, or to any of the CARLA team members.

Overview

The main goal of the CARLA Autonomous Driving Leaderboard is to evaluate the driving proficiency of autonomous agents in realistic traffic scenarios. The leaderboard serves as an open platform for the community to perform fair and reproducible evaluations of autonomous vehicle agents, simplifying the comparison between different approaches. Leaderboard is currently at version 2.0, version 1.0 is still supported.

Index

Task

The CARLA AD Leaderboard challenges AD agents to drive through a set of predefined routes. For each route, agents will be initialized at a starting point and directed to drive to a destination point, provided with a description of the route through GPS style coordinates, map coordinates or route instructions. Routes are defined in a variety of situations, including freeways, urban areas, residential districts and rural settings. The Leaderboard evaluates AD agents in a variety of weather conditions, including daylight scenes, sunset, rain, fog, and night, among others.



Scenarios

Agents will face multiple traffic scenarios based on the NHTSA typology. The full list of traffic scenarios can be reviewed in this page, but here are some examples.

  • Lane merging.
  • Lane changing.
  • Negotiations at traffic intersections.
  • Negotiations at roundabouts.
  • Handling traffic lights and traffic signs.
  • Yielding to emergency vehicles.
  • Coping with pedestrians, cyclists, and other elements.
placeholder image 1 placeholder image 2 placeholder image 2
Illustration of traffic situations present in the CARLA AD leaderboard.

Participation modalities

The leaderboard offers two participation modalities, SENSORS and MAP. These modalities differ in the type of input data that your agent can request from the platform.

Additionally, agents in both modalities will receive a high-level route description indicating the key points that the path has to follow in order to reach the destination. The route is represented as a list of tuples and has two variations.

For the first case, the first element of the tuple contains a waypoint, expressed as a latitude, a longitude, and a z component.

[({'z': 0.0, 'lat': 48.99822669411668, 'lon': 8.002271601998707}, RoadOption.LEFT),
 ({'z': 0.0, 'lat': 48.99822669411668, 'lon': 8.002709765148996}, RoadOption.RIGHT),
 ...
 ({'z': 0.0, 'lat': 48.99822679980298, 'lon': 8.002735250105061}, RoadOption.STRAIGHT)]

The second case is very similar to the previous one, but instead of using GPS coordinates, the route is expressed in world coordinates instead.

[({'x': 153.7, 'y': 15.6, 'z': 0.0}, RoadOption.LEFT),
 ({'x': 148.9, 'y': 67.8, 'z': 0.0}, RoadOption.RIGHT),
 ...
 ({'x': 180.7, 'y': 45.1, 'z': 1.2}, RoadOption.STRAIGHT)]

The distance between two consecutive waypoints could be up to hundreds of meters. Do not rely on these as your principal mechanism to navigate the environment.

The second element contains a high-level command. The set of available high-level commands is:

  • RoadOption.CHANGELANELEFT: Move one lane to the left.
  • RoadOption.CHANGELANERIGHT: Move one lane to the right.
  • RoadOption.LANEFOLLOW: Continue in the current lane.
  • RoadOption.LEFT: Turn left at the intersection.
  • RoadOption.RIGHT: Turn right at the intersection.
  • RoadOption.STRAIGHT: Keep straight at the intersection.

There might be cases where the semantics of left and right is ambiguous. In order to disambiguate these situations, you could consider the GPS position of the next waypoints.

Important: You are not allowed to make use of any privilege information offered by the CARLA simulator, including planners or any type of ground truth. Submissions using these features will be rejected and teams will be banned from the platform.

SENSORS track

On this track agents can request access to the following sensors.


GNSS
IMU
LIDAR
RADAR
RGB camera
Speedometer
sensor.other.gnss sensor.other.imu sensor.lidar.ray_cast sensor.other.radar sensor.camera.rgb sensor.other.speedometer
0-1 units 0-1 units 0-2 units 0-4 units 0-8 units 0-1 units
GPS sensor returning geo location data. 6-axis Inertial Measurement Unit. Velodyne 64 LIDAR. Long-range RADAR (up to 100 meters). Regular camera that captures images. Pseudosensor that provides an approximation of your linear velocity.

Units of each sensor are limited to keep the computational budget under control.

MAP track

Provides the same set of sensor as the SENSORS track does. Additionally, agents can request to access an HD map, which is provided as an OpenDRIVE file parsed as a string.

You are fully responsible to parse or convert this file into a representation that can be useful to your agent.


GNSS
IMU
LIDAR
RADAR
RGB camera
Speedometer
sensor.other.gnss sensor.other.imu sensor.lidar.ray_cast sensor.other.radar sensor.camera.rgb sensor.other.speedometer
0-1 units 0-1 units 0-2 units 0-4 units 0-8 units 0-1 units
GPS sensor returning geo location data. 6-axis Inertial Measurement Unit. Velodyne 64 LIDAR. Long-range RADAR (up to 100 meters). Regular camera that captures images. Pseudosensor that provides an approximation of your linear velocity.

OpenDRIVE map
sensor.opendrive_map
0-1 unit
Pseudosensor that exposes the HD map in OpenDRIVE format parsed as a string.

Qualifying tracks

The Leaderboard 2.0 features 2 qualifying tracks in which your agent will be required to safely navigate a straightforward route through an unseen map featuring simple driving scenarios. Each of the SENSOR and MAP tracks has its own respective qualifier. You must reach a minimum driving score in the qualifiers to be permitted to submit agents to the Leaderboard 2.0 main tracks.

To save computing resources the qualifiers will permit a more restrictive sensor suite than the main competition, with 0-4 RGB cameras, 0-2 RADARs and 0-1 LIDARs, along with GNSS, IMU and speedometer (and OpenDRIVE for the MAP track). We encourage you to use the qualifier to ensure that your AD stack is properly configured and functioning correctly with the Leaderboard.

Evaluation and metrics

The driving proficiency of an agent can be characterized by multiple metrics. For this leaderboard we have selected a set of metrics that help understand different aspects of driving. While all routes have the same type of metrics, their respective values are calculated separately. The specific metrics are as follows:

  • Driving score: \({R_i P_i},\) — Main metric of the leaderboard, serving as the product between the route completion and the infractions penalty. Here \(Ri\) is the percentage of completion of the \(i−th\) route, and \(Pi\), the infraction penalty of the \(i−th\) route.

  • Route completion: Percentage of the route distance completed by an agent.

  • Infraction penalty: \(\prod_j^{\text{ped., ..., stop}} ({p_i^j})^{\text{#infractions}_j}.\) — The leaderboard tracks several types of infractions and this metric aggregates all of these infractions triggered by an agent as a geometric series. Agents start with an ideal 1.0 base score, which is reduced each type an infraction is commited.

When all routes have been completed, a global metric for each of the previous three types is also generated, being the arithmetic mean of all the individual routes combined. The global driving score is the main metric on which you will be classified with respect to other participants.

Infractions and shutdown events

The CARLA leaderboard offers individual metrics for a series of infractions. Each of these has a penalty coefficient that will be applied everytime it happens. Ordered by severity, the infractions are the following.

  • Collisions with pedestrians — 0.50.
  • Collisions with other vehicles — 0.60.
  • Collisions with static elements — 0.65.
  • Running a red light — 0.70.
  • Running a stop sign — 0.80.

Some scenarios feature behaviors that can block the ego-vehicle indefinitely. These scenarios will have a timeout of 4 minutes after which the ego-vehicle will be released to continue the route. However, a penalty is applied when the time limit is breached:

  • Scenario timeout — 0.7

The agent is expected to maintain a minimum speed in keeping with nearby traffic. The agent’s speed will be compared with the speed of nearby vehicles. Failure to maintain a suitable speed will result in a penalty. The penalty applied is dependent on the magnitude of the speed difference, up to the following value:

  • Failure to maintain minimum speed — 0.7

The agent should yield to emergency vehicles coming from behind. Failure to allow the emergency vehicle to pass will incur a penalty:

  • Failure to yield to emergency vehicle — 0.7

Besides these, there is one additional infraction which has no coefficient, and instead affects the computation of the route completion (\(R_i\)).

  • Off-road driving — If an agent drives off-road, that percentage of the route will not be considered towards the computation of the route completion score.

Additionally, some events will interrupt the simulation, preventing the agent to continue. In these cases, the route which is being simulated will be shut down, and the leaderboard will move onto the next one, triggering it normally.

  • Route deviation — If an agent deviates more than 30 meters from the assigned route.
  • Agent blocked — If an agent doesn’t take any actions for 180 simulation seconds.
  • Simulation timeout — If no client-server communication can be established in 60 seconds.
  • Route timeout — If the simulation of a route takes too long to finish.

Each time any of the above happens, several details are recorded, which will be displayed as a list for you to see at the route’s individual metrics. Below is an example of a route where the agent both run a red light and deviated from the route.

"infractions": {
  "Collisions with layout": [],
  "Collisions with pedestrians": [],
  "Collisions with vehicles": [],
  "Red lights infractions": [
        "Agent ran a red light 203 at (x=341.25, y=209.1, z=0.104)"
  ],
  "Stop sign infractions": [],
  "Off-road infractions": [],
  "Min speed infractions": [],
  "Yield to emergency vehicle infractions": [],
  "Scenario timeouts": [],
  "Route deviations": [
        "Agent deviated from the route at (x=95.92, y=165.673, z=0.138)"
  ],
  "Agent blocked": [],
  "Route timeouts": []
}

The global infractions compress the individual route’s data into a single value and is given as the number of events per Km.

Get started

In order to get familiar with the leaderboard we recommend you to read carefully through the Get started section. Please, spend enough time making sure everything works as expected locally.

Once you are ready, check the Submit section to learn how to prepare your submission.

Leaderboard 1.0

Leaderboard is currently at version 2.0. Leaderboard version 1.0 is still available to support your previous work. Please consult the Get started and Submit pages for Leaderboard version 1.0 submissions.

Sign up!

Terms and Conditions

The CARLA Autonomous Driving Leaderboard is offered for free as a service to the research community thanks to the generosity of our sponsors and collaborators.

Each submission will be evaluated in AWS using a g5.12xlarge instance. This gives users access to a dedicated node with a modern GPU and CPU.

Teams are provided a finite number of submissions (currently 5 submissions) for a given a month.

Submission allowance is automatically refilled every month. The organizers of the CARLA leaderboard reserve the right to assign additional allowances to a team. The organization also reserves the right to modify the default values of the monthly allowance for submissions.

It is strictly prohibited to misuse or attack the infrastructure of the CARLA leaderboard, including all software and hardware that is used to run the service. Actions that deviate from the spirit of the CARLA leaderboard could result in the termination of a team.

For further instructions, please read the terms and conditions.