The information on this page concerns Leaderboard 2.0, the latest version. If you are using the previous version of leaderboard please consult the Leaderboard 1.0 section.

News

The start of the CARLA Autonomous Driving Challenge 2023 comes with several important changes, which this section will highlight in detail. We recommend everyone, regardless of how familiar they are with the CARLA Leaderboard, to take their time reading.

Also, remember to join our Discord server to be up to date with all the Leaderboard related news, and feel free to ask any questions about the Leaderboard in the CARLA AD Leaderboard channel, or to any of the CARLA team members.

Temporary closure of the Leaderboard 1.0

For the duration of the CARLA Autonomous Driving Challenge 2023, submission to the Leaderboard 1.0 won’t be accepted to encourage everyone to actively participate and try out the new Leaderboard 2.0. Try out the improved version of the Leaderboard with brand new maps and scenarios.

All Leaderboard 2.0 changes can be found in the CHANGELOG.md files at ScenarioRunner and the Leaderboard

Leaderboard 2.0 maps

One of the features of CARLA’s 0.9.14 version was the addition of the new large map Town12, which has been specifically designed to provide a different driving experience than previous towns. The Leaderboard 2.0 makes full use of Town12’s features to create a new set of training routes and scenarios that will challenge even the higher scoring stacks in the Leaderboard 1.0.

Additionally, CARLA 0.9.15 will also release with another new large map, Town13. It has been created with a similar approach to Town12, but making sure that the map feels different enough to allow it to be used as a validation tool. And again, similar to the training routes, the Leaderboard 2.0 comes prepared with a new set of validation routes and scenarios.

The CARLA package available in the Get Started section has been updated to include these two new maps (along with all the other AdditionalMaps).

Lastly, a third large map has been created, Town14, which is used by the Leaderboard to test the user’s submissions. While it follows a similar approach to Town12 and Town13, it’s exact nature will be kept secret to ensure a correct testing environment.

Training database creation tools

As previously mentioned, the Leaderboard 2.0 comes with new more challenging scenarios that will test even the best Leaderboard 1.0 stacks. Thse scenarios are more complex so creating a training database has become more challenging. To remedy this, we are providing a set of CARLA logs that showcase a perfectly executed run for each of the new scenarios, along with a script to run these logs with any desired sensor, saving their data.

More information can be found in here.

New frontend

The CARLA Autonomous Driving Challenge 2023 has been created in collaboration with our new partner EvalAI, an open source evaluation platform, which has been a huge help in the addition of the CARLA Challenge to their frontend. For more information, the Submit section has a detailed explanation of the steps needed to submit to this new frontend.

New Leaderboard cloud infraestructure

Finally, with the start of the CARLA Autonomous Driving Challenge 2023, the Leaderboard cloud infraestructure has been redesigned, hugely improving the submission’s testing speed for both Leaderboard 1.0 and Leaderboard 2.0. This is due to two factors.

First of all, the AWS instance types have been changed from a g3.8xlarge to a g5.12xlarge. This, in and of itself, already increases the performance, but these new instances also come with 4 GPUs, allowing to paralellize the computation of the Leaderboard results.

Note that the official Leaderboard 1.0 branches have been updated to leaderboard-1.0 for both Scenario Runner and the Leaderboard, which include new arguments and scripts to allow this new merging feature. The evaluation itself has remained unchanged, so any results created with these new branches will still be comparable with the old ones.

Overview

The main goal of the CARLA Autonomous Driving Leaderboard is to evaluate the driving proficiency of autonomous agents in realistic traffic scenarios. The leaderboard serves as an open platform for the community to perform fair and reproducible evaluations of autonomous vehicle agents, simplifying the comparison between different approaches. Leaderboard is currently at version 2.0, version 1.0 is still supported.

Index

Task

The CARLA AD Leaderboard challenges AD agents to drive through a set of predefined routes. For each route, agents will be initialized at a starting point and directed to drive to a destination point, provided with a description of the route through GPS style coordinates, map coordinates or route instructions. Routes are defined in a variety of situations, including freeways, urban areas, residential districts and rural settings. The Leaderboard evaluates AD agents in a variety of weather conditions, including daylight scenes, sunset, rain, fog, and night, among others.



Scenarios

Agents will face multiple traffic scenarios based on the NHTSA typology. The full list of traffic scenarios can be reviewed in this page, but here are some examples.

  • Lane merging.
  • Lane changing.
  • Negotiations at traffic intersections.
  • Negotiations at roundabouts.
  • Handling traffic lights and traffic signs.
  • Yielding to emergency vehicles.
  • Coping with pedestrians, cyclists, and other elements.
placeholder image 1 placeholder image 2 placeholder image 2
Illustration of traffic situations present in the CARLA AD leaderboard.

Participation modalities

The leaderboard offers two participation modalities, SENSORS and MAP. These modalities differ in the type of input data that your agent can request from the platform.

Additionally, agents in both modalities will receive a high-level route description indicating the key points that the path has to follow in order to reach the destination. The route is represented as a list of tuples and has two variations.

For the first case, the first element of the tuple contains a waypoint, expressed as a latitude, a longitude, and a z component.

[({'z': 0.0, 'lat': 48.99822669411668, 'lon': 8.002271601998707}, RoadOption.LEFT),
 ({'z': 0.0, 'lat': 48.99822669411668, 'lon': 8.002709765148996}, RoadOption.RIGHT),
 ...
 ({'z': 0.0, 'lat': 48.99822679980298, 'lon': 8.002735250105061}, RoadOption.STRAIGHT)]

The second case is very similar to the previous one, but instead of using GPS coordinates, the route is expressed in world coordinates instead.

[({'x': 153.7, 'y': 15.6, 'z': 0.0}, RoadOption.LEFT),
 ({'x': 148.9, 'y': 67.8, 'z': 0.0}, RoadOption.RIGHT),
 ...
 ({'x': 180.7, 'y': 45.1, 'z': 1.2}, RoadOption.STRAIGHT)]

The distance between two consecutive waypoints could be up to hundreds of meters. Do not rely on these as your principal mechanism to navigate the environment.

The second element contains a high-level command. The set of available high-level commands is:

  • RoadOption.CHANGELANELEFT: Move one lane to the left.
  • RoadOption.CHANGELANERIGHT: Move one lane to the right.
  • RoadOption.LANEFOLLOW: Continue in the current lane.
  • RoadOption.LEFT: Turn left at the intersection.
  • RoadOption.RIGHT: Turn right at the intersection.
  • RoadOption.STRAIGHT: Keep straight at the intersection.

There might be cases where the semantics of left and right is ambiguous. In order to disambiguate these situations, you could consider the GPS position of the next waypoints.

Important: You are not allowed to make use of any privilege information offered by the CARLA simulator, including planners or any type of ground truth. Submissions using these features will be rejected and teams will be banned from the platform.

SENSORS track

On this track agents can request access to the following sensors.


GNSS
IMU
LIDAR
RADAR
RGB camera
Speedometer
sensor.other.gnss sensor.other.imu sensor.lidar.ray_cast sensor.other.radar sensor.camera.rgb sensor.other.speedometer
0-1 units 0-1 units 0-1 units 0-2 units 0-4 units 0-1 units
GPS sensor returning geo location data. 6-axis Inertial Measurement Unit. Velodyne 64 LIDAR. Long-range RADAR (up to 100 meters). Regular camera that captures images. Pseudosensor that provides an approximation of your linear velocity.

Units of each sensor are limited to keep the computational budget under control.

MAP track

Provides the same set of sensor as the SENSORS track does. Additionally, agents can request to access an HD map, which is provided as an OpenDRIVE file parsed as a string.

You are fully responsible to parse or convert this file into a representation that can be useful to your agent.


GNSS
IMU
LIDAR
RADAR
RGB camera
Speedometer
sensor.other.gnss sensor.other.imu sensor.lidar.ray_cast sensor.other.radar sensor.camera.rgb sensor.other.speedometer
0-1 units 0-1 units 0-1 units 0-2 units 0-4 units 0-1 units
GPS sensor returning geo location data. 6-axis Inertial Measurement Unit. Velodyne 64 LIDAR. Long-range RADAR (up to 100 meters). Regular camera that captures images. Pseudosensor that provides an approximation of your linear velocity.

OpenDRIVE map
sensor.opendrive_map
0-1 unit
Pseudosensor that exposes the HD map in OpenDRIVE format parsed as a string.

Evaluation and metrics

The driving proficiency of an agent can be characterized by multiple metrics. For this leaderboard we have selected a set of metrics that help understand different aspects of driving. While all routes have the same type of metrics, their respective values are calculated separately. The specific metrics are as follows:

  • Driving score: \({R_i P_i},\) — Main metric of the leaderboard, serving as the product between the route completion and the infractions penalty. Here \(Ri\) is the percentage of completion of the \(i−th\) route, and \(Pi\), the infraction penalty of the \(i−th\) route.

  • Route completion: Percentage of the route distance completed by an agent.

  • Infraction penalty: \(\prod_j^{\text{ped., ..., stop}} ({p_i^j})^{\text{#infractions}_j}.\) — The leaderboard tracks several types of infractions and this metric aggregates all of these infractions triggered by an agent as a geometric series. Agents start with an ideal 1.0 base score, which is reduced each type an infraction is commited.

When all routes have been completed, a global metric for each of the previous three types is also generated, being the arithmetic mean of all the individual routes combined. The global driving score is the main metric on which you will be classified with respect to other participants.

Infractions and shutdown events

The CARLA leaderboard offers individual metrics for a series of infractions. Each of these has a penalty coefficient that will be applied everytime it happens. Ordered by severity, the infractions are the following.

  • Collisions with pedestrians — 0.50.
  • Collisions with other vehicles — 0.60.
  • Collisions with static elements — 0.65.
  • Running a red light — 0.70.
  • Running a stop sign — 0.80.

Some scenarios feature behaviors that can block the ego-vehicle indefinitely. These scenarios will have a timeout of 4 minutes after which the ego-vehicle will be released to continue the route. However, a penalty is applied when the time limit is breached:

  • Scenario timeout — 0.7

The agent is expected to maintain a minimum speed in keeping with nearby traffic. The agent’s speed will be compared with the speed of nearby vehicles. Failure to maintain a suitable speed will result in a penalty. The penalty applied is dependent on the magnitude of the speed difference, up to the following value:

  • Failure to maintain minimum speed — 0.7

The agent should yield to emergency vehicles coming from behind. Failure to allow the emergency vehicle to pass will incur a penalty:

  • Failure to yield to emergency vehicle — 0.7

Besides these, there is one additional infraction which has no coefficient, and instead affects the computation of the route completion (\(R_i\)).

  • Off-road driving — If an agent drives off-road, that percentage of the route will not be considered towards the computation of the route completion score.

Additionally, some events will interrupt the simulation, preventing the agent to continue. In these cases, the route which is being simulated will be shut down, and the leaderboard will move onto the next one, triggering it normally.

  • Route deviation — If an agent deviates more than 30 meters from the assigned route.
  • Agent blocked — If an agent doesn’t take any actions for 180 simulation seconds.
  • Simulation timeout — If no client-server communication can be established in 60 seconds.
  • Route timeout — If the simulation of a route takes too long to finish.

Each time any of the above happens, several details are recorded, which will be displayed as a list for you to see at the route’s individual metrics. Below is an example of a route where the agent both run a red light and deviated from the route.

"infractions": {
  "Collisions with layout": [],
  "Collisions with pedestrians": [],
  "Collisions with vehicles": [],
  "Red lights infractions": [
        "Agent ran a red light 203 at (x=341.25, y=209.1, z=0.104)"
  ],
  "Stop sign infractions": [],
  "Off-road infractions": [],
  "Min speed infractions": [],
  "Yield to emergency vehicle infractions": [],
  "Scenario timeouts": [],
  "Route deviations": [
        "Agent deviated from the route at (x=95.92, y=165.673, z=0.138)"
  ],
  "Agent blocked": [],
  "Route timeouts": []
}

The global infractions compress the individual route’s data into a single value and is given as the number of events per Km.

Get started

In order to get familiar with the leaderboard we recommend you to read carefully through the Get started section. Please, spend enough time making sure everything works as expected locally.

Once you are ready, check the Submit section to learn how to prepare your submission.

Leaderboard 1.0

Leaderboard is currently at version 2.0. Leaderboard version 1.0 is still available to support your previous work. Please consult the Get started and Submit pages for Leaderboard version 1.0 submissions.

Sign up!

Terms and Conditions

The CARLA Autonomous Driving Leaderboard is offered for free as a service to the research community thanks to the generosity of our sponsors and collaborators.

Each submission will be evaluated in AWS using a g5.12xlarge instance. This gives users access to a dedicated node with a modern GPU and CPU.

Teams are provided a finite number of submissions (currently 5 submissions) for a given a month.

Submission allowance is automatically refilled every month. The organizers of the CARLA leaderboard reserve the right to assign additional allowances to a team. The organization also reserves the right to modify the default values of the monthly allowance for submissions.

It is strictly prohibited to misuse or attack the infrastructure of the CARLA leaderboard, including all software and hardware that is used to run the service. Actions that deviate from the spirit of the CARLA leaderboard could result in the termination of a team.

For further instructions, please read the terms and conditions.