Submitting an agent

By submitting your agent to the CARLA Autonomous Driving Leaderboard you are accepting the terms of use.

General steps

The instructions below assume that you have already experience making agents using the leaderboard and scenario_runner projects. The leaderboard project is the main engine used to execute and evaluate your agent. The CARLA leaderboard uses a copy of this repository with some modifications to run your code in the cloud. The leaderboard uses scenario_runner as a library to run traffic simulation.

In order to create and submit your agent you should have a copy of the leaderboard project and scenario_runner. If this is not the case, please visit the Get started section first.

1. Create a dockerfile to build your agent

In order for the CARLA leaderboard to evaluate your agent, it needs to be encapsulated in a docker image, following a set of conventions.

  • Create a new directory to host your code. You can call this directory as you prefer, but we typically call it team_agent. For a submission to be valid, it is important to create this directory outside the leaderboard root (${LEADERBOARD_ROOT}). We will refer to this directory as ${TEAM_CODE_ROOT}.

  • Find the dockerfile at ${LEADERBOARD_ROOT}/scripts/Dockerfile.master. This template serves as a starting point for users. You need to include the dependencies and additional packages required by your agent. We recommend you to add the new commands in the area delimited by the tags “BEGINNING OF USER COMMANDS” and “END OF USER COMMANDS”.

  • Update the variable TEAM_AGENT to set your agent file, i.e. the entry file that inherits from AutonomousAgent. Do not change the rest of the path “/workspace/team_code“. If your agent needs a configuration file for initialization, set the variable TEAM_CONFIG to the configuration file.

2. Build your agent into a docker image

Once your dockerfile is ready invoke the make_docker utility as follows:

export TEAM_CODE_ROOT=YOUR_FOLDER; ${LEADERBOARD_ROOT}/scripts/make_docker.sh

This will generate your docker image. Before running make_docker.sh make sure all the required environment variable are correctly defined, i.e. ${CARLA_ROOT}, ${SCENARIO_RUNNER_ROOT}, ${LEADERBOARD_ROOT}, and ${TEAM_CODE_ROOT}. The simplest way to do so is to add them to your ${HOME}/.bashrc file.

export CARLA_ROOT=...
export SCENARIO_RUNNER_ROOT=...
export LEADERBOARD_ROOT=...
export TEAM_CODE_ROOT=...

Feel free to adjust this script to copy additional files and resources as you see it fits.

After this step is completed your docker image should be ready, under the name leaderboard-user.

3. Register a new user at AlphaDrive site

Our partners at AlphaDrive have created the official web UI and the cloud ecosystem that runs the CARLA leaderboard. In order to make a submission you need to sign up at their site.

4. Create a new team

After signing up in AlphaDrive site you will create a new team by clicking in the Teams section as shown below.


Team creation screen in AlphaDrive UI.


Decide the name of the team and make sure to fill in your affiliation. Your team will be then reviewed for approval. If any incidence occurs the administrator will contact you. In the meantime add some colleagues to your team using the menu shown below.


Invite additional team members.

5. Apply to the CARLA benchmark

Go to the benchmark section and click on the CARLA Leaderboard as in the image shown below. Once there, just apply to join the leaderboard.


Benchmark section.

6. Navigate to the Overview and Leaderboard sections

Make sure you familiarize yourself with the UI. In the Overview tab you will find general information about the CARLA leaderboard, the remaining time budget for your team and the remaining submissions. These numbers will automatically reset monthly.

Here you will also find the two available modalities of the leaderboard:

  • SENSORS-test-2020
  • MAP-test-2020

In the Leaderboard tab you will have access to the results obtained for all the submissions along with information about teams, the type of sensors they used for a given submission, and a description of their approach.

7. Install AlphaDrive cli

curl http://dist.alphadrive.ai/install-ubuntu.sh | sh -

This command line tool will be used to submit your docker image and check the status of your submissions.

8. Login to alphadrive in cli

alpha login

A browser will pop up requesting your credentials. Introduce your credentials and continue.

9. Submit your docker image to the CARLA leaderboard

The following command will push your local docker image to the cloud-based leaderboard and will start the execution:

alpha benchmark:submit  --split {2, 3} ${YOUR_DOCKERIMAGE_NAME}

Select between the MAP (split 2) and the SENSOR modality (split 3). By default ${YOUR_DOCKERIMAGE_NAME} will be leaderboard-user:latest.

10. Check the status of a submission

After your submission has started you can keep track of its status by visiting the web UI or running the following command through the cli:

alpha ps

Additionally you may want to check the status of your session using:

alpha session

And if needed kill a given submission running:

alpha session:kill

11. Make your submission public

By default your submission will remain private. In order to display it in the CARLA AD leaderboard you will have to make it public. To do so,

  • Go to your submissions in the AlphaDrive UI.
  • Click on the ID of the submission you want to make public
  • Click on the 3 vertical dots at the top right corner of the submission page
  • Click on “Make public”
  • Fill in the submission form providing all the relevant details needed to identify the submission.

If you are preparing a submission as part of a double-blind publication, remember to check the “Anonymous submission” box. This will anonymize your submission removing authors’ details.

12. Congratulations, your submission is now on the CARLA Leaderboard!