Submitting an agent to Leaderboard 1.0

This page covers submitting an agent to Leaderboard version 1.0. If you are intending to use the latest Leaderboard version 2.0, please go to this Submit guide.

By submitting your agent to the CARLA Autonomous Driving Leaderboard you are accepting the terms of use.

General steps

In order to create and submit your agent you should have a copy of the leaderboard project and scenario_runner. If this is not the case, please visit the Get started section first.

Index

1. Create a dockerfile to build your agent

In order for the CARLA leaderboard to evaluate your agent, it needs to be encapsulated within a docker image. For a successful one, you will have to be able to run your agent with the leaderboard inside your docker. You can either create your image from scratch, or follow the guidelines below:

  • Create a new directory to host your code. For a submission to be valid, it is necessary to create this directory outside the ${LEADERBOARD_ROOT}. We will refer to this directory as ${TEAM_CODE_ROOT}.

  • Find the dockerfile at ${LEADERBOARD_ROOT}/scripts/Dockerfile.master. This template serves as a starting point for users. Here all the dependecies needed for scenario runner and leaderboard have already been set up. Include the dependencies and additional packages required by your agent. We recommend you add the new commands in the area delimited by the tags “BEGINNING OF USER COMMANDS” and “END OF USER COMMANDS”.

  • Update the variable TEAM_AGENT to set your agent file, i.e. the entry file that inherits from AutonomousAgent. Do not change the rest of the path “/workspace/team_code“. If your agent needs a configuration file for initialization, set the variable TEAM_CONFIG to the configuration file.

  • Make sure anything you want to source is inside ${HOME}/agent_sources.sh as sources added anywhere else will not be invoked. This file is automatically sourced before running the agent in the cloud.

Please take note that any changes you make to the Leaderboard or Scenario Runner repositories will be overwritten when you submit your entry to the cloud.

1.1 ROS Stacks

As stated above, all sources have to be inside ${HOME}/agent_sources.sh. In order to source ROS, you can add the following command to your Dockerfile, which creates the file and sources the ROS environment:

RUN /bin/echo \
  'source /opt/ros/$ROS_DISTRO/setup.bash; \
  source $TEAM_CODE_ROOT/install/setup.bash;' >> ${HOME}/agent_sources.sh

This is a template to source the ROS environment and your agent. Feel free to modify it in case your setup is different

2. Build your agent into a docker image

Once the dockerfile is ready, make sure that all the needed environment variables are correctly defined, i.e. ${CARLA_ROOT}, ${SCENARIO_RUNNER_ROOT}, ${LEADERBOARD_ROOT}, and ${TEAM_CODE_ROOT}. The simplest way to do so is to add them to your ${HOME}/.bashrc file.

export CARLA_ROOT=...
export SCENARIO_RUNNER_ROOT=...
export LEADERBOARD_ROOT=...
export TEAM_CODE_ROOT=...

Feel free to adjust this script to copy additional files and resources as you see fit.

Then, invoke the make_docker.sh utility as follows:

${LEADERBOARD_ROOT}/scripts/make_docker.sh

This will generate your docker image, creating the docker image under the name leaderboard-user.

You can test your docker image by running the leaderboard locally at your computer

3. Register a new user at AlphaDrive site

Our partners at AlphaDrive have created the official web UI and the cloud ecosystem that runs the CARLA leaderboard. In order to make a submission you need to sign up at their site.

4. Create a new team

After signing up in AlphaDrive site you will create a new team by clicking in the Teams section as shown below.


Team creation screen in AlphaDrive UI.


Decide the name of the team and make sure to fill in your affiliation, otherwise it will not be accepted. Your team will be then reviewed for approval. If any incidence occurs the administrator will contact you. In the meantime add some colleagues to your team using the menu shown below.


Invite additional team members.

5. Apply to the CARLA benchmark

Go to the benchmark section and click on the CARLA Leaderboard as in the image shown below. Once there, just apply to join the leaderboard.


Benchmark section.

6. Navigate to the Overview and Leaderboard sections

Make sure you familiarize yourself with the UI. In the Overview tab you will find general information about the CARLA leaderboard, the remaining time budget for your team and the remaining submissions. These numbers will automatically reset monthly.

Here you will also find the two available modalities of the leaderboard:

  • SENSORS-test-2020
  • MAP-test-2020

In the Leaderboard tab you will have access to the results obtained for all the submissions along with information about teams, the type of sensors they used for a given submission, and a description of their approach.

7. Install AlphaDrive cli

curl http://dist.alphadrive.ai/install-ubuntu.sh | sh -

This command line tool will be used to submit your docker image and check the status of your submissions.

8. Login to alphadrive in cli

alpha login

A browser will pop up requesting your credentials. Introduce your credentials and continue.

9. Submit your docker image to the CARLA leaderboard

The following command will push your local docker image to the cloud-based leaderboard and will start the execution:

alpha benchmark:submit  --split {2, 3} ${YOUR_DOCKERIMAGE_NAME}

Select between the MAP (split 2) and the SENSOR modality (split 3). By default ${YOUR_DOCKERIMAGE_NAME} will be leaderboard-user:latest.

Only one submission can be active at the same time, so make sure that none are

10. Useful cli commands

While the AlphaDrive web UI already provides several tools to interact with your sessions, here are several functionalities that can be used through the cli. You can keep track of your submission’s status with:

alpha ps

Additionally, you may want to check the status of your session using:

alpha session

If needed, you can also kill a given submission by running:

alpha session:kill

Previosuly submitted sessions can also be restarted with:

alpha benchmark:restart SESSIONID

# Or use --resume to restart a session from the last checkpoint
alpha benchmark:restart --resume SESSIONID

11. Make your submission public

By default your submission will remain private. In order to display it in the CARLA AD leaderboard you will have to make it public. To do so,

  • Go to your submissions in the AlphaDrive UI.
  • Click on the ID of the submission you want to make public
  • Click on the 3 vertical dots at the top right corner of the submission page
  • Click on “Make public”
  • Fill in the submission form providing all the relevant details needed to identify the submission.

If you are preparing a submission as part of a double-blind publication, remember to check the “Anonymous submission” box. This will anonymize your submission removing authors’ details.

12. Congratulations, your submission is now on the CARLA Leaderboard!