leaderboard.carla.org Open in urlscan Pro
2606:50c0:8002::153  Public Scan

Form analysis
0 forms found in the DOM

Text Content

SKIP LINKS

 * Skip to primary navigation
 * Skip to content
 * Skip to footer

CARLA Autonomous Driving Leaderboard
 * Home
 * Get started
 * Submit
 * Leaderboard
 * Challenge
 * About
 * Help

Toggle menu










The information on this page concerns Leaderboard 2.0, the latest version. If
you are using the previous version of leaderboard please consult the Leaderboard
1.0 section.


NEWSPERMALINK


2024 CVPR AUTONOMOUS GRAND CHALLENGEPERMALINK

This year, we are thrilled to announce that the CARLA AD Leaderboard is part of
this year’s CVPR Autonomous Grand Challenge! Showcase your AV development skills
and compete for a remarkable prize of $10,000 or an HP workstation! Check out
the CVPR Autonomous Grand Challenge website.

Teams participating in the CVPR challenge will be required to complete two
qualifying rounds, ensuring their agents are adept at basic navigation. Once
qualified, you’ll gain access to submit agents for the complete Leaderboard 2.0
route selection. The Leaderboard 2.0 now allows double the number of sensors
compared to previous competitions, including 8 RGB cameras, 2 LIDARs and 4
RADARs.

Please note: In preparation for the competition, we’ll be temporarily closing
Leaderboard 1.0. Leaderboard 2.0 will be reopened for submissions on March 25th.

Remember to join our Discord server to be up to date with all the Leaderboard
related news, and feel free to ask any questions about the Leaderboard in the
CARLA AD Leaderboard channel, or to any of the CARLA team members.

--------------------------------------------------------------------------------


OVERVIEWPERMALINK



The main goal of the CARLA Autonomous Driving Leaderboard is to evaluate the
driving proficiency of autonomous agents in realistic traffic scenarios. The
leaderboard serves as an open platform for the community to perform fair and
reproducible evaluations of autonomous vehicle agents, simplifying the
comparison between different approaches. Leaderboard is currently at version
2.0, version 1.0 is still supported.

--------------------------------------------------------------------------------

INDEX

 * Overview
 * Task
 * Participation modalities
    * SENSORS track
    * MAP track

 * Evaluation and metrics
    * Infractions
    * Additional events

 * Get started
 * Leaderboard 1.0
 * Terms and conditions

--------------------------------------------------------------------------------


TASKPERMALINK

The CARLA AD Leaderboard challenges AD agents to drive through a set of
predefined routes. For each route, agents will be initialized at a starting
point and directed to drive to a destination point, provided with a description
of the route through GPS style coordinates, map coordinates or route
instructions. Routes are defined in a variety of situations, including freeways,
urban areas, residential districts and rural settings. The Leaderboard evaluates
AD agents in a variety of weather conditions, including daylight scenes, sunset,
rain, fog, and night, among others.






SCENARIOSPERMALINK

Agents will face multiple traffic scenarios based on the NHTSA typology. The
full list of traffic scenarios can be reviewed in this page, but here are some
examples.

 * Lane merging.
 * Lane changing.
 * Negotiations at traffic intersections.
 * Negotiations at roundabouts.
 * Handling traffic lights and traffic signs.
 * Yielding to emergency vehicles.
 * Coping with pedestrians, cyclists, and other elements.

Illustration of traffic situations present in the CARLA AD leaderboard.

--------------------------------------------------------------------------------


PARTICIPATION MODALITIESPERMALINK

The leaderboard offers two participation modalities, SENSORS and MAP. These
modalities differ in the type of input data that your agent can request from the
platform.

Additionally, agents in both modalities will receive a high-level route
description indicating the key points that the path has to follow in order to
reach the destination. The route is represented as a list of tuples and has two
variations.

For the first case, the first element of the tuple contains a waypoint,
expressed as a latitude, a longitude, and a z component.

[({'z': 0.0, 'lat': 48.99822669411668, 'lon': 8.002271601998707}, RoadOption.LEFT),
 ({'z': 0.0, 'lat': 48.99822669411668, 'lon': 8.002709765148996}, RoadOption.RIGHT),
 ...
 ({'z': 0.0, 'lat': 48.99822679980298, 'lon': 8.002735250105061}, RoadOption.STRAIGHT)]


The second case is very similar to the previous one, but instead of using GPS
coordinates, the route is expressed in world coordinates instead.

[({'x': 153.7, 'y': 15.6, 'z': 0.0}, RoadOption.LEFT),
 ({'x': 148.9, 'y': 67.8, 'z': 0.0}, RoadOption.RIGHT),
 ...
 ({'x': 180.7, 'y': 45.1, 'z': 1.2}, RoadOption.STRAIGHT)]


The distance between two consecutive waypoints could be up to hundreds of
meters. Do not rely on these as your principal mechanism to navigate the
environment.

The second element contains a high-level command. The set of available
high-level commands is:

 * RoadOption.CHANGELANELEFT: Move one lane to the left.
 * RoadOption.CHANGELANERIGHT: Move one lane to the right.
 * RoadOption.LANEFOLLOW: Continue in the current lane.
 * RoadOption.LEFT: Turn left at the intersection.
 * RoadOption.RIGHT: Turn right at the intersection.
 * RoadOption.STRAIGHT: Keep straight at the intersection.

There might be cases where the semantics of left and right is ambiguous. In
order to disambiguate these situations, you could consider the GPS position of
the next waypoints.

Important: You are not allowed to make use of any privilege information offered
by the CARLA simulator, including planners or any type of ground truth.
Submissions using these features will be rejected and teams will be banned from
the platform.


SENSORS TRACKPERMALINK

On this track agents can request access to the following sensors.


GNSS
IMU
LIDAR
RADAR
RGB camera
Speedometer sensor.other.gnss sensor.other.imu sensor.lidar.ray_cast
sensor.other.radar sensor.camera.rgb sensor.other.speedometer 0-1 units 0-1
units 0-2 units 0-4 units 0-8 units 0-1 units GPS sensor returning geo location
data. 6-axis Inertial Measurement Unit. Velodyne 64 LIDAR. Long-range RADAR (up
to 100 meters). Regular camera that captures images. Pseudosensor that provides
an approximation of your linear velocity.

Units of each sensor are limited to keep the computational budget under control.


MAP TRACKPERMALINK

Provides the same set of sensor as the SENSORS track does. Additionally, agents
can request to access an HD map, which is provided as an OpenDRIVE file parsed
as a string.

You are fully responsible to parse or convert this file into a representation
that can be useful to your agent.


GNSS
IMU
LIDAR
RADAR
RGB camera
Speedometer sensor.other.gnss sensor.other.imu sensor.lidar.ray_cast
sensor.other.radar sensor.camera.rgb sensor.other.speedometer 0-1 units 0-1
units 0-2 units 0-4 units 0-8 units 0-1 units GPS sensor returning geo location
data. 6-axis Inertial Measurement Unit. Velodyne 64 LIDAR. Long-range RADAR (up
to 100 meters). Regular camera that captures images. Pseudosensor that provides
an approximation of your linear velocity.


OpenDRIVE map sensor.opendrive_map 0-1 unit Pseudosensor that exposes the HD map
in OpenDRIVE format parsed as a string.

--------------------------------------------------------------------------------


QUALIFYING TRACKSPERMALINK

The Leaderboard 2.0 features 2 qualifying tracks in which your agent will be
required to safely navigate a straightforward route through an unseen map
featuring simple driving scenarios. Each of the SENSOR and MAP tracks has its
own respective qualifier. You must reach a minimum driving score in the
qualifiers to be permitted to submit agents to the Leaderboard 2.0 main tracks.

To save computing resources the qualifiers will permit a more restrictive sensor
suite than the main competition, with 0-4 RGB cameras, 0-2 RADARs and 0-1
LIDARs, along with GNSS, IMU and speedometer (and OpenDRIVE for the MAP track).
We encourage you to use the qualifier to ensure that your AD stack is properly
configured and functioning correctly with the Leaderboard.

--------------------------------------------------------------------------------


EVALUATION AND METRICSPERMALINK

The driving proficiency of an agent can be characterized by multiple metrics.
For this leaderboard we have selected a set of metrics that help understand
different aspects of driving. While all routes have the same type of metrics,
their respective values are calculated separately. The specific metrics are as
follows:

 * Driving score: RiPi,RiPi, — Main metric of the leaderboard, serving as the
   product between the route completion and the infractions penalty. Here RiRi
   is the percentage of completion of the i−thi−th route, and PiPi, the
   infraction penalty of the i−thi−th route.

 * Route completion: Percentage of the route distance completed by an agent.

 * Infraction penalty: ∏ped., ..., stopj(pji)#infractionsj.∏jped., ...,
   stop(pij)#infractionsj. — The leaderboard tracks several types of infractions
   and this metric aggregates all of these infractions triggered by an agent as
   a geometric series. Agents start with an ideal 1.0 base score, which is
   reduced each type an infraction is commited.

When all routes have been completed, a global metric for each of the previous
three types is also generated, being the arithmetic mean of all the individual
routes combined. The global driving score is the main metric on which you will
be classified with respect to other participants.


INFRACTIONS AND SHUTDOWN EVENTSPERMALINK

The CARLA leaderboard offers individual metrics for a series of infractions.
Each of these has a penalty coefficient that will be applied everytime it
happens. Ordered by severity, the infractions are the following.

 * Collisions with pedestrians — 0.50.
 * Collisions with other vehicles — 0.60.
 * Collisions with static elements — 0.65.
 * Running a red light — 0.70.
 * Running a stop sign — 0.80.

Some scenarios feature behaviors that can block the ego-vehicle indefinitely.
These scenarios will have a timeout of 4 minutes after which the ego-vehicle
will be released to continue the route. However, a penalty is applied when the
time limit is breached:

 * Scenario timeout — 0.7

The agent is expected to maintain a minimum speed in keeping with nearby
traffic. The agent’s speed will be compared with the speed of nearby vehicles.
Failure to maintain a suitable speed will result in a penalty. The penalty
applied is dependent on the magnitude of the speed difference, up to the
following value:

 * Failure to maintain minimum speed — 0.7

The agent should yield to emergency vehicles coming from behind. Failure to
allow the emergency vehicle to pass will incur a penalty:

 * Failure to yield to emergency vehicle — 0.7

Besides these, there is one additional infraction which has no coefficient, and
instead affects the computation of the route completion (RiRi).

 * Off-road driving — If an agent drives off-road, that percentage of the route
   will not be considered towards the computation of the route completion score.

Additionally, some events will interrupt the simulation, preventing the agent to
continue. In these cases, the route which is being simulated will be shut down,
and the leaderboard will move onto the next one, triggering it normally.

 * Route deviation — If an agent deviates more than 30 meters from the assigned
   route.
 * Agent blocked — If an agent doesn’t take any actions for 180 simulation
   seconds.
 * Simulation timeout — If no client-server communication can be established in
   60 seconds.
 * Route timeout — If the simulation of a route takes too long to finish.

Each time any of the above happens, several details are recorded, which will be
displayed as a list for you to see at the route’s individual metrics. Below is
an example of a route where the agent both run a red light and deviated from the
route.

"infractions": {
  "Collisions with layout": [],
  "Collisions with pedestrians": [],
  "Collisions with vehicles": [],
  "Red lights infractions": [
        "Agent ran a red light 203 at (x=341.25, y=209.1, z=0.104)"
  ],
  "Stop sign infractions": [],
  "Off-road infractions": [],
  "Min speed infractions": [],
  "Yield to emergency vehicle infractions": [],
  "Scenario timeouts": [],
  "Route deviations": [
        "Agent deviated from the route at (x=95.92, y=165.673, z=0.138)"
  ],
  "Agent blocked": [],
  "Route timeouts": []
}


The global infractions compress the individual route’s data into a single value
and is given as the number of events per Km.

--------------------------------------------------------------------------------


GET STARTEDPERMALINK

In order to get familiar with the leaderboard we recommend you to read carefully
through the Get started section. Please, spend enough time making sure
everything works as expected locally.

Once you are ready, check the Submit section to learn how to prepare your
submission.

--------------------------------------------------------------------------------


LEADERBOARD 1.0PERMALINK

Leaderboard is currently at version 2.0. Leaderboard version 1.0 is still
available to support your previous work. Please consult the Get started and
Submit pages for Leaderboard version 1.0 submissions.

Sign up!

--------------------------------------------------------------------------------


TERMS AND CONDITIONSPERMALINK

The CARLA Autonomous Driving Leaderboard is offered for free as a service to the
research community thanks to the generosity of our sponsors and collaborators.

Each submission will be evaluated in AWS using a g5.12xlarge instance. This
gives users access to a dedicated node with a modern GPU and CPU.

Teams are provided a finite number of submissions (currently 5 submissions) for
a given a month.

Submission allowance is automatically refilled every month. The organizers of
the CARLA leaderboard reserve the right to assign additional allowances to a
team. The organization also reserves the right to modify the default values of
the monthly allowance for submissions.

It is strictly prohibited to misuse or attack the infrastructure of the CARLA
leaderboard, including all software and hardware that is used to run the
service. Actions that deviate from the spirit of the CARLA leaderboard could
result in the termination of a team.

For further instructions, please read the terms and conditions.

   
 * 
 * Social:
 * Twitter
 * LinkedIn
 * GitHub

   
   
 * 
 * Sponsors:
 * 
 * 
 * 
 * 
 * 
 * 

© 2024 CARLA Autonomous Driving Leaderboard.