Operations

How to run

We provide various basic operations allowing users to modify the built database for ML applications. These operations include building database from different data providers;aggregating datasets from diverse source; splitting datasets to training/test set;sanity check/filtering scenarios. All commands can be run with python -m scenarionet.[command], e.g. python -m scenarionet.list for listing available operations. The parameters for each script can be found by adding a -h flag.

Note

When running python -m, make sure the directory you are at doesn’t contain a folder called scenarionet. Otherwise, the running may fail. This usually happens if you install ScenarioNet or MetaDrive via git clone and put it under a directory you usually work with like home directory.

List

This command can list all operations with detailed descriptions:

python -m scenarionet.list

Convert

ScenarioNet doesn’t provide any data. Instead, it provides converters to parse common open-sourced driving datasets to an internal scenario description, which comprises scenario databases. Thus converting scenarios to our internal scenario description is the first step to build the databases. Currently,we provide convertors for Waymo, nuPlan, nuScenes (Lyft) datasets.

Convert Waymo

python -m scenarionet.convert_waymo [-h] [--database_path DATABASE_PATH]
                        [--dataset_name DATASET_NAME] [--version VERSION]
                        [--overwrite] [--num_workers NUM_WORKERS]
                        [--raw_data_path RAW_DATA_PATH]
                        [--start_file_index START_FILE_INDEX]
                        [--num_files NUM_FILES]

Build database from Waymo scenarios

optional arguments:
  -h, --help            show this help message and exit
  --database_path DATABASE_PATH, -d DATABASE_PATH
                        A directory, the path to place the converted data
  --dataset_name DATASET_NAME, -n DATASET_NAME
                        Dataset name, will be used to generate scenario files
  --version VERSION, -v VERSION
                        version
  --overwrite           If the database_path exists, whether to overwrite it
  --num_workers NUM_WORKERS
                        number of workers to use
  --raw_data_path RAW_DATA_PATH
                        The directory stores all waymo tfrecord
  --start_file_index START_FILE_INDEX
                        Control how many files to use. We will list all files
                        in the raw data folder and select
                        files[start_file_index: start_file_index+num_files]
  --num_files NUM_FILES
                        Control how many files to use. We will list all files
                        in the raw data folder and select
                        files[start_file_index: start_file_index+num_files]

This script converted the recorded scenario into our scenario descriptions. Detailed guide is available at Section Waymo.

Convert nuPlan

python -m scenarionet.convert_nuplan [-h] [--database_path DATABASE_PATH]
                     [--dataset_name DATASET_NAME] [--version VERSION]
                     [--overwrite] [--num_workers NUM_WORKERS]
                     [--raw_data_path RAW_DATA_PATH] [--test]

Build database from nuPlan scenarios

optional arguments:
  -h, --help            show this help message and exit
  --database_path DATABASE_PATH, -d DATABASE_PATH
                        A directory, the path to place the data
  --dataset_name DATASET_NAME, -n DATASET_NAME
                        Dataset name, will be used to generate scenario files
  --version VERSION, -v VERSION
                        version of the raw data
  --overwrite           If the database_path exists, whether to overwrite it
  --num_workers NUM_WORKERS
                        number of workers to use
  --raw_data_path RAW_DATA_PATH
                        the place store .db files
  --test                for test use only. convert one log

This script converted the recorded nuPlan scenario into our scenario descriptions. It needs to install nuplan-devkit and download the source data from https://www.nuscenes.org/nuplan. Detailed guide is available at Section nuPlan.

Convert nuScenes (Lyft)

python -m scenarionet.convert_nuscenes [-h] [--database_path DATABASE_PATH]
                           [--dataset_name DATASET_NAME]
                           [--split
{v1.0-mini,mini_val,v1.0-test,train,train_val,val,mini_train,v1.0-trainval}]
                           [--dataroot DATAROOT] [--map_radius MAP_RADIUS]
                           [--future FUTURE] [--past PAST] [--overwrite]
                           [--num_workers NUM_WORKERS]

Build database from nuScenes/Lyft scenarios

optional arguments:
  -h, --help            show this help message and exit
  --database_path DATABASE_PATH, -d DATABASE_PATH
                        directory, The path to place the data
  --dataset_name DATASET_NAME, -n DATASET_NAME
                        Dataset name, will be used to generate scenario files
  --split
    {v1.0-mini,mini_val,v1.0-test,train,train_val,val,mini_train,v1.0-trainval}
                        Which splits of nuScenes data should be sued. If set
                        to ['v1.0-mini', 'v1.0-trainval', 'v1.0-test'], it
                        will convert the full log into scenarios with 20
                        second episode length. If set to ['mini_train',
                        'mini_val', 'train', 'train_val', 'val'], it will
                        convert segments used for nuScenes prediction
                        challenge to scenarios, resulting in more converted
                        scenarios. Generally, you should choose this parameter
                        from ['v1.0-mini', 'v1.0-trainval', 'v1.0-test'] to
                        get complete scenarios for planning unless you want to
                        use the converted scenario files for prediction task.
  --dataroot DATAROOT   The path of nuscenes data
  --map_radius MAP_RADIUS The size of map
  --future FUTURE       6 seconds by default. How many future seconds to
                        predict. Only available if split is chosen from
                        ['mini_train', 'mini_val', 'train', 'train_val',
                        'val']
  --past PAST           2 seconds by default. How many past seconds are used
                        for prediction. Only available if split is chosen from
                        ['mini_train', 'mini_val', 'train', 'train_val',
                        'val']
  --overwrite           If the database_path exists, whether to overwrite it
  --num_workers NUM_WORKERS

This script converted the recorded nuScenes scenario into our scenario descriptions. It needs to install nuscenes-devkit and download the source data from https://www.nuscenes.org/nuscenes. For Lyft datasets, this API can only convert the old version Lyft data as the old Lyft data can be parsed via nuscenes-devkit. However, Lyft is now a part of Woven Planet and the new data has to be parsed via new toolkit. We are working on support this new toolkit to support the new Lyft dataset. Detailed guide is available at Section nuScenes.

Convert PG

python -m scenarionet.convert_pg [-h] [--database_path DATABASE_PATH]
                     [--dataset_name DATASET_NAME] [--version VERSION]
                     [--overwrite] [--num_workers NUM_WORKERS]
                     [--num_scenarios NUM_SCENARIOS]
                     [--start_index START_INDEX]

Build database from synthetic or procedurally generated scenarios

optional arguments:
  -h, --help            show this help message and exit
  --database_path DATABASE_PATH, -d DATABASE_PATH
                        directory, The path to place the data
  --dataset_name DATASET_NAME, -n DATASET_NAME
                        Dataset name, will be used to generate scenario files
  --version VERSION, -v VERSION
                        version
  --overwrite           If the database_path exists, whether to overwrite it
  --num_workers NUM_WORKERS
                        number of workers to use
  --num_scenarios NUM_SCENARIOS
                        how many scenarios to generate (default: 30)
  --start_index START_INDEX
                        which index to start

PG refers to Procedural Generation. Scenario database generated in this way are created by a set of rules with hand-crafted maps. These scenarios are collected by driving the ego car with an IDM policy in different scenarios. Detailed guide is available at Section PG.

Merge

This command is for merging existing databases to build a larger one. This is why we can build a ScenarioNet! After converting data recorded in different format to this unified scenario description, we can aggregate them freely and enlarge the database.

python -m scenarionet.merge [-h] --to DATABASE_PATH --from FROM [FROM ...]
                [--exist_ok] [--overwrite] [--filter_moving_dist]
                [--sdc_moving_dist_min SDC_MOVING_DIST_MIN]

Merge a list of databases. e.g. scenario.merge --from db_1 db_2 db_3...db_n
--to db_dest

optional arguments:
-h, --help            show this help message and exit
--database_path DATABASE_PATH, -d DATABASE_PATH, --to DATABASE_PATH
                    The name of the new combined database. It will create
                    a new directory to store dataset_summary.pkl and
                    dataset_mapping.pkl. If exists_ok=True, those two .pkl
                    files will be stored in an existing directory and turn
                    that directory into a database.
--from FROM [FROM ...]
                    Which datasets to combine. It takes any number of
                    directory path as input
--exist_ok            Still allow to write, if the dir exists already. This
                    write will only create two .pkl files and this
                    directory will become a database.
--overwrite           When exists ok is set but summary.pkl and map.pkl
                    exists in existing dir, whether to overwrite both
                    files
--filter_moving_dist  add this flag to select cases with SDC moving dist >
                    sdc_moving_dist_min
--sdc_moving_dist_min SDC_MOVING_DIST_MIN
                    Selecting case with sdc_moving_dist > this value. We
                    will add more filter conditions in the future.

Split

The split action is for extracting a part of scenarios from an existing one and building a new database. This is usually used to build training/test/validation set.

python -m scenarionet.split [-h] --from FROM --to TO [--num_scenarios NUM_SCENARIOS]
            [--start_index START_INDEX] [--random] [--exist_ok]
            [--overwrite]

Build a new database containing a subset of scenarios from an existing
database.

optional arguments:
  -h, --help            show this help message and exit
  --from FROM           Which database to extract data from.
  --to TO               The name of the new database. It will create a new
                        directory to store dataset_summary.pkl and
                        dataset_mapping.pkl. If exists_ok=True, those two .pkl
                        files will be stored in an existing directory and turn
                        that directory into a database.
  --num_scenarios NUM_SCENARIOS
                        how many scenarios to extract (default: 30)
  --start_index START_INDEX
                        which index to start
  --random              If set to true, it will choose scenarios randomly from
                        all_scenarios[start_index:]. Otherwise, the scenarios
                        will be selected sequentially
  --exist_ok            Still allow to write, if the to_folder exists already.
                        This write will only create two .pkl files and this
                        directory will become a database.
  --overwrite           When exists ok is set but summary.pkl and map.pkl
                        exists in existing dir, whether to overwrite both
                        files

Copy (Move)

As the the database built by ScenarioNet stores the scenarios with virtual mapping, directly move or copy an existing database to a new location with cp or mv command will break the soft link. For moving or copying the scenarios to a new path, one should use this command. When --remove_source is added, this copy command will be changed to move.

python -m scenarionet.cp [-h] --from FROM --to TO [--remove_source] [--copy_raw_data]
               [--exist_ok] [--overwrite]

Move or Copy an existing database

optional arguments:
  -h, --help       show this help message and exit
  --from FROM      Which database to move.
  --to TO          The name of the new database. It will create a new
                   directory to store dataset_summary.pkl and
                   dataset_mapping.pkl. If exists_ok=True, those two .pkl
                   files will be stored in an existing directory and turn that
                   directory into a database.
  --remove_source  Remove the `from_database` if set this flag
  --copy_raw_data  Instead of creating virtual file mapping, copy raw
                   scenario.pkl file
  --exist_ok       Still allow to write, if the to_folder exists already. This
                   write will only create two .pkl files and this directory
                   will become a database.
  --overwrite      When exists ok is set but summary.pkl and map.pkl exists in
                   existing dir, whether to overwrite both files

Num

Report the number of scenarios in a database.

python -m scenarionet.num [-h] --database_path DATABASE_PATH

The number of scenarios in the specified database

optional arguments:
  -h, --help            show this help message and exit
  --database_path DATABASE_PATH, -d DATABASE_PATH
                        Database to check number of scenarios

Filter

Some scenarios contain overpasses, short ego-car trajectory or traffic signals. This scenarios can be filtered out from the database by using this command. Now, we only provide filters for ego car moving distance, number of objects, traffic lights, overpasses and scenario ids. If you would like to contribute new filters, feel free to create an issue or pull request on our Github repo.

python -m scenarionet.filter [-h] --database_path DATABASE_PATH --from FROM
                      [--exist_ok] [--overwrite] [--moving_dist]
                      [--sdc_moving_dist_min SDC_MOVING_DIST_MIN]
                      [--num_object] [--max_num_object MAX_NUM_OBJECT]
                      [--no_overpass] [--no_traffic_light] [--id_filter]
                      [--exclude_ids EXCLUDE_IDS [EXCLUDE_IDS ...]]

Filter unwanted scenarios out and build a new database

optional arguments:
  -h, --help            show this help message and exit
  --database_path DATABASE_PATH, -d DATABASE_PATH
                        The name of the new database. It will create a new
                        directory to store dataset_summary.pkl and
                        dataset_mapping.pkl. If exists_ok=True, those two .pkl
                        files will be stored in an existing directory and turn
                        that directory into a database.
  --from FROM           Which dataset to filter. It takes one directory path
                        as input
  --exist_ok            Still allow to write, if the dir exists already. This
                        write will only create two .pkl files and this
                        directory will become a database.
  --overwrite           When exists ok is set but summary.pkl and map.pkl
                        exists in existing dir, whether to overwrite both
                        files
  --moving_dist         add this flag to select cases with SDC moving dist >
                        sdc_moving_dist_min
  --sdc_moving_dist_min SDC_MOVING_DIST_MIN
                        Selecting case with sdc_moving_dist > this value.
  --num_object          add this flag to select cases with object_num <
                        max_num_object
  --max_num_object MAX_NUM_OBJECT
                        case will be selected if num_obj < this argument
  --no_overpass         Scenarios with overpass WON'T be selected
  --no_traffic_light    Scenarios with traffic light WON'T be selected
  --id_filter           Scenarios with indicated name will NOT be selected
  --exclude_ids EXCLUDE_IDS [EXCLUDE_IDS ...]
                        Scenarios with indicated name will NOT be selected

Build from Errors

This script is for generating a new database to exclude (include) broken scenarios. This is useful for debugging broken scenarios or building a completely clean datasets for training or testing.

python -m scenarionet.generate_from_error_file [-h] --database_path DATABASE_PATH --file
                               FILE [--overwrite] [--broken]

Generate a new database excluding or only including the failed scenarios
detected by 'check_simulation' and 'check_existence'

optional arguments:
  -h, --help            show this help message and exit
  --database_path DATABASE_PATH, -d DATABASE_PATH
                        The path of the newly generated database
  --file FILE, -f FILE  The path of the error file, should be xyz.json
  --overwrite           If the database_path exists, overwrite it
  --broken              By default, only successful scenarios will be picked
                        to build the new database. If turn on this flog, it
                        will generate database containing only broken
                        scenarios.

Sim

Load a database to simulator and replay the scenarios. We provide different render mode allows users to visualize them. For more details of simulation, please check Section Simulation or the MetaDrive document.

python -m scenarionet.sim [-h] --database_path DATABASE_PATH
          [--render {none,2D,3D,advanced,semantic}]
          [--scenario_index SCENARIO_INDEX]

Load a database to simulator and replay scenarios

optional arguments:
  -h, --help            show this help message and exit
  --database_path DATABASE_PATH, -d DATABASE_PATH
                        The path of the database
  --render {none,2D,3D,advanced,semantic}
  --scenario_index SCENARIO_INDEX
                        Specifying a scenario to run

Check Existence

We provide a tool to check if the scenarios in a database are runnable and exist on your machine. This is because we include the scenarios to a database, a folder, through a virtual mapping. Each database only records the path of each scenario relative to the database directory. Thus this script is for making sure all original scenario file exists and can be loaded.

If it manages to find some broken scenarios, an error file will be generated to the specified path. By using generate_from_error_file, a new database can be created to exclude or only include these broken scenarios. In this way, we can debug the broken scenarios to check what causes the error or just ignore and remove the broke scenarios to make the database intact.

python -m scenarionet.check_existence [-h] --database_path DATABASE_PATH
                          [--error_file_path ERROR_FILE_PATH] [--overwrite]
                          [--num_workers NUM_WORKERS] [--random_drop]

Check if the database is intact and all scenarios can be found and recorded in
internal scenario description

optional arguments:
  -h, --help            show this help message and exit
  --database_path DATABASE_PATH, -d DATABASE_PATH
                        Dataset path, a directory containing summary.pkl and
                        mapping.pkl
  --error_file_path ERROR_FILE_PATH
                        Where to save the error file. One can generate a new
                        database excluding or only including the failed
                        scenarios.For more details, see operation
                        'generate_from_error_file'
  --overwrite           If an error file already exists in error_file_path,
                        whether to overwrite it
  --num_workers NUM_WORKERS
                        number of workers to use
  --random_drop         Randomly make some scenarios fail. for test only!

Check Simulation

This is a upgraded version of existence check. It not only detect the existence and the completeness of the database, but check whether all scenarios can be loaded and run in the simulator.

python -m scenarionet.check_simulation [-h] --database_path DATABASE_PATH
                       [--error_file_path ERROR_FILE_PATH] [--overwrite]
                       [--num_workers NUM_WORKERS] [--random_drop]

Check if all scenarios can be simulated in simulator. We recommend doing this
before close-loop training/testing

optional arguments:
  -h, --help            show this help message and exit
  --database_path DATABASE_PATH, -d DATABASE_PATH
                        Dataset path, a directory containing summary.pkl and
                        mapping.pkl
  --error_file_path ERROR_FILE_PATH
                        Where to save the error file. One can generate a new
                        database excluding or only including the failed
                        scenarios.For more details, see operation
                        'generate_from_error_file'
  --overwrite           If an error file already exists in error_file_path,
                        whether to overwrite it
  --num_workers NUM_WORKERS
                        number of workers to use
  --random_drop         Randomly make some scenarios fail. for test only!

Check Overlap

This script is for checking if there are some overlaps between two databases. The main goal of this command is to ensure that the training and test sets are isolated.

python -m scenarionet.check_overlap [-h] --d_1 D_1 --d_2 D_2 [--show_id]

Check if there are overlapped scenarios between two databases. If so, return
the number of overlapped scenarios and id list

optional arguments:
  -h, --help  show this help message and exit
  --d_1 D_1   The path of the first database
  --d_2 D_2   The path of the second database
  --show_id   whether to show the id of overlapped scenarios