SC2InfoExtractorGo

A tool meant to allow for quick data extraction from StarCraft 2 replay files ".SC2Replay".

Example Usage (Docker)

Note

This steps assumes that you have Docker installed on your machine and working. Additionally, you will either need to pull the docker image from DockerHub or build it locally as described in Build Docker Images.

The easiest way to run this tool is to use the provided Docker image release. Please note that the dependencies directory can be mounted to see what dependency files are downloaded, and what dependency files were pre-seeded with the Docker Image. Other command line options are described in CLI Options.

Commands below showcases the exampe usage of the tool. Steps:

1. Pull the Docker image from DockerHub or Build Docker Images:

docker pull kaszanas/sc2infoextractorgo:latest

2. Place your .SC2Replay files in the directory which you want to mount as the input directory. The default directory for replays within the container is /app/replays/input.
3. Pick one of the following commands.
   • Run the docker container and mount the entire directory:

   docker run -it --rm \
   -v </path/to/your/directory>:/app \
   sc2infoextractorgo:latest -help

   • Run the Docker container and mount specific paths. Replace the <path> with the appropriate paths on your system:

   docker run -it --rm \
   -v </path/to/your/replays>:/app/replays/input \
   -v </path/to/your/output>:/app/replays/output \
   -v </path/to/your/logs>:/app/logs \
   -v </path/to/your/dependencies>:/app/dependencies \
   sc2infoextractorgo:latest -help

4. When you execute the processing command, please note that you need to pass the appropriate paths to the tool that are withing the container. The default paths are:
   • Input directory: /app/replays/input
   • Output directory: /app/replays/output
   • Logs directory: /app/logs
   • Dependency directory: /app/dependencies
   • Additionally the tool outputs some auxiliary files that can be used for debugging purposes. These are created in the main directory of the container /app/.

Note

If you wish to see other files that are generated by the tool, you will have to mount the appropriate directories. The tool generates logs in the logs directory.

Example Usage (Local)

Note

These steps assume that you downloaded the release binary or you will have to compile the tool from source.

1. Place .SC2Replay files in ./replays/input
2. Run SC2InfoExtractorGo.exe as follows:

SC2InfoExtractorGo.exe -input ./replays/input -output ./replays/output

3. Verify the output in ./replays/output
4. If The output packages do not contain any processed replays, proceed to verify ./logs/.

CLI Options

To see the full list of available options, run the tool with the -help flag:

SC2InfoExtractorGo.exe -help

The following flags are available:

  -dependency_directory string
        Directory where the replay dependencies will be downloaded as a result of the replay processing. (default "./dependencies/")                                                    
  -game_mode_filter int
        Specifies which game mode should be included from the processed files in a format of a binary flag: AllGameModes: 0b11111111 (default 0b11111111) (default 255)
  -help
        Show command usage
  -input string
        Input directory where .SC2Replay files are held. (default "./replays/input")
  -log_dir string
        Specifies directory which will hold the logging information. (default "./logs/")
  -log_level int
        Specifies a log level from 1-7:
        Panic - 1, Fatal - 2,
        Error - 3, Warn - 4,
        Info - 5, Debug - 6,
        Trace - 7 (default 4)
  -max_procs int
        Specifies the number of logic cores of a processor that will be used for processing (default runtime.NumCPU()). (default 24)
  -number_of_packages int
        Provide a number of zip packages to be created and compressed
        into a zip archive. Please remember that this number needs to be lower
        than the number of processed files. If set to 0, will ommit the
        zip packaging and output .json directly to drive. (default 1)
  -only_dependency_download
        Flag specifying if the tool is supposed to only download
        the replay dependencies and not process the replays.
  -output string
        Output directory where compressed zip packages will be saved. (default "./replays/output")
  -perform_chat_anonymization
        Flag, specifying if the chat anonymization should be performed.
  -perform_cleanup
        Flag specifying if the tool is supposed to perform the cleaning
        functions within the processing pipeline.
  -perform_filtering
        Flag, specifying if the pipeline ought to verify different hard coded game modes.
        If set to false completely bypasses the filtering.
  -perform_integrity_checks
        Flag specifying if the software is supposed to check the hardcoded
        integrity checks for the provided replays
  -perform_player_anonymization
        Flag specifying if the tool is supposed to perform player anonymization
        functions within the processing pipeline.
        If set to true please remember to download and run
        an anonymization server: https://doi.org/10.5281/zenodo.5138313
  -perform_validity_checks
        Flag, specifying if the tool is supposed to use hardcoded validity checks
        and verify if the replay file variables are within 'common sense' ranges.
  -skip_dependency_download
        Flag specifying if the tool is supposed to skip the dependency download.
  -with_cpu_profiler string
        Set path to the file where pprof cpu profiler will save its information.
        If this is empty no profiling is performed.

Dataset Preparation

If You have a pack of replays with nested directories and You would like to automatically flatten the directory structure, We have published a tool that can be used for that, please see SC2DatasetPreparator: https://doi.org/10.5281/zenodo.5296664

Anonymization (Optional)

In order to anonymize the replays please make sure to download and run our open-source implementation of an anonymization server the SC2AnonServerPy: https://doi.org/10.5281/zenodo.5138313

This is required because of the multiprocessing nature of our code that needs to perform synchronization with an existing database of unique toons (player IDs) that are mapped to arbitrary incrementing integer.

Map Translation Support

Existing implementation downloads the maps from the Blizzard servers. This is to normalize the map names to English language. When there is no internet connection available, our tool should fallback to reading the map names from the files placed in the ./dependencies directory.

Filtering Capabilities

Currently the software supports some game mode filtering capabilities which can be used with -game_mode flag. The flag itself is a binary flag where 0b11111111 is all game modes which is the default.

Other ways to set the flag:

• 0b00000001: 1v1 Ranked Games
• 0b00000010: 2v2 Ranked Games
• 0b00000100: 3v3 Ranked Games
• 0b00001000: 4v4 Ranked Games
• 0b00010000: 1v1 Custom Games
• 0b00100000: 2v2 Custom Games
• 0b01000000: 3v3 Custom Games
• 0b10000000: 4v4 Custom Games

License / Dual Licensing

This repository is licensed under GNU GPL v3 license. If You would like to acquire a different license please contact me directly.

Cite Us!

@software{BialeckiExtractor2021,
  author    = {Białecki, Andrzej and
               Białecki, Piotr and
               Krupiński, Leszek},
  title     = {{Kaszanas/SC2InfoExtractorGo: 2.1.3 
               SC2InfoExtractorGo Release}},
  month     = {jun},
  year      = {2022},
  publisher = {Zenodo},
  version   = {2.1.3},
  doi       = {10.5281/zenodo.5296788},
  url       = {https://doi.org/10.5281/zenodo.5296788}
}