Shovel is a web application that offers a graphical user interface to explore Suricata Extensible Event Format (EVE) outputs. Its primary focus is to help Capture-the-Flag players analyse network flows during stressful and time-limited attack-defense games such as FAUSTCTF, ENOWARS or ECSC. Shovel is developed in the context of ECSC Team France training.
You might also want to have a look at these other awesome traffic analyser tools:
- https://github.com/secgroup/flower (first commit in 2018)
- https://github.com/eciavatta/caronte (first commit in 2020)
- https://github.com/OpenAttackDefenseTools/tulip (fork from flower in May 2022)
Compared to these traffic analyser tools, Shovel only relies on Suricata while making opinionated choices for the frontend. This has a few nice implications:
- dissection of all application protocols supported by Suricata (HTTP2, modbus, SMB, DNS, etc),
- flows payloads and dissections are stored inside SQLite databases for fast queries,
- ingest can be a folder of pcaps for non-root CTF, or a live capture (less delay),
- tags are defined using Suricata rules (regex, libmagic match, HTTP header, etc),
- no heavy build tools needed, Shovel is easy to tweak.
Moreover, Shovel is batteries-included with some Suricata alert rules.
┌────────────────────────┐
device │ Suricata with: │ eve.db ┌───────────────┐
or pcap │ - Eve SQLite plugin ├────────────►│ │
───────►│ - TCP payloads plugin │ payload.db │ Python webapp │
│ - UDP payloads plugin ├────────────►│ │
└────────────────────────┘ └────▲──────────┘
.env │
──────┘
Shovel is configured using environment variables.
Copy example.env to .env and update the optional configuration parameters.
You may update this file later and restart only the webapp.
Add the flag format in suricata/rules/suricata.rules if needed.
If you modify this file after starting Suricata, you may reload rules using
pkill -USR2 suricata.
Shovel currently implements 3 capture modes:
- Mode A: pcap replay (slower, for archives replay or rootless CTF).
- Mode B: capture interface (fast, requires root on vulnbox and in Docker).
- Mode C: PCAP-over-IP (fast, requires root on vulnbox).
Please prefer mode B or C to get the best latency between the game network and Suricata. Use mode A only if you are not root on the vulnbox and have access to pcap files indirectly.
Place pcap files in a folder such as input_pcaps/.
If you are continuously adding new pcap, add --pcap-file-continuous to
Suricata command line.
A sample configuration can be found in docker-compose-a.yml.
If you don't want to use Docker, you may manually launch Suricata and the web application using the two following commands:
./suricata/entrypoint.sh -r input_pcaps
(cd webapp && uvicorn --host 127.0.0.1 main:app)Warning
Please note that restarting Suricata will cause all network capture files to be loaded again. It might add some delay before observing new flows.
Tip
For a Microsoft Windows system, you may capture network traffic using the following command (3389 is RDP) inside a PowerShell console:
&'C:\Program Files\Wireshark\tshark.exe' -i game -w Z:\ -f "tcp port not 3389" -b duration:60This mode requires to have direct access to the game network interface.
This can be achieved by mirroring vulnbox traffic through a tunnel,
see FAQ for more details.
Here this device is named tun5.
A sample configuration can be found in docker-compose-b.yml.
If you don't want to use Docker, you may manually launch Suricata and the web application using the two following commands:
sudo ./suricata/entrypoint.sh -i tun5
(cd webapp && uvicorn --host 127.0.0.1 main:app)Warning
Please note that stopping Suricata will stop network capture.
You may also run sudo tcpdump -n -i tun5 -G 30 -w trace-%Y-%m-%d_%H-%M-%S.pcap
for archiving purposes.
This mode requires to have access to a TCP listener exposing PCAP-over-IP. Such server can be easily spawned using:
tcpdump -U --immediate-mode -ni game -s 65535 -w - not tcp port 22 | nc -l 57012If you need to route PCAP-over-IP to multiple clients, you should consider using
pcap-broker.
A sample configuration is given in docker-compose-c.yml.
If you don't want to use Docker, you may manually launch Suricata and the web application using the two following commands:
PCAP_OVER_IP=pcap-broker:4242 ./suricata/entrypoint.sh -r /dev/stdin
(cd webapp && uvicorn --host 127.0.0.1 main:app)Warning
Please note that stopping Suricata will stop network capture.
Shovel provides a convenient ./start.py script to manage containers and configuration. The script supports multiple commands with a user-friendly interface.
If you run ./start.py without arguments, it will enter interactive mode where you can select actions and configure parameters through guided prompts.
Here you can find a list of available commands to use with ./start.py:
start- Build and start containersstop- Stop running containersclear- Clear data (Suricata output, config files, PCAP files) and stop containersstatus- Show container statuslogs- Follow container logshelp- Show help information
To start Shovel, use the start command with one of the following capture modes:
--mode-a- Mode A, for pcap replay mode--mode-b- Mode B, for live capture interface mode--mode-c- Mode C, for live capture using PCAP-over-IP (default if not specified)
When using Mode C, you can specify additional parameters:
--target-ip TARGET_IP(or-ip): IP address of the vulnbox (MANDATORY for Mode C)--date START_DATE: CTF start date in ISO formatYYYY-MM-DDThh:mm+ZZ:zz--tick-length LENGTH(or-t): Tick length in seconds (default: 120)--refresh-rate RATE(or-r): Auto-refresh rate in seconds (default: 30)--key ALGORITHM(or-k): SSH key algorithm for connection (default: ed25519)
Here you can find an example with the full configuration:
./start.py start --mode-c \
--target-ip 10.60.0.1 \
--date "1970-01-01T10:00+02:00" \
--tick-length 120 \
--refresh-rate 30 \
--key ed25519--no-build: Skip building Docker images (use existing images)--no-clean: Skip cleaning environment (keep existing data)
To stop running containers:
./start.py stopThe clear command provides granular control over data cleanup. You can use the interactive clear by running:
./start.py clearAlternatively, you can specify what to clean:
./start.py clear --config # (or -c): Clear .env and services_config.json
./start.py clear --suricata # (or -s): Clear Suricata output and stop containers
./start.py clear --pcap # (or -p): Clear PCAP filesTo rapidly delete everything listed above you can use the flag --all (or -A).
To avoid the cleanup, you can add the flag --no-clean to the startup command.
Check container status:
./start.py statusFollow container logs:
# Follow all container logs
./start.py logs
# Follow logs with tail limit
./start.py logs --tail 100
# Follow specific service logs
./start.py logs webapp --tail 50For a complete list of commands and options, run:
./start.py helpCommon command examples:
# Quick start Mode C with target IP
./start.py start --mode-c --target-ip 10.60.2.1
# Start without rebuilding images
./start.py start --mode-c --no-build
# Start preserving existing data
./start.py start --mode-c --no-clean
# Stop everything
./start.py stop
# Clean everything and stop
./start.py clear --all
# Monitor containers
./start.py status
./start.py logsTo setup the services mapping, you can edit the .env file by hand. In alternative, once Shovel is started you can customize the configuration directly from the web interface.
Click on the settings icon in the top left corner to open the Service Manager modal. Here you can add:
- service name,
- port numbers (one or more, separated by commas or new lines),
- colour (helps to identify the service in the flow list).
After adding a service, you can also edit it by clicking on the pencil icon next to the service name.
Warning
Note that if Shovel is restarted without the --no-clean flag, the configuration in the web interface will be lost. Use ./start.py start --no-clean to preserve existing configuration.
Shovel includes an auto-refresh feature that automatically refreshes the flow list with an interval specified by the REFRESH_RATE parameter in the .env file. After updating, the interface will scroll to the previously selected flow (if any).
The default value of REFRESH_RATE can be set during Shovel startup using the --refresh-rate parameter:
./start.py start --refresh-rate 60You can also update this value at runtime from the Service Manager in the web interface. The new value will be saved locally in the user's browser to not interfere with other users.
The auto-refresh feature can be toggled by clicking the Auto-Update button in the top left corner of the web interface.
flow_id is derived from timestamp (ms scale) and current flow parameters (such
as source and destination ports and addresses). See source code:
https://github.com/OISF/suricata/blob/suricata-6.0.13/src/flow.h#L680.
Most CTF uses OpenVPN or Wireguard for the "game" network interface on the vulnbox,
which means you can duplicate the traffic to an OpenSSH tun tunnel.
Using this method, Shovel can run on another machine in live capture mode.
Warning
If you need to clone a physical Ethernet interface such as eth0,
you will need to use -o Tunnel=ethernet -w 5:5 in the SSH command line to create a tap.
To achieve traffic mirroring, you may use these steps as reference:
-
Enable SSH tunneling in vulnbox OpenSSH server:
echo -e 'PermitTunnel yes' | sudo tee -a /etc/ssh/sshd_config systemctl restart ssh
-
Create
tun5tunnel from the local machine to the vulnbox and uptun5on vulnbox:sudo ip tuntap add tun5 mode tun user $USER ssh -w 5:5 root@10.20.9.6 ip link set tun5 up
-
Up
tun5on the local machine and starttcpdumpto create pcap files:sudo ip link set tun5 up sudo tcpdump -n -i tun5 -G 30 -Z root -w trace-%Y-%m-%d_%H-%M-%S.pcap -
Mirror
gametraffic totun5on the vulnbox. This can be done using Nftables netdevdupoption oningressandegress.
You can edit suricata rules in suricata/rules/suricata.rules, then reload the rules
using:
pkill -USR2 suricata