Skip to content

AIK-3167 : Switch the python firewall agent to an IPC socket #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 32 commits into from
Jul 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
0414da0
Update aikido env variables
Jul 19, 2024
c2627c1
Update agent __init__.py : Create IPC
Jul 19, 2024
be15ca9
Create a reporting thread
Jul 19, 2024
e15bf5b
Merge branch 'main' into AIK-3167
Jul 22, 2024
b028c78
Actually read from queue and send to queue, also temp crit. logs
Jul 22, 2024
d4fe2e8
Actually read from queue and send to queue, also temp crit. logs
Jul 22, 2024
175ca4e
Merge branch 'AIK-3167' of github.com:AikidoSec/firewall-python into …
Jul 22, 2024
51a7550
Add tests for agent init function
Jul 23, 2024
91e8f9c
Add test cases for the start_ipc and get_ipc functions
Jul 23, 2024
e307854
Add tests to make sure AIKIDO_SECRET_KEY needs to be set
Jul 23, 2024
4220ec6
Add some tests for protect() funtion
Jul 23, 2024
0abfd66
Test connection refused
Jul 23, 2024
53849e5
Linting
Jul 23, 2024
5b4c426
Use monkeypatch for environment vars instead of mocker
Jul 23, 2024
29890c5
Comment out bc not working in CI?CD
Jul 23, 2024
fe05991
Testing/Linting
Jul 23, 2024
364b142
get environment variables using the more standard way
Jul 24, 2024
01571d6
Update testing so that init test does not fail bc of env vars
Jul 24, 2024
6a91bca
Update how con is imported for testing + basic test case (no asserts)
Jul 24, 2024
81fae9c
Change interval to 10 minutes
Jul 25, 2024
4db88c6
Remove any mention of Agent and use Background process instead
Jul 25, 2024
7853f5f
Renames
Jul 25, 2024
772b34b
Remove AIKIDO_SECRET_KEY and generate on the spot
Jul 25, 2024
f7dfb36
Remove AIKIDO_SECRET_KEY and generate on the spot
Jul 25, 2024
5fb1694
Merge branch 'AIK-3167' of github.com:AikidoSec/firewall-python into …
Jul 25, 2024
8553645
Merge branch 'main' into AIK-3167
bitterpanda63 Jul 25, 2024
d2c98e7
Rename action to "ATTACK" for comms
Jul 25, 2024
8015a98
Merge branch 'AIK-3167' of github.com:AikidoSec/firewall-python into …
Jul 25, 2024
682512f
Merge branch 'main' into AIK-3167
bitterpanda63 Jul 25, 2024
bb4ce9f
Update aikido_firewall/background_process/__init__.py
willem-delbare Jul 30, 2024
8b92619
Update aikido_firewall/background_process/__init__.py
willem-delbare Jul 30, 2024
24061c7
Linting
Jul 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions aikido_firewall/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,17 @@
# Import logger
from aikido_firewall.helpers.logging import logger

# Import agent
from aikido_firewall.agent import start_agent
# Import background process
from aikido_firewall.background_process import start_background_process

# Load environment variables
load_dotenv()


def protect(module="any"):
"""Start Aikido agent"""
"""
Protect user's application
"""
# Import sources
import aikido_firewall.sources.django

Expand All @@ -27,4 +29,4 @@ def protect(module="any"):
import aikido_firewall.sinks.pymysql

logger.info("Aikido python firewall started")
start_agent()
start_background_process()
74 changes: 0 additions & 74 deletions aikido_firewall/agent/__init__.py

This file was deleted.

126 changes: 126 additions & 0 deletions aikido_firewall/background_process/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
"""
Aikido background process, this will create a new process
and listen for data sent by our sources and sinks
"""

import time
import os
import secrets
import multiprocessing.connection as con
from multiprocessing import Process
from threading import Thread
from queue import Queue
from aikido_firewall.helpers.logging import logger

REPORT_SEC_INTERVAL = 600 # 10 minutes
IPC_ADDRESS = ("localhost", 9898) # Specify the IP address and port


class AikidoBackgroundProcess:
"""
Aikido's background process consists of 2 threads :
- (main) Listening thread which listens on an IPC socket for incoming data
- (spawned) reporting thread which will collect the IPC data and send it to a Reporter
"""

def __init__(self, address, key):
logger.debug("Background process started")
listener = con.Listener(address, authkey=key)
self.queue = Queue()

Check warning on line 29 in aikido_firewall/background_process/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_firewall/background_process/__init__.py#L27-L29

Added lines #L27 - L29 were not covered by tests
# Start reporting thread :
Thread(target=self.reporting_thread).start()

Check warning on line 31 in aikido_firewall/background_process/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_firewall/background_process/__init__.py#L31

Added line #L31 was not covered by tests

while True:
conn = listener.accept()
logger.debug("connection accepted from %s", listener.last_accepted)
while True:
data = conn.recv()
logger.error(data) # Temporary debugging
if data[0] == "ATTACK":
self.queue.put(data[1])
elif data[0] == "CLOSE": # this is a kind of EOL for python IPC
conn.close()
break

Check warning on line 43 in aikido_firewall/background_process/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_firewall/background_process/__init__.py#L33-L43

Added lines #L33 - L43 were not covered by tests

def reporting_thread(self):
"""Reporting thread"""
logger.debug("Started reporting thread")
while True:
self.send_to_reporter()
time.sleep(REPORT_SEC_INTERVAL)

Check warning on line 50 in aikido_firewall/background_process/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_firewall/background_process/__init__.py#L47-L50

Added lines #L47 - L50 were not covered by tests

def send_to_reporter(self):
"""
Reports the found data to an Aikido server
"""
items_to_report = []
while not self.queue.empty():
items_to_report.append(self.queue.get())
logger.debug("Reporting to aikido server")
logger.critical("Items to report : %s", items_to_report)

Check warning on line 60 in aikido_firewall/background_process/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_firewall/background_process/__init__.py#L56-L60

Added lines #L56 - L60 were not covered by tests
# Currently not making API calls


# pylint: disable=invalid-name # This variable does change
ipc = None


def get_comms():
"""
Returns the globally stored IPC object, which you need
to communicate to our background process.
"""
return ipc


def start_background_process():
"""
Starts a process to handle incoming/outgoing data
"""
# pylint: disable=global-statement # We need this to be global
global ipc
# Generate a secret key :
generated_key_bytes = secrets.token_bytes(32)

ipc = IPC(IPC_ADDRESS, generated_key_bytes)
ipc.start_aikido_listener()


class IPC:
"""
Facilitates Inter-Process communication
"""

def __init__(self, address, key):
self.address = address
self.key = key
self.background_process = None

def start_aikido_listener(self):
"""
This will start the aikido background process which listens
and makes calls to the API
"""
self.background_process = Process(
target=AikidoBackgroundProcess,
args=(
self.address,
self.key,
),
)
logger.debug("Starting the background process")
self.background_process.start()

def send_data(self, action, obj):
"""
This creates a new client for comms to the background process
"""
try:
conn = con.Client(self.address, authkey=self.key)
logger.debug("Created connection %s", conn)
conn.send((action, obj))
conn.send(("CLOSE", {}))
conn.close()
logger.debug("Connection closed")
except Exception as e:
logger.info("Failed to send data to bg process : %s", e)
50 changes: 50 additions & 0 deletions aikido_firewall/background_process/init_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
import pytest
from aikido_firewall.background_process import (
IPC,
start_background_process,
get_comms,
IPC_ADDRESS,
)


def test_ipc_init():
address = ("localhost", 9898)
key = "secret_key"
ipc = IPC(address, key)

assert ipc.address == address
assert ipc.background_process is None


# Following function does not work
def test_start_background_process(monkeypatch):
assert get_comms() == None
start_background_process()

assert get_comms() != None
assert get_comms().address == IPC_ADDRESS

get_comms().background_process.kill()


def test_send_data_exception(monkeypatch, caplog):
def mock_client(address, authkey):
raise Exception("Connection Error")

Check warning on line 32 in aikido_firewall/background_process/init_test.py

View check run for this annotation

Codecov / codecov/patch

aikido_firewall/background_process/init_test.py#L32

Added line #L32 was not covered by tests

monkeypatch.setitem(globals(), "Client", mock_client)
monkeypatch.setitem(globals(), "logger", caplog)

ipc = IPC(("localhost", 9898), "mock_key")
ipc.send_data("ACTION", "Test Object")

assert "Failed to send data to bg" in caplog.text
# Add assertions for handling exceptions


def test_send_data_successful(monkeypatch, caplog, mocker):
ipc = IPC(("localhost"), "mock_key")
mock_client = mocker.MagicMock()
monkeypatch.setattr("multiprocessing.connection.Client", mock_client)

# Call the send_data function
ipc.send_data("ACTION", {"key": "value"})
18 changes: 18 additions & 0 deletions aikido_firewall/init_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import pytest
from aikido_firewall import protect
from aikido_firewall.background_process import get_comms


def test_protect_with_django(monkeypatch, caplog):
monkeypatch.setitem(
globals(), "aikido_firewall.sources.django", "dummy_django_module"
)
monkeypatch.setitem(
globals(), "aikido_firewall.sinks.pymysql", "dummy_pymysql_module"
)

protect(module="django")

assert "Aikido python firewall started" in caplog.text
assert get_comms() != None
get_comms().background_process.kill()
2 changes: 2 additions & 0 deletions aikido_firewall/sinks/pymysql.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
check_context_for_sql_injection,
)
from aikido_firewall.vulnerabilities.sql_injection.dialects import MySQL
from aikido_firewall.background_process import get_comms

logger = logging.getLogger("aikido_firewall")

Expand All @@ -36,6 +37,7 @@

logger.info("sql_injection results : %s", json.dumps(result))
if result:
get_comms().send_data("ATTACK", result)

Check warning on line 40 in aikido_firewall/sinks/pymysql.py

View check run for this annotation

Codecov / codecov/patch

aikido_firewall/sinks/pymysql.py#L40

Added line #L40 was not covered by tests
raise Exception("SQL Injection [aikido_firewall]")
return prev_query_function(_self, sql, unbuffered=False)

Expand Down
2 changes: 2 additions & 0 deletions sample-apps/django-mysql-gunicorn/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,6 @@ MYSQL_DATABASE="db"
MYSQL_USER="user"
MYSQL_PASSWORD="password"
MYSQL_ROOT_PASSWORD="password"

# Aikido keys
AIKIDO_DEBUG=true
2 changes: 2 additions & 0 deletions sample-apps/django-mysql/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,6 @@ MYSQL_DATABASE="db"
MYSQL_USER="user"
MYSQL_PASSWORD="password"
MYSQL_ROOT_PASSWORD="password"

# Aikido keys
AIKIDO_DEBUG=true
2 changes: 1 addition & 1 deletion sample-apps/flask-mongo/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

app = Flask(__name__)
if __name__ == '__main__':
app.run(threaded=True) # Run threaded so we can test our agent's capabilities
app.run(threaded=True) # Run threaded so we can test how our bg process works
app.config["MONGO_URI"] = "mongodb://admin:password@db:27017/my_database?authSource=admin"
mongo = PyMongo(app)

Expand Down
2 changes: 1 addition & 1 deletion sample-apps/flask-postgres/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

app = Flask(__name__)
if __name__ == '__main__':
app.run(threaded=True) # Run threaded so we can test our agent's capabilities
app.run(threaded=True) # Run threaded so we can test how our bg process works

def get_db_connection():
return psycopg2.connect(
Expand Down
Loading