Skip to content

INTEGRITY: Continuing set.dat's processing #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 28 commits into
base: integrity
Choose a base branch
from
Open

INTEGRITY: Continuing set.dat's processing #32

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
161abd5
INTEGRITY: Increase character limit size for log text
ShivangNagta Jul 4, 2025
94cd1d9
INTEGRITY: Separate the additional checksum add logic from insert_fil…
ShivangNagta Jul 4, 2025
cc82c6f
INTEGRITY: Add filtering by platform for set.dat to reduce manual merge.
ShivangNagta Jul 4, 2025
599d2f1
INTEGRITY: Avoid adding a fileset as candidate if it was marked as pa…
ShivangNagta Jul 7, 2025
6ea2bba
INTEGRITY: Add additional filtering logic for glk engines
ShivangNagta Jul 7, 2025
3028d19
INTEGRITY: Add timestamp field in scan.dat and filtering support via …
ShivangNagta Jul 7, 2025
19b19c0
INTEGRITY: Add all size variants to scan.dat - size, size-r and size-rd.
ShivangNagta Jul 7, 2025
074da92
INTEGRITY: Fix clear database hang issue. Now the database is dropped…
ShivangNagta Jul 8, 2025
31b7d4f
INTEGRITY: Remove global database connection object from fileset.py, …
ShivangNagta Jul 8, 2025
e125227
INTEGRITY: Filter manual merge candidates if size mismatch.
ShivangNagta Jul 9, 2025
898ffd0
INTEGRITY: Add metadata for set.dat
ShivangNagta Jul 10, 2025
8970cd6
INTEGRITY: Add navbar with logo.
ShivangNagta Jul 10, 2025
bd3f2f4
INTEGRITY: Add modification timestamps for macfiles
ShivangNagta Jul 10, 2025
8575f8e
INTEGIRTY: Add punycode encoding for scan utlity.
ShivangNagta Jul 10, 2025
df41d7a
INTEGRITY: Fix the navbar on top.
ShivangNagta Jul 10, 2025
c38881c
INTEGRITY: Limit match fileset to 1 in remove_manual_merge_if_size_mi…
ShivangNagta Jul 14, 2025
e86f982
INTEGRITY: Improve console logging with progress update.
ShivangNagta Jul 14, 2025
7dcb20b
INTEGRITY: Remove custom recursive path split function.
ShivangNagta Jul 14, 2025
96d9cf4
INTEGRITY: Use INFORMATION_SCHEMA.COLUMNS instead of relying on error…
ShivangNagta Jul 14, 2025
4c9a5e7
INTEGRITY: Add scan processing logic.
ShivangNagta Jul 14, 2025
33cac5a
INTEGRITY: Add additional modification-time column in file table.
ShivangNagta Jul 14, 2025
90ffe1a
INTEGRITY: Additional error handling while extracing keys from scummv…
ShivangNagta Jul 14, 2025
493acb5
INTEGRITY: Traverse set.dat instead of candidate fileset while search…
ShivangNagta Jul 14, 2025
ff9934f
INTEGRITY: Add checksum based filtering in set.dat, when possible.
ShivangNagta Jul 16, 2025
ca9d4a7
INTEGRITY: Remove 'obsolete' fileset status entirely.
ShivangNagta Jul 17, 2025
a303645
INTEGRITY: Add checksum based filtering before filtering by maximum n…
ShivangNagta Jul 17, 2025
4dd7e29
INTEGRITY: Merge one of the entries from dropped duplicate entries. D…
ShivangNagta Jul 17, 2025
57df340
INTEGRITY: Merge filtering logic for glk with existing set.dat filter…
ShivangNagta Jul 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
309 changes: 244 additions & 65 deletions compute_hash.py
View file
Open in desktop

Large diffs are not rendered by default.

395 changes: 356 additions & 39 deletions db_functions.py
View file
Open in desktop

Large diffs are not rendered by default.

119 changes: 75 additions & 44 deletions fileset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,28 +23,15 @@
user_integrity_check,
db_connect,
create_log,
db_connect_root,
)
from collections import defaultdict
from schema import init_database

app = Flask(__name__)

secret_key = os.urandom(24)

base_dir = os.path.dirname(os.path.abspath(__file__))
config_path = os.path.join(base_dir, "mysql_config.json")
with open(config_path) as f:
mysql_cred = json.load(f)

conn = pymysql.connect(
host=mysql_cred["servername"],
user=mysql_cred["username"],
password=mysql_cred["password"],
db=mysql_cred["dbname"],
charset="utf8mb4",
cursorclass=pymysql.cursors.DictCursor,
autocommit=False,
)


@app.route("/")
def index():
Expand All @@ -55,7 +42,12 @@ def index():
<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='style.css') }}">
</head>
<body>
<h1>Fileset Database</h1>
<nav style="position: fixed; top: 0; left: 0; right: 0; background: white; padding: 3px; border-bottom: 1px solid #ccc;">
<a href="{{ url_for('index') }}">
<img src="{{ url_for('static', filename='integrity_service_logo_256.png') }}" alt="Logo" style="height:60px; vertical-align:middle;">
</a>
</nav>
<h1 style="margin-top: 80px;">Fileset Database</h1>
<h2>Fileset Actions</h2>
<ul>
<li><a href="{{ url_for('fileset') }}">Fileset</a></li>
Expand All @@ -79,21 +71,13 @@ def index():
@app.route("/clear_database", methods=["POST"])
def clear_database():
try:
conn = db_connect()
(conn, db_name) = db_connect_root()
with conn.cursor() as cursor:
cursor.execute("SET FOREIGN_KEY_CHECKS = 0;")
cursor.execute("TRUNCATE TABLE filechecksum")
cursor.execute("TRUNCATE TABLE history")
cursor.execute("TRUNCATE TABLE transactions")
cursor.execute("TRUNCATE TABLE queue")
cursor.execute("TRUNCATE TABLE file")
cursor.execute("TRUNCATE TABLE fileset")
cursor.execute("TRUNCATE TABLE game")
cursor.execute("TRUNCATE TABLE engine")
cursor.execute("TRUNCATE TABLE log")
cursor.execute("SET FOREIGN_KEY_CHECKS = 1;")
cursor.execute(f"DROP DATABASE IF EXISTS {db_name}")
conn.commit()
print("DATABASE CLEARED")
print("DATABASE DROPPED")
init_database()
print("DATABASE INITIALISED")
except Exception as e:
print(f"Error clearing database: {e}")
finally:
Expand Down Expand Up @@ -159,7 +143,12 @@ def fileset():
<link rel="stylesheet" type="text/css" href="{{{{ url_for('static', filename='style.css') }}}}">
</head>
<body>
<h2><u>Fileset: {id}</u></h2>
<nav style="position: fixed; top: 0; left: 0; right: 0; background: white; padding: 3px; border-bottom: 1px solid #ccc;">
<a href="{{{{ url_for('index') }}}}">
<img src="{{{{ url_for('static', filename='integrity_service_logo_256.png') }}}}" alt="Logo" style="height:60px; vertical-align:middle;">
</a>
</nav>
<h2 style="margin-top: 80px;"><u>Fileset: {id}</u></h2>
<table>
"""
html += f"<button type='button' onclick=\"location.href='/fileset/{id}/merge'\">Manual Merge</button>"
Expand All @@ -176,18 +165,32 @@ def fileset():
(id,),
)
row = cursor.fetchone()
print(row)
if row:
id = row["fileset"]
cursor.execute(f"SELECT * FROM fileset WHERE id = {id}")
cursor.execute("SELECT status FROM fileset WHERE id = %s", (id,))
status = cursor.fetchone()["status"]

if status == "dat":
cursor.execute(
"""SELECT id, game, status, src, `key`, megakey, `delete`, timestamp, set_dat_metadata FROM fileset WHERE id = %s""",
(id,),
)
else:
cursor.execute(
"""SELECT id, game, status, src, `key`, megakey, `delete`, timestamp, detection_size, user_count FROM fileset WHERE id = %s""",
(id,),
)

result = cursor.fetchone()
print(result)
html += "<h3>Fileset details</h3>"
html += "<table>\n"
if result["game"]:
cursor.execute(
f"SELECT game.name as 'game name', engineid, gameid, extra, platform, language FROM fileset JOIN game ON game.id = fileset.game JOIN engine ON engine.id = game.engine WHERE fileset.id = {id}"
)
if status == "dat":
query = """SELECT game.name as 'game name', engineid, gameid, extra, platform, language, fileset.set_dat_metadata FROM fileset JOIN game ON game.id = fileset.game JOIN engine ON engine.id = game.engine WHERE fileset.id = %s"""
else:
query = """SELECT game.name as 'game name', engineid, gameid, extra, platform, language FROM fileset JOIN game ON game.id = fileset.game JOIN engine ON engine.id = game.engine WHERE fileset.id = %s"""
print(query)
cursor.execute(query, (id,))
result = {**result, **cursor.fetchone()}
else:
# result.pop('key', None)
Expand Down Expand Up @@ -354,7 +357,7 @@ def fileset():
html += "<th>Description</th>\n"
html += "<th>Log Text</th>\n"

related_filesets = get_all_related_filesets(id, conn)
related_filesets = get_all_related_filesets(id, connection)

cursor.execute(
f"SELECT * FROM history WHERE fileset IN ({','.join(map(str, related_filesets))}) OR oldfileset IN ({','.join(map(str, related_filesets))})"
Expand Down Expand Up @@ -473,7 +476,12 @@ def match_fileset_route(id):
<link rel="stylesheet" type="text/css" href="{{{{ url_for('static', filename='style.css') }}}}">
</head>
<body>
<h2>Matched Filesets for Fileset: {id}</h2>
<nav style="position: fixed; top: 0; left: 0; right: 0; background: white; padding: 3px; border-bottom: 1px solid #ccc;">
<a href="{{{{ url_for('index') }}}}">
<img src="{{{{ url_for('static', filename='integrity_service_logo_256.png') }}}}" alt="Logo" style="height:60px; vertical-align:middle;">
</a>
</nav>
<h2 style="margin-top: 80px;">Matched Filesets for Fileset: {id}</h2>
<table>
<tr>
<th>Fileset ID</th>
Expand Down Expand Up @@ -560,7 +568,12 @@ def merge_fileset(id):
<link rel="stylesheet" type="text/css" href="{{{{ url_for('static', filename='style.css') }}}}">
</head>
<body>
<h2>Search Results for '{search_query}'</h2>
<nav style="position: fixed; top: 0; left: 0; right: 0; background: white; padding: 3px; border-bottom: 1px solid #ccc;">
<a href="{{{{ url_for('index') }}}}">
<img src="{{{{ url_for('static', filename='integrity_service_logo_256.png') }}}}" alt="Logo" style="height:60px; vertical-align:middle;">
</a>
</nav>
<h2 style="margin-top: 80px;">Search Results for '{search_query}'</h2>
<form method="POST">
<input type="text" name="search" placeholder="Search fileset">
<input type="submit" value="Search">
Expand Down Expand Up @@ -594,7 +607,12 @@ def merge_fileset(id):
<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='style.css') }}">
</head>
<body>
<h2>Search Fileset to Merge</h2>
<nav style="position: fixed; top: 0; left: 0; right: 0; background: white; padding: 3px; border-bottom: 1px solid #ccc;">
<a href="{{ url_for('index') }}">
<img src="{{ url_for('static', filename='integrity_service_logo_256.png') }}" alt="Logo" style="height:60px; vertical-align:middle;">
</a>
</nav>
<h2 style="margin-top: 80px;">Search Fileset to Merge</h2>
<form method="POST">
<input type="text" name="search" placeholder="Search fileset">
<input type="submit" value="Search">
Expand Down Expand Up @@ -648,7 +666,12 @@ def possible_merge_filesets(id):
<link rel="stylesheet" type="text/css" href="{{{{ url_for('static', filename='style.css') }}}}">
</head>
<body>
<h2>Possible Merges for fileset-'{id}'</h2>
<nav style="position: fixed; top: 0; left: 0; right: 0; background: white; padding: 3px; border-bottom: 1px solid #ccc;">
<a href="{{{{ url_for('index') }}}}">
<img src="{{{{ url_for('static', filename='integrity_service_logo_256.png') }}}}" alt="Logo" style="height:60px; vertical-align:middle;">
</a>
</nav>
<h2 style="margin-top: 80px;">Possible Merges for fileset-'{id}'</h2>
<table>
<tr><th>ID</th><th>Game Name</th><th>Platform</th><th>Language</th><th>Extra</th><th>Details</th><th>Action</th></tr>
"""
Expand Down Expand Up @@ -755,7 +778,12 @@ def highlight_differences(source, target):
<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='style.css') }}">
</head>
<body>
<h2>Confirm Merge</h2>
<nav style="position: fixed; top: 0; left: 0; right: 0; background: white; padding: 3px; border-bottom: 1px solid #ccc;">
<a href="{{ url_for('index') }}">
<img src="{{ url_for('static', filename='integrity_service_logo_256.png') }}" alt="Logo" style="height:60px; vertical-align:middle;">
</a>
</nav>
<h2 style="margin-top: 80px;">Confirm Merge</h2>
<table border="1">
<tr><th>Field</th><th>Source Fileset</th><th>Target Fileset</th></tr>
"""
Expand Down Expand Up @@ -977,9 +1005,12 @@ def validate():
del json_response["files"]
json_response["status"] = "no_metadata"

fileset_id = user_insert_fileset(json_object, ip, conn)
conn = db_connect()
try:
fileset_id = user_insert_fileset(json_object, ip, conn)
finally:
conn.close()
json_response["fileset"] = fileset_id
print(f"Response: {json_response}")
return jsonify(json_response)

matched_map = {}
Expand Down
7 changes: 6 additions & 1 deletion pagination.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,13 @@ def create_page(
<link rel="stylesheet" type="text/css" href="{{ url_for('static', filename='style.css') }}">
</head>
<body>
<nav style="position: fixed; top: 0; left: 0; right: 0; background: white; padding: 3px; border-bottom: 1px solid #ccc;">
<a href="{{ url_for('index') }}">
<img src="{{ url_for('static', filename='integrity_service_logo_256.png') }}" alt="Logo" style="height:60px; vertical-align:middle;">
</a>
</nav>
<form id='filters-form' method='GET' onsubmit='remove_empty_inputs()'>
<table>
<table style="margin-top: 80px;">
"""
if not results:
return "No results for given filters"
Expand Down
Loading