Skip to content

Commit 6133e20

Browse files
michaelehabmichaelehab
committed
Add GSoC 2024 VulnTotal Browser Extension Project Report
Signed-off-by: Michael Ehab <michael.ehab@hotmail.com>
1 parent e48c509 commit 6133e20

File tree

2 files changed

+289
-0
lines changed

2 files changed

+289
-0
lines changed

‎docs/source/archive/gsoc-toc.rst

100644100755
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ GSoC 2024
1616

1717
gsoc/reports/2024/scancode_toolkit_swastkk
1818
gsoc/reports/2024/scancodeio_scorecode_pranay
19+
gsoc/reports/2024/vulntotal_extension_michael
1920

2021
GSoC 2022
2122
---------

‎docs/source/archive/gsoc/reports/2024/vulntotal_extension_michael.rst

Lines changed: 288 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,288 @@
1+
VulnTotal Browser Extension: Cross-validating vulnerability right within the browser
2+
====================================================================================
3+
4+
Organization - `AboutCode <https://www.aboutcode.org>`_
5+
-----------------------------------------------------------
6+
| **Michael Ehab Mikhail**
7+
| GitHub: `michaelehab <https://github.com/michaelehab>`_
8+
| LinkedIn: `@michaelehab16 <https://www.linkedin.com/in/michaelehab16/>`_
9+
| Project: `VulnTotal Browser Extension <https://github.com/aboutcode-org/vulntotal-extension>`_
10+
| Proposal: `Link <https://docs.google.com/document/d/1K7oGBlfHxbrxah3TJW1PcUGgaLMR6q0ctK7_Y10hpVY/edit?usp=sharing>`_
11+
12+
Overview
13+
--------
14+
15+
This project aims to simplify the process of vulnerability analysis by
16+
developing a browser extension for Chrome-based browsers (Google Chrome,
17+
Firefox, Arc, etc.). The extension leverages the power of `VulnTotal
18+
<https://aboutcode.readthedocs.io/en/latest/archive/gsoc/reports/2022/vulnerablecode_vulntotal_keshav.html>`_,
19+
a Python-based tool that queries multiple vulnerability data sources, such
20+
as OSV and Snyk, to provide comprehensive security assessments of software
21+
packages. The key innovation in this project is the use of Pyodide, a port
22+
of CPython to WebAssembly/Emscripten, which enables Python code to run
23+
natively in the browser.
24+
25+
By integrating Pyodide with VulnTotal, the extension can perform
26+
vulnerability analysis directly within the browser environment,
27+
eliminating the need for server-side components or external applications.
28+
The extension provides a user-friendly interface that displays the results
29+
of vulnerability analyses in a clear and interpretable table format. The
30+
table allows users to quickly assess the security status of the software
31+
packages they are examining.
32+
33+
This project not only enhances the functionality and reach of VulnTotal but
34+
also provides a valuable tool for developers and security professionals who
35+
seek to ensure the safety of their software directly from their browsers.
36+
By simplifying and streamlining the vulnerability-checking process, this
37+
extension contributes to safer and more secure software development practices.
38+
39+
Extension Preview
40+
-----------------
41+
42+
.. figure:: https://github.com/user-attachments/assets/07966663-9f89-4b80-9db3-0840e246d2c5
43+
:alt: Extension Preview
44+
:align: center
45+
:width: 70%
46+
47+
VulnTotal Extension takes PURL and displays vulnerability data from
48+
various data sources. By default, vulnerability data is grouped by CVE.
49+
50+
.. note::
51+
A PURL is a URL string used to identify and locate a software package
52+
in a mostly universal and uniform way across programming languages,
53+
package managers, packaging conventions, tools, APIs, and databases.
54+
`more on PURL <https://github.com/package-url>`_
55+
56+
Project Design and Architecture
57+
-------------------------------
58+
59+
The design of this project revolves around the integration of VulnTotal
60+
with a browser environment, enabling seamless vulnerability analysis
61+
without the need for server-side processing. This is achieved by leveraging
62+
Pyodide, which runs Python code directly in the browser through WebAssembly.
63+
Below, we discuss the core components of the project and how they interact
64+
to deliver a functional and user-friendly extension.
65+
66+
Pyodide Integration
67+
^^^^^^^^^^^^^^^^^^^
68+
69+
Pyodide is a critical component of this project, allowing the VulnTotal
70+
Python library to be executed within the browser. Pyodide converts CPython
71+
to WebAssembly, enabling Python code to run natively in a web environment.
72+
This setup eliminates the need for external servers or backend
73+
infrastructure, making the extension self-contained and easy to deploy.
74+
75+
.. figure:: https://github.com/user-attachments/assets/2bf61204-e26b-4322-8403-f126b3d44748
76+
:alt: Pyodide integration within the browser
77+
:align: center
78+
:width: 70%
79+
80+
In the context of this extension, Pyodide serves as the bridge between
81+
the browser's JavaScript environment and the Python-based VulnTotal tool.
82+
The extension uses Pyodide to load the VulnTotal code, perform
83+
vulnerability checks, and return the results to the user interface,
84+
where they are displayed in a structured format.
85+
86+
Implementation
87+
--------------
88+
89+
The implementation of the browser extension involved several key steps,
90+
each crucial to ensuring the extension's functionality and integration
91+
with VulnTotal. The process began with designing the frontend and progressed
92+
through setting up Pyodide, creating core classes, and integrating all
93+
components. Below is a detailed account of the implementation process.
94+
95+
Frontend Design
96+
^^^^^^^^^^^^^^^
97+
98+
The initial phase of development focused on designing the frontend of the
99+
extension. To maintain a consistent user experience, I adopted a design
100+
approach similar to that used in VulnerableCode. This design choice aimed
101+
to provide users with a familiar and intuitive interface for vulnerability
102+
analysis.
103+
104+
I created a full demo using JavaScript to simulate how the extension should
105+
function. This demo utilized a JSON response from the VulnTotal Python
106+
package, allowing me to test and validate the user interface and interaction
107+
flow before integrating with Pyodide.
108+
109+
Pyodide Integration
110+
^^^^^^^^^^^^^^^^^^^
111+
112+
The most challenging part of the project was integrating the Pyodide Python
113+
package into the browser environment. Pyodide allows Python code to run
114+
within the browser using WebAssembly. To achieve this, I performed the
115+
following steps:
116+
117+
* Setting Up Pyodide: I configured Pyodide, ensuring that the environment
118+
was properly set up to run Python code. This included loading the
119+
necessary Python packages into the environment.
120+
* Using Python Code in JavaScript: I wrote the VulnTotalWorker.js class to
121+
manage the interaction between the Python environment and the browser.
122+
This class handles various tasks:
123+
124+
* Environment Initialization: It ensures that the Pyodide environment
125+
is correctly initialized and ready for executing Python code.
126+
* Communication with UI: It updates the user interface with loading
127+
progress and other relevant information.
128+
* API Key Management: It manages API keys required for VulnTotal
129+
operations within the Python environment.
130+
* Request Handling: It processes scan requests, which include the
131+
Package URL (PURL) and the enabled or disabled data sources, and
132+
returns the analysis results.
133+
134+
Web Worker Integration
135+
^^^^^^^^^^^^^^^^^^^^^^
136+
137+
The VulnTotalWorker class operates within a web worker to handle the actual
138+
requests for vulnerability analysis. The web worker allows the extension to
139+
perform background tasks without blocking the main thread, ensuring a
140+
smooth user experience. It processes requests and communicates with the
141+
VulnTotalWorker class to fetch and return results.
142+
143+
Background Script
144+
^^^^^^^^^^^^^^^^^
145+
146+
The background script is responsible for managing user settings and API keys.
147+
It stores these preferences and ensures that they are accessible to other
148+
components of the extension. This worker plays a crucial role in maintaining
149+
user-specific configurations and providing a seamless experience.
150+
151+
Content Script
152+
^^^^^^^^^^^^^^
153+
154+
The content script acts as the glue between different components of the
155+
extension. It facilitates communication between the background script, web
156+
worker, and user interface. The content script ensures that all components
157+
work together effectively, coordinating the flow of data and requests to
158+
deliver the desired functionality.
159+
160+
Build Script
161+
^^^^^^^^^^^^
162+
163+
The final part was adding a script to build the extension. This script uses
164+
an installed chrome-based browser to bundle all the necessary files,
165+
including the Pyodide core package and the VulnTotal Python code, into a
166+
single extension package. This package can then be installed in the browser
167+
to enable vulnerability analysis directly within the browser environment.
168+
169+
.. figure:: https://github.com/user-attachments/assets/8f13128e-a32c-46f9-af47-4ec18c7e84d4
170+
:alt: Extension Architecture
171+
:align: center
172+
:width: 70%
173+
174+
The architecture of the extension consists of multiple components that
175+
work together to provide vulnerability analysis within the browser. The
176+
frontend interacts with the content script, which coordinates requests
177+
and data flow between the background script, web worker, and Pyodide
178+
environment. This setup ensures a seamless and efficient user experience.
179+
180+
Challenges and Solutions
181+
------------------------
182+
183+
1. WebAssembly and Project Scale
184+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
185+
186+
The first significant challenge was utilizing WebAssembly (Wasm) for a
187+
project of this scale. While most tutorials and available content
188+
demonstrate running simple scripts in the browser, this project involved a
189+
more complex scenario. Integrating the VulnTotal tool, which has multiple
190+
dependencies and a larger codebase, required careful optimization and
191+
adaptation to ensure it ran smoothly within the browser environment.
192+
Overcoming this challenge involved extensive testing and tweaking of the
193+
Pyodide setup to ensure compatibility and performance.
194+
195+
2. Browser Context Incompatibilities
196+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
197+
198+
Running Python scripts in a browser context introduces restrictions that do
199+
not exist when running the same scripts in a standard Python environment.
200+
One of the key challenges was handling HTTP requests. Certain requests that
201+
work seamlessly in a Python script faced issues in the browser due to the
202+
lack of request origin, which is a crucial aspect of web security.
203+
204+
To address this, running requests directly in the content script was not
205+
feasible. The solution was to leverage Web Workers, which allowed for more
206+
flexible request handling within the browser. This approach is recommended
207+
by Google and provides a way to bypass some of the restrictions imposed by
208+
the browser environment.
209+
210+
3. GitLab Data Source Integration
211+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
212+
213+
Another challenge was integrating the GitLab data source into the extension.
214+
The original VulnTotal tool had specific configurations and data retrieval
215+
methods for GitLab that required adaptation for the browser environment.
216+
Ensuring proper data retrieval and presentation from GitLab required
217+
modifying the existing VulnTotal code and handling data fetching and
218+
parsing within the browser constraints.
219+
220+
4. Extension Size Optimization
221+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
222+
223+
The size of the Pyodide release posed a practical challenge. The compressed
224+
release size for Pyodide version 0.26.1 was approximately 281MB, which is
225+
impractical to include within the extension files. Additionally, relying on
226+
a CDN for such a large package would be challenging due to browser
227+
limitations and potential issues with internet consumption.
228+
229+
To solve this, I opted to use only the Pyodide core package, which has a
230+
much smaller compressed size of 5.31MB. I then bundled only the necessary
231+
wheels for the required packages, reducing the overall extension size to
232+
about 20MB. This made the extension more efficient and practical for users
233+
to install and use.
234+
235+
5. Patching the Requests Module
236+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
237+
238+
Another challenge involved patching the Python requests module to work
239+
seamlessly within the browser environment. Pyodide provides a patch that
240+
allows requests to use XHR for non-streaming requests and the Fetch API for
241+
streaming requests. However, this patch only worked correctly for the
242+
initial VulnTotal request. To ensure consistent and correct results, I
243+
implemented a mechanism to apply this patch every time the Python code was
244+
executed. This ensured that all requests were handled appropriately,
245+
maintaining the functionality of the extension.
246+
247+
Linked Pull Requests
248+
--------------------
249+
250+
.. list-table::
251+
:widths: 10 40 20 30
252+
:header-rows: 1
253+
254+
* - Sr. no
255+
- Name
256+
- Link
257+
- Status
258+
* - 1
259+
- Frontend Demo
260+
- `aboutcode-org/vulntotal-extension#5 <https://github.com/aboutcode-org/vulntotal-extension/pull/5>`_
261+
- Merged
262+
* - 2
263+
- Gitlab Change
264+
- `aboutcode-org/vulnerablecode#1524 <https://github.com/aboutcode-org/vulnerablecode/pull/1524>`_
265+
- Merged
266+
* - 3
267+
- Backend Integration
268+
- `aboutcode-org/vulntotal-extension#6 <https://github.com/aboutcode-org/vulntotal-extension/pull/6>`_
269+
- Merged
270+
* - 4
271+
- Build Scripts and Readme
272+
- `aboutcode-org/vulntotal-extension#9 <https://github.com/aboutcode-org/vulntotal-extension/pull/9>`_
273+
- Merged
274+
275+
Closing Thoughts
276+
-------------------
277+
278+
I really enjoyed working on such a challenging project. I loved exploring
279+
the new technologies I used, like WebAssembly, and getting it running in a
280+
complex project with packages and multiple files like VulnTotal.
281+
282+
I appreciated the weekly status calls and the feedback I received from my
283+
mentors and the amazing team. They were really helpful and supportive.
284+
- `Philippe Ombredanne <https://github.com/pombredanne>`_
285+
- `Ayan Sinha Mahapatra <https://github.com/AyanSinhaMahapatra>`_
286+
- `Ziad Hany <https://github.com/ziadhany>`_
287+
- `Keshav Priyadarshi <https://github.com/keshav-space>`_
288+
- `Omkar Phansopkar <https://github.com/OmkarPh>`_

0 commit comments

Comments
 (0)