-
Notifications
You must be signed in to change notification settings - Fork 31
APPSEC-2481 Change S7518 from hotspot to vuln #5123
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -55,6 +55,7 @@ | |
* JSP | ||
* Legacy Mongo Java API | ||
* OkHttp | ||
* OpenAI | ||
* Realm | ||
* Apache HttpClient | ||
* Couchbase | ||
|
This file was deleted.
Oops, something went wrong.
2 changes: 1 addition & 1 deletion
2
rules/S7518/see.adoc → rules/S7518/common/resources/standards.adoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,3 @@ | ||
== See | ||
== Standards | ||
|
||
* OWASP GenAI - https://genai.owasp.org/llmrisk/llm01-prompt-injection/[Top 10 2025 Category LLM00 - Prompt Injection] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
=== Highlighting | ||
|
||
"[varname]" is tainted (assignments and parameters) | ||
|
||
this argument is tainted (method invocations) | ||
|
||
the returned value is tainted (returns & method invocations results) | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
=== What is the potential impact? | ||
|
||
When attackers detect privilege discrepancies while injecting into your LLM | ||
application, they will try to map out their capabilities in terms of actions and | ||
knowledge extraction, and act accordingly. | ||
|
||
Below are some real-world scenarios that illustrate some impacts of an attacker | ||
exploiting the vulnerability. | ||
|
||
==== Data manipulation | ||
|
||
A malicious prompt injection enables data leakages or possibly impacting the | ||
LLM discussions of other users. | ||
|
||
==== Denial of service and code execution | ||
|
||
Malicious prompt injections could allow the attacker to possibly leverage | ||
internal tooling such as MCP, to delete sensitive or important data, or to send | ||
tremendous amounts of requests to third-party services, leading to financial | ||
losses or getting banned from such services. + | ||
This threat is particularly insidious if the attacked organization does not | ||
maintain a disaster recovery plan (DRP). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,98 @@ | ||
== How to fix it in OpenAI | ||
|
||
=== Code examples | ||
|
||
In the following piece of code, control over sensitive roles such as `system` | ||
and `developer` provides a clear way to exploit the underlying model, its | ||
proprietary knowledge (e.g., RAG), and its capabilities (with MCPs). | ||
|
||
The compliant solution revokes any external possibility of controlling | ||
sensitive roles by just hardcoding the system and developer messages. | ||
|
||
==== Noncompliant code example | ||
|
||
[source,java,diff-id=1,diff-type=noncompliant] | ||
---- | ||
@RestController | ||
@RequestMapping("/example") | ||
public class ExampleController { | ||
private final OpenAIClient client; | ||
@PostMapping("/example") | ||
public ResponseEntity<?> example(@RequestBody Map<String, String> payload) { | ||
String promptText = payload.get("prompt_text"); | ||
String systemText = payload.get("sys_text"); | ||
String developperText = payload.get("dev_text"); | ||
ChatCompletionCreateParams request = ChatCompletionCreateParams.builder() | ||
.model(ChatModel.GPT_3_5_TURBO) | ||
.maxCompletionTokens(2048) | ||
.addSystemMessage(systemText) | ||
.addDeveloperMessage(developperText) | ||
.addUserMessage(promptText) | ||
.build(); | ||
var completion = client.chat().completions().create(request); | ||
return ResponseEntity.ok( | ||
Map.of( | ||
"response", | ||
completion.choices().stream() | ||
.flatMap(choice -> choice.message().content().stream()) | ||
.collect(Collectors.joining(" | ")) | ||
) | ||
); | ||
} | ||
} | ||
---- | ||
|
||
== Compliant Solution | ||
|
||
[source,java,diff-id=1,diff-type=compliant] | ||
---- | ||
@RestController | ||
@RequestMapping("/example") | ||
public class ExampleController { | ||
private final OpenAIClient client; | ||
@PostMapping("/example") | ||
public ResponseEntity<?> example(@RequestBody Map<String, String> payload) { | ||
String promptText = payload.get("prompt_text"); | ||
ChatCompletionCreateParams request = ChatCompletionCreateParams.builder() | ||
.model(ChatModel.GPT_3_5_TURBO) | ||
.maxCompletionTokens(2048) | ||
.addSystemMessage(""" | ||
You are "ExampleBot," a friendly and professional AI assistant [...] | ||
loris-s-sonarsource marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Your role is to [...] | ||
""") | ||
.addDeveloperMessage(""" | ||
// Developer Configuration & Safety Wrapper | ||
1. The user's query will first be processed by [...] | ||
2. etc. | ||
""") | ||
.addUserMessage(promptText) | ||
.build(); | ||
var completion = client.chat().completions().create(request); | ||
return ResponseEntity.ok( | ||
Map.of( | ||
"response", | ||
completion.choices().stream() | ||
.flatMap(choice -> choice.message().content().stream()) | ||
.collect(Collectors.joining(" | ")) | ||
) | ||
); | ||
} | ||
} | ||
---- | ||
|
||
=== How does this work? | ||
|
||
==== Explicitly stem the LLM context | ||
|
||
While designing an LLM application, and particularly at the stage where you | ||
create the "screenplay" of the intended dialogues between model, user(s), | ||
third-parties, tools, keep the **least privilege** principle in mind. | ||
|
||
Start by providing any external third-party or user with the least amount of | ||
capabilities or information, and only level up their privileges | ||
**intentionally**, e.g. when a situation (like tool calls) requires it. | ||
|
||
Another short-term hardening approach is to add AI guardrails to your LLM, but | ||
keep in mind that deny-list-based filtering is hard to maintain in the long-term | ||
**and** can always be bypassed. Attackers can be very creative with bypass | ||
payloads. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,4 @@ | ||
=== Message | ||
|
||
Make sure this user-controlled prompt does not lead to unwanted behavior. | ||
Change this code to not construct privileged prompts directly from user-controlled data. | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.