💉 Prompt injection

[hejny]

prevent by samples?

AI Firewall

[hejny]

#17

[painter99]

Myšlenka:

1. Validační nástroje (a knihovny): LLM-Guard pro základní ochranu nebo odnože jako Guardian AI, Prompt Shield, LangKit
2. Middleware jako: Rebuff, AI Safety Filters, LangChain Guards, Giskard AI
3. Frameworky pro testování bezpečnosti: OWASP LLM Top 10 jako metodologi nebo odnože LLM-Bench, PromptInject
4. Méně vhodné jsou ještě monitorovací nástroje jako: PromptWatch, LLM-Monitor, Helicone

[hejny]

https://www.youtube.com/watch?v=vTKHB4rkDZM

[hejny]

By @JorgeSquared

https://lovable.dev/projects/3eed7264-7c42-481b-957b-8ed478a2ec3b

[JorgeSquared]

@hejny Let's discuss and sort themes. It is possible that some of the themes should be discussed in other discussions within Promptbook organization.

Questions

1. What is the purpose of the page: https://github.com/JorgeSquared/promptbook-safe-worlds

• promote Promptbook as a tool for safe prompt processing
• validate the theme itself (does anybody want it?)
• something other: what?

2. Which of these theme fits another discussion (and which one?):

• Tools / Function‑Calling Hijack (a.k.a. “Function Injection”)
• “Prison Escape”
• Context Poisoning
• Instruction Leak
• Out‑of‑Context
• Vendor Smell
• LLM Grooming
• Cognitive Safety

3. What exactly do you mean by:

• Function Injection meaning: When an LLM is given the ability to call external tools or functions (e.g. “book a flight” or “run code”), an attacker can craft prompts that redirect those calls to unintended targets or malicious payloads.
• “Prison Escape” meaning: LLMs are often constrained by a “system” or “sandbox” prompt that tells them what not to do. A “prison escape” attack seeks to bypass those guardrails.
• Context Poisoning: An attacker injects malicious or misleading content into the context window so that subsequent model outputs are skewed or corrupted.
• Instruction Leak: Here the concern is that secret or sensitive instructions—for example your proprietary “system” prompts—leak out in the model’s responses.
• Out‑of‑Context: “Out‑of‑context” refers to misinterpretation of user instructions because the model lacks the real‑world background it needs. It can also describe when a chunk of conversation is fed to the model without proper framing.
• Vendor Smell: A “vendor smell” is a red flag that indicates your LLM provider’s solution may have hidden pitfalls—be it overly broad permissions, opaque pricing, or one‑size‑fits‑all safety features.
• LLM Grooming: https://www.linkedin.com/posts/hanacepova_zat%C3%ADmco-sv%C4%9Bt-nad%C5%A1en%C4%9B-testuje-nov%C3%A9-ai-modely-activity-7319665957856030720-t_Fg?utm_source=share&utm_medium=member_desktop&rcm=ACoAAB3d6hwBNk9xwhaSn5hoWTatZ-YB2zeYIQk
• Cognitive Safety: https://www.linkedin.com/posts/jan-ma%C5%A1ek-29652418_brainbreakfast-brainbreakfastbusiness-brainbreakfastdigestiv-activity-7317934480558559233-wNW7?utm_source=share&utm_medium=member_desktop&rcm=ACoAAB3d6hwBNk9xwhaSn5hoWTatZ-YB2zeYIQk

[JorgeSquared]

Another interesting thing is the theme of false (self-)validation and self-regulation: https://www.linkedin.com/posts/mrdavediamond_chatgpt-told-me-i-was-brilliant-and-i-activity-7318390230959407105-CwQ8?utm_source=share&utm_medium=member_desktop&rcm=ACoAAB3d6hwBNk9xwhaSn5hoWTatZ-YB2zeYIQk

Another, loosely connected theme is that of human well-being in the AI era: https://medium.com/aimonks/the-dark-side-of-ai-companionship-lessons-from-sewell-setzers-tragic-story-34e4505d9da2

Where do these themes fit in and how does Promptbook mitigate the risks?

[JorgeSquared]

@hejny By the way, the domain that could be connected with the "safe-worlds" webpage above could read eldamar.io, eldamar.ai or eldamar.ptbk.io.

Explanation: In Tolkien’s lore, Eldamar is the blessed region on the shores of Valinor where the Valar first set foot in Middle‑earth’s “Undying Lands.” It conveys the same sense of inviolate safety and purity as Valinor, but with its own distinct ring. ;-)