CWE-1427—Improper Neutralization of Input Used for LLM Prompting

名称

Improper Neutralization of Input Used for LLM Prompting

描述

The product uses externally-provided data to build prompts provided to large language models (LLMs), but the way these prompts are constructed causes the LLM to fail to distinguish between user-supplied inputs and developer provided system directives.

When prompts are constructed using externally controllable data, it is often possible to cause an LLM to ignore the original guidance provided by its creators (known as the "system prompt") by inserting malicious instructions in plain human language or using bypasses such as special characters or tags. Because LLMs are designed to treat all instructions as legitimate, there is often no way for the model to differentiate between what prompt language is malicious when it performs inference and returns data. Many LLM systems incorporate data from other adjacent products or external data sources like Wikipedia using API calls and retrieval augmented generation (RAG). Any external sources in use that may contain untrusted data should also be considered potentially malicious.

常见后果

范围：

Confidentiality, Integrity, Availability

影响：

Execute Unauthorized Code or Commands, Varies by Context

注释：

The consequences are entirely contextual, depending on the system that the model is integrated into. For example, the consequence could include output that would not have been desired by the model designer, such as using racial slurs. On the other hand, if the output is attached to a code interpreter, remote code execution (RCE) could result.

范围：

Confidentiality

影响：

Read Application Data

注释：

An attacker might be able to extract sensitive information from the model.

范围：

Integrity

影响：

Modify Application Data, Execute Unauthorized Code or Commands

注释：

The extent to which integrity can be impacted is dependent on the LLM application use case.

范围：

Access Control

影响：

Read Application Data, Modify Application Data, Gain Privileges or Assume Identity

注释：

The extent to which access control can be impacted is dependent on the LLM application use case.

相关 CWE

• CWE-77Improper Neutralization of Special Elements used in a Command ('Command Injection')High