PromptQL Design

Challenges with in-context tool chaining

Challenge #1: Accuracy & repeatability deteriorate as the instruction complexity increases and/or the amount of data in-context increases.

Ignores rule that advanced plan > free plan in priority.

Challenge #2: In-context approaches risk running into hard LLM limitations around input and output token limits. For example, if retrieved data from a data tool call needs to be passed to a code execution tool, the entire data has to be printed as a variable into the source code of the program. Or, the response of a data retrieval tool call might result in crossing the size of the context window.

Passing data between tools in context runs against the output token limit

Introducing PromptQL

PromptQL's goal is to ensure that AI connected to data and systems can realistically be introduced into business operations & workflows. This requires a high degree of accuracy and repeatability.

To do this, PromptQL separates the creation of a query plan that describes the interaction with the business data, from the execution of the query plan.

PromptQL uses the LLM to first create a program to compose the different tool calls. PromptQL has programmatic primitives for LLM tasks. This allows retrieval, computational and "cognitive" tasks to be arbitrarily composed.

Once the program is created, it is executed in code.

This approach has a few important implications:

1. It removes input and output data generated during the execution of the plan from the current context.
2. Programmatic execution of the desired plan makes it deterministic and repeatable.
3. It allows the user to steer the generation of the plan.

There are 3 key components of PromptQL:

1. PromptQL programs are Python programs that read & write data via python functions. PromptQL programs are generated by LLMs.
2. PromptQL primitives are LLM primitives that are available as python functions in the PromptQL program to perform common "AI" tasks on data.
3. PromptQL artifacts are stores of data and can be referenced from PromptQL programs. PromptQL programs can create artifacts.

A PromptQL agent creates and runs PromptQL programs.

PromptQL programs

A PromptQL program is a concrete representation of the user's intended interaction with the business data.

# Fetch last 10 unread emails
emails = fetch_emails(limit=10, unread=True)

# Calculate average
average = sum(emails) / len(emails)

PromptQL programs can read/write data, or search through data by invoking Python functions. These tools are implemented outside of PromptQL, and should simply be provided as dependencies to the PromptQL program. Here are some typical examples of tools:

1. Search (vector, attribute, keyword, etc)
2. Reading and writing data from a database
3. Interacting with an API

The most important factor that determines the effectiveness of PromptQL are:

1. The quality of the tools provided to PromptQL
2. The ability of the PromptQL agent to consistently generate high quality PromptQL programs

PromptQL primitives

PromptQL primitives are AI functions that are available as python functions in the PromptQL program to perform common AI tasks on data. These primitives can create structured information from unstructured and structured data and allow the composition of "cognitive" tasks with "computational" tasks.

For example, a simple search based RAG system can be represented as a PromptQL program with a retrieval function followed by a PromptQL primitive that then generates the result.

These are the 3 basic PromptQL primitives:

1. classify
2. summarize
3. extract

# A PromptQL program that uses AI to analyze an email

email_text = """From: [email protected]
Date: Thu, 14 Mar 2024 15:23:47 +0000
To: [email protected]
Subject: URGENT: Your Assistance Required - $25M Inheritance

Dear Beloved Friend,

I am Prince Mohammed Ibrahim, the son of late King Ibrahim of Nigeria. I am writing to request your urgent assistance in transferring the sum of $25,000,000 (Twenty-Five Million United States Dollars) from my father's account to your account.

As the sole heir to the throne and my father's fortune, I need a trusted foreign partner to help move these funds out of the country due to political instability. In return for your assistance, I am prepared to offer you 25% of the total sum.

To proceed, I only require:
1. Your full name
2. Bank account details
3. A small processing fee of $1,000

Please treat this matter with utmost confidentiality and respond urgently.

Best regards,
Prince Mohammed Ibrahim
Royal Family of Nigeria
Tel: +234 801 234 5678
"""

# Extract the sender's email using the extract primitive
json_schema = {
    "type": "object",
    "properties": {
        "sender_email": {
            "type": "string",
            "description": "The email address of the sender"
        }
    }
}

extracted_info = primitives_extract(
    json_schema=json_schema,
    instructions="Extract the sender's email address from the email header (From: field)",
    input=email_text
)

# Classify if it's spam
classification = primitives_classify(
    instructions="Determine if this email is likely to be spam/scam based on its content, tone, and characteristics",
    inputs_to_classify=[email_text],
    categories=['Likely Spam', 'Legitimate Email'],
    allow_multiple=False
)

# Store results in an artifact
result = [{
    'email_content': email_text,
    'extracted_sender': extracted_info.get('sender_email'),
    'spam_classification': classification[0]
}]

print(result)

PromptQL artifacts

PromptQL artifacts are stores of data and can be referenced from PromptQL programs. PromptQL programs can create artifacts. These are the 2 necessary artifacts for PromptQL:

1. Text artifacts
2. Table artifacts

# Fetch last 10 unread emails
emails = fetch_emails(limit=10, unread=True)

# Create a list with one dictionary containing the emails
result = []
for email in emails:
    result.append({
        'email': email
    })

# Store as an artifact
store_table_artifact(
    'emails',
    'Last 10 unread emails',
    'table',
    result
)

PromptQL programs can load previously generated artifacts, and this creates a form of structured memory for PromptQL programs, allowing them to surpass limitations introduced by passing data in context.

# Create a function to extract the sender's email address from an email
def get_sender_email(email):
  extracted_info = primitives_extract(
      json_schema=json_schema,
      instructions="Extract the sender's email address from the email header (From: field)",
      input=email
  )
  return extracted_info.get('sender_email')

# Create a function to classify an email as spam/scam
def classify_email(email):
  classification = primitives_classify(
    instructions="Determine if this email is likely to be spam/scam based on its content, tone, and characteristics",
    inputs_to_classify=[email],
    categories=['Likely Spam', 'Legitimate Email'],
    allow_multiple=False
  )
  return classification[0]

emails = get_table_artifact('emails')

# For each email, extract the sender's email address and classify if it's spam
result = []
for email in emails:
    result.append({
        'email': email,
        'sender_email': get_sender_email(email),
        'spam_classification': classify_email(email)
    })

# Run an action to mark the email as spam
for email in result:
    if email['spam_classification'] == 'Likely Spam':
        mark_email_as_spam(email['email'])