Replies: 4 comments 6 replies
-
|
Some of my findings are:
The more energy the AI has to spend processing the prompt / writing the response, the less it will have to do the job. Once I get a working prompt, I run experiments trimming it (just like with neural networks) to find how much I can remove without affecting the results.
Dummy example ahead which can get very costly because it may have many instances The second example may have longer text on the input (json schema), but it will get N times shorter on the output. As of 2024-03-08, openai output tokens are 3 times the cost of input tokens. But more important than that is the 'employ your energy on the task more than on writing the results' . I'd like to see if someone has found the opposite to be true (maybe it does in some cases). |
Beta Was this translation helpful? Give feedback.
-
|
I asked to Perplexity: and this was the answer. Based on the search results, it appears that there are some issues and challenges with using the OpenAI API's function call feature, which can result in lower quality outputs compared to using the API for plain text generation. Here are the key points:
In summary, the OpenAI API's function call feature appears to have some limitations that can lead to lower quality outputs compared to plain text generation. Users should be aware of these challenges and be prepared to experiment to find the best approach for their use case. [1][2][3][4] Citations: I asked that because I have a prompt that works very well with text output, but with function call (a very simple one) has some chance of doing mistakes (in the content, not the format). And I thought, If I combine |
Beta Was this translation helpful? Give feedback.
-
|
The last proposed approach has worked very well for me. I use it to make the LLM chew the Input, chew it again, and then output the result in plain text, and on the last call, I capture the output with Instructor. Only responseN is an Object, the previous ones are text. This is a format that works very well for the output, that will be sent to the next call. This usually produces something like As you may have noticed, it is very similar to Pydantic. |
Beta Was this translation helpful? Give feedback.
-
|
One rule I learnt interacting with the LLM is this:
You may think, But it is easy to forget (for me) when creating prompt+output_templates. What happens if you forget the rule? The LLM will respond, with rubbish / overthinking / hallucination As far as I know you cannot include conditions while creating the class. However you can build the Example: Which produces And then, conditionally increase it: Which produces: |
Beta Was this translation helpful? Give feedback.
-
|
Did you know .... ? That given a Pydantic model (example) from pydantic import BaseModel
from typing import List, Literal
class UserProfile(BaseModel):
username: str
email: str
is_active: bool
roles: List[Literal['admin', 'editor', 'viewer']]
tags: List[str]you can ask the LLM to convert it to YAML UserProfile:
type: object
properties:
username: { type: string }
email: { type: string }
is_active: { type: boolean }
roles:
type: array
items: { type: string, enum: ['admin', 'editor', 'viewer'] }
tags:
type: array
items: { type: string }
required: [username, email, is_active, roles, tags]and then, at the end of your prompt you can use that (adding comments where required/beneficial)
Output and Format:UserProfile:
type: object
properties:
username: { type: string } # User's unique identifier
email: { type: string }
is_active: { type: boolean }
roles: # < SOME TEXT EXPLAINING YOUR REQUIREMENTS/CONDITIONS/ETC. >
type: array
items: { type: string, enum: ['admin', 'editor', 'viewer'] } # Allowed roles
tags: # < SOME TEXT EXPLAINING YOUR REQUIREMENTS/CONDITIONS/ETC. >
type: array
items: { type: string }
required: [username, email, is_active, roles, tags]and the LLM will follow that format without having to use Example output. username: johndoe
email: johndoe@example.com
is_active: true
roles: [admin, editor]
tags: [python, pydantic]Why would this be useful?:
Once your prompt design is solid ... I know this would throw away the While troubleshooting, you can add a additional_comments: { type: string } # Write here anything you want to say if you find problems completing the task, or you find it unclear, ambiguous, .... blah blah blah .... to the output format so that the LLM can have a place to vent out its frustration with your prompt :o| |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if I understood. By converting to YAML you are giving this as part of the prompt instead of the response_model argument? |
Beta Was this translation helpful? Give feedback.
-
|
Correct, the promp itself has a specification of the format. That specification could be json, markdown, yaml or whatever works for you. I found out yaml works well (very light and almost natural language). The formatting of the markdown here makes it difficult to see it clearly. Maybe this is better Here you have a paper about formats. However, one (I think very important) thing that paper misses, is to BE EXPLICIT. Even worse if the complex data is not enclosed with ``` |
Beta Was this translation helpful? Give feedback.
-
|
But would this be more effective than using response_model? I've always thought that providing a pydantic model would be somehow lead to better results when compared to explicity asking the output format in the system prompt |
Beta Was this translation helpful? Give feedback.
-
|
Asking for reasoning on the outputs is something I have been doing as:
But recently I have opted for 'requesting & storing the output in full (with reasoning)' but when retrieving and showing the examples (k-shots) only show the input & output without reasoning. I noticed that the reasoning of the examples was influencing too much the reasoning and value of the output, which was detrimental. Plus less distraction as there is less to read/parse. Output and Format:But then, the examples only show the Input and this part of the Output: Note: Each code section uses ``` yaml to specify the format of what is inside. |
Beta Was this translation helpful? Give feedback.
-
|
I've already tried that before. I would also recommend switch both, first reasoning and then value. That way it would work in a similar way to CoT. |
Beta Was this translation helpful? Give feedback.
-
There are lots of literature about Prompts, but this is (in my opinion) a slightly different game.
With Instructor, you have the prompt and you have the class definition that you pass to the LLM/AI/openai call.
Both need to be aligned and one needs to consider the extra cost (in tokens) of processing the JSON schema together with generating the output that fulfills that class definition.
I am sure each one has found some tricks/caveats/quirks and sharing them would/may provide some light to others.
Beta Was this translation helpful? Give feedback.
All reactions