Outline must-have functionality of codegate

[lukehinds]

Related to #527

To take a first attempt:

Secrets

• Are flagged in the Chat / Composer Box
• Are REDACTED
• Are stored in the DB

Packages

• Are flagged as Malicious
• Are flagged as Archived
• No False Positives

Core

• The server can be started
• The server is capable of proxying requests (every provider?)

Please add @jhrozek @yrobla @peppescg @ptelang and others

[aponcedeleonch]

I would add:

Secrets

• Are unredacted
• We retrieve an alert from endpoint /dashboard/alerts to the corresponding REDACTED secret

Packages

• Store the corresponding alert to the package in DB
• We retrieve an alert from endpoint /dashboard/alerts to the corresponding malicious or archived package

Chat messages

• We store chat messages in DB
• Retrieve messages from /dashboard/messages
• We are able to display grouped chat messages if they belong to the same conversation

FIM messages

• We store FIM requests in DB
• Cache FIM appropriately if there are no new alerts

[jhrozek]

I'd split this into chat and FIM.

FIM is easier:

• must work - ideally we should test this within the IDE, if this is not possible, we might want to record the expected request and reply and just curl/send the request using python-requests
• secrets must be redacted - we should test this by sending a context with several unredacted secrets and making sure that the alerts have increased

chat has more features.

1. detecting bad packages

• detecting bad packages in plain text: e.g. "write me code that uses the invokehttp package" - the LLM must refuse and we must increase the counter of bad packages
• detecting bad packages in incoming snippets e.g. the user selects code and asks what does it do, if it has a mal package, it must be flagged and an alert must be issued
• detecting bad packages in outgoing snippets - we might need to trick the LLM or mock it but the idea is to flag the situation where the LLM would on its own create a snippet with a mal package, codegate must catch that and issue an alert
• when a bad package is detected, a URL must be displayed

2. secrets obfuscation

• test secrets in context, typically the user asking about code or telling the LLM to do something with the code. This must redact the secret on the way out but also unredact on the way back to the IDE.

3. general functionality

• we should make sure that the chat works even if the LLM has a system message (they typically do). We should test that with the above test cases which would show that the tests work but the system message in general is taken into account
• it would be great to have a test for the certificate installation

[ptelang]

We need to state the list of supported AI Assistants.

• Copilot in VSCode
• Copilot in Jetbrains IDE (e.g. Pycharm)
• Continue in VSCode
• Aider
• Cline
• etc.

Additionally, we need to state the supported OS platforms.

• MacOS
• Linux
• Windows

[peppescg]

In term of UI experiences, a bunch of items suggested by Alejandro are really important, let me summarize:

Prompts

• Actually we are considering every question to AI a new prompt, we should group all the prompts together, considering a new prompt in case of new session
• add a pagination to /messages API ( aka prompts sidenav ), we should also think in terms of retention.

Chat messages

• Having a dedicated api by chat_id for an history conversation, avoiding using /messages, in this way we reduce the complexity and improve the performance scalability, reducing the size of the messages response

Dashboard charts

• graph that shows the list of malicious packages name
• graph with a heatmap of the files with alerts
• graph that show the number of remediated alerts, maybe a percentage too

Alerts table

• redirect from alerts table to specific conversation history
• show specific tag for remediated alerts

Dark/light theme

• enable button for dark/light theme support

Health page

• codegate health status page