@gimmechocolate @volpeon like ultimately it falls onto the developer to verify the work performed by the tools that they use, and to adjust the code accordingly

@errant From what I've seen from the developers, they're well aware of the risks and capabilities of AI. You aren't wrong that careless (non-)developers are too confident in AI, but this doesn't imply the inverse: that all users of AI are automatically careless. As long as they consistently demonstrate responsible use of it, I personally see no problem from a security standpoint.

@charlotte @volpeon
So like, first off, you're clearly responding to what you thought I'd say, and not what I actually said, because I didn't even mention lack of awareness.

Second, copy-paste is actually notoriously bad for creating errors, and it's typically considered good practice to use things like constants, macros, and functions instead. And that's again, ignoring the fact that copy-paste doesn't have any of the problems I mentioned in my post.

@Tvorsk @volpeon I liked the design of KPXC and I'm not a fan of the old look of KP2, but that's a thing of personal taste. If you like KP2, then that's absolutely awesome!

@SteffoSpieler @volpeon Yeah, kind of matter of taste for sure... just sayin' it's an option.
I believe it even runs under dotnet runtime on linux, although that'd be... probably not very convenient.

@charlotte @volpeon yeah the people on that github issue were bizarrely saying things like software is untestable, like you couldn't write a test to verify correctness because the nefarious ai can insert tricks that human cognition weaknesses cannot detect

@gimmechocolate @volpeon feelings imply awareness, no? but copy-paste doesn’t have feelings either.

and like yeah it does create issues. i am aware that it creates issues. that is specifically why i mention it. it is still an occasionally useful tool which can be leveraged by programmers. The person liable for (mis)use is the developer using the tool, and copy-paste errors are also something that may not be immediately obvious in a code review either

@volpeon@icy.wyvern.rip me trying to explain to people that just because it's open source doesn't mean it's automatically secure.

Just because you can read the source doesn't mean any single individual has complete understanding of the entire source code and even if that was possible, silence can be bought.

Open source fanatics are clueless sometimes.

@sun @charlotte @volpeon and the actual issue was some basic straightforward JSON import with tests

@volpeon yes, this.

@volpeon @errant AI is really easy to accidentally prompt in such a way that it goes off the rails, can make all sorts of mistakes and copyright issues (even subtle ones), and can do it at scale. Hypothetically it might be possible to use it in such a way to not trigger these issues, but surely all the checking and prerequisite expertise would nix most advantages of using it?

And it looks like for most AI PRs in KeepassXC, the person working with the AI ultimately approves the code (example: https://github.com/keepassxreboot/keepassxc/pull/12588)... hardly a rock solid review process. Usually, there's two humans in the loop.

@sitcom_nemesis @errant
> but surely all the checking and prerequisite expertise would nix most advantages of using it?

Sure, but why would this be a concern for anyone but the user themself? I'm sure I use things which other people may not like, such as VSCode or GNOME. Is it valid for them to tell me what to use and how?
> And it looks like for most AI PRs in KeepassXC, the person working with the AI ultimately approves the code... hardly a rock solid review process. Usually, there's two humans in the loop.

The way the AI is integrated in GitHub makes it a separate entity from the reviewer with an interaction workflow akin to iterating a PR with its author until it matches the project's standards. In both cases, the PR author — AI or human — is untrustworthy and the reviewer is trustworthy. There are also non-AI PRs where only one developer conducted the review, so there's no difference between AI and non-AI standards.

If this strikes you as flawed, then your concerns should lie with the review process itself.

@charlotte @volpeon
Yeah, but no feelings wasn't a criticism of LLMs, I was saying it does actually behave more or less like what you said but without the malice part. To be clear it's bad because it makes code that looks good, but isn't — it's just mashes up the semantic ideas from its training data without regard for how those different patterns will interact; the result will always *look* right, but won't necessarily be right, often for very subtle reasons, making it a poor tool that people simply shouldn't use. I wouldn't trust a project that put instructions on how to copy-paste from stackoverflow in their repo either.

As for the tool thing, like, I can see ways where a LLM could be a useful tool. For example, it's very annoying to me how the results from the LSP autocomplete are in some arbitrary order, and it could be nice to have an LLM step in and rank em!! It's a tool, but like, only as a part of a well-designed machine-learning pipeline, made by the people who understand the limits of the technology — not something used by just about everyone almost raw.

The whole thing is just annoying to me though — I've been interested in machine learning for over a decade now, and when I took my first course in it, I was warned about this exact scenario. There's a cycle of AI spring and winter, where the AI business leaders push the idea that these tools can do anything. People metaphorically saying "Uhm, when all you have is a hammer, actually you can solve all your problems by whacking everything with it.". Now I get to see what they were talking about firsthand and it's extremely frustrating to feel like I am shouting it from the rooftops and no one listens.

@sun @charlotte @volpeon i mean in my experience the pattern matching leading to outputting stuff that is very good at looking correct during ocular inspection but isn't correct is very real

@volpeon @errant

The way the AI is integrated in GitHub makes it a separate entity from the reviewer with an interaction workflow akin to iterating a PR with its author until it matches the project's standards. In both cases, the PR author — AI or human — is untrustworthy and the reviewer is trustworthy.

I'd argue that integrating AI into GitHub this way is part of the problem. It's not an agent - it's a word guessing machine with access to an API. It fundamentally doesn't think like a human, trustworthy or otherwise. We have methods of understanding context, intention and trustworthiness with other humans - AI strips that all away while still claiming to be analogous to a human. That's, in part, what makes it so risky.

It's one thing to allow AI code with the caveat that the human needs to take full responsibility (e.g. the Fedora guidelines), but that doesn't seem to be happening with KeepassXC. Hence the concern.

@sitcom_nemesis @errant

It doesn't matter what the AI is or isn't. GitHub's presentation of it leads to the AI having the same status as an external contributor, which incentivizes the thinking that it must be held up to the same standards as human contributors. I'd even say that this is the only healthy approach because implicitly trusting humans more means that in case they use AI outside of GitHub and act like it's their own work, your own bias may prevent you from seeing flaws you'd pay closer attention to when reviewing an AI's output.

I'm not sure what the problem with responsibility is. Isn't this a project governance issue, i.e. "you take full responsibility" means you'll get kicked out if you contribute garbage AI code? How is this a security problem?

I saw this on my fedi timeline. I don't think people need to flat-out lie just to criticise the use of AI generated code - UZDoom going for the copyright/GPL angle during that whole thing made a lot more sense IMO.

From what I saw, KeePassXC implements the same policy as Mesa and Fedora, so I don't see what the problem is.

That being said, the code of my password manager of choice is proprietary, they could very well be using AI generated code and I wouldn't even know (or care, since I trust the company enough).

i don't even use a password manager and use firefox's internal password manager thing, which is A BAD IDEA and i need to change it sooner

@sun @charlotte @volpeon ppl can't be that schewpid