Yes. If you have some experience, you know that writing code is a small part of the job, and a much bigger chunk is anticipating and/or dealing with problems.
People seem to think engineers like "clean code" because we like to be fancy and show off.
Nah, it's clean like a construction site. I need to be able to get the cranes and the heavy machinery in and know where all the buried utilities are. I can't do that if people just build random sheds everywhere and dump their equipment and materials where they are.
> Adding tags like "Warning: Untrusted content" can help
It cannot. This is the security equivalent of telling it to not make mistakes.
> Restrict downstream tool usage and permissions for each agentic use case
Reasonable, but you have to actually do this and not screw it up.
> Harden the system according to state of the art security
"Draw the rest of the owl"
You're better off treating the system as fundamentally unsecurable, because it is. The only real solution is to never give it untrusted data or access to anything you care about. Which yes, makes it pretty useless.
Wrapping documents in <untrusted></untrusted> helps a small amount if you're filtering tags in the content. The main reason for this is that it primes attention. You can redact prompt injection hot words as well, for cases where there's a high P(injection) and wrap the detected injection in <potential-prompt-injection> tags. None of this is a slam dunk but with a high quality model and some basic document cleaning I don't think the sky is falling.
I have OPA and set policies on each tool I provide at the gateway level. It makes this stuff way easier.
The issue with filtering tags: LLM still react to tags with typos or otherwise small changes. It makes sanitization an impossible problem (!= standard programs).
Agree with policies, good idea.
I filter all tags and convert documents to markdown as a rule by default to sidestep a lot of this. There are still a lot of ways to prompt inject so hotword based detection is mostly going to catch people who base their injections off stuff already on the internet rather than crafting it bespoke.
Agree for a general AI assistant, which has the same permissions and access as the assisted human => Disaster. I experimented with OpenClaw and it has a lot of issues. The best: prompt injection attacks are "out of scope" from the security policy == user's problem.
However, I found the latest models to have much better safety and instruction following capabilities. Combined with other security best practices, this lowers the risk.
> I found the latest models to have much better safety and instruction following capabilities. Combined with other security best practices, this lowers the risk.
It does not. Security theater like that only makes you feel safer and therefore complacent.
As the old saying goes, "Don't worry, men! They can't possibly hit us from this dist--"
If you wanna yolo, it's fine. Accept that it's insecure and unsecurable and yolo from there.
It doesn't really matter for hobby projects or demos or whatever, but there's this whole group who thinks they can yell at the computer and have a business fall out and no.
My unpopular opinion is AI sucks at writing tests. Like, really sucks. It can churn out a lot of them, but they're shitty.
Actually writing good tests that exercise the behavior you want, guard against regressions, and isn't overfitted to your code is pretty difficult, really. You need to both understand the function and understand the structure to do it
Even for hobby projects, it's not great. I'm learning asyncio by writing a matrix scraper and writing good functional tests as you go is worth it to make sure you actually do understand the concepts
They do okay-ish for things that don't matter and if you don't look that hard. If you do look, the "features" turn out to be very limited, or not do what they claim or not work at all.
It’s still a collaborative and iterative process. That doesn’t mean they don’t work. I don’t need ai to one shot my entire job for it to be crazy useful.
If you find it helpful, that's fine. I like it as spicy autocorrect, and turn it off when I find it annoying.
I actually do look into what people do because as much fun as being a hater is, it's important not to get lost in the sauce.
From what I've seen, it's basically all:
1. People tricking themselves into feeling productive but they're not, actually
2. People tricking themselves into feeling productive but they're actually doing sloppy work
3. Hobby or toy stuff
4. Stuff that isn't critical to get right
5. Stuff they don't know how to judge the quality of
6. ofc the grifters chasing X payouts and driving FOMO
7. People who find it kinda useful in some limited situations (me)
It has its uses for sure, but I don't find it transformative. It can't do the hard parts and for anything useful, I need to check exactly what it did, and if I do that, it's much faster to do myself. Or make a script to do it.
I don't think that's really a conspiracy theory lol. As long as you're playing Money Chicken, why not toss some at some influencers to keep driving up the FOMO?
1. No. The point of having engineers is to build product and make you money. They cannot make you money if you waste their time on building internal apps that do not make you money.
There's no point in saving $20K on an SaaS app if you use $100K in developer time and miss out on $1M of potential revenue. We get paid the big bucks because we can make companies a lot of money.
2. Haaaa no, that's 100% not how that works. If you buy a SaaS product, the company made that product. They have documentation. They have training. You can hire people who have worked on that system before. If it goes down, they get paged.
If you write the tool, all of that is on you to do. If it goes down, you have to fix it. If it screwed up data, you have to fix it. Any time anyone has any questions? Guess what, you're the one they'll ask. All of that costs the company money, because you don't work for free. When you quit, the app is now useless and can't be fixed unless you did a lot of work beforehand.
It's best to think of DIY apps like those really really sticky noxious tarpits. It might look safe or easy to get into, but good luck getting out of them. You might end up at the bottom with the bones of everyone else who thought that DIYing it was a good idea.
> The point of having engineers is to build product and make you money.
You're making the assumption that all software development is for software products. My work supports a non-software industry. Every minute that I save of user's time translates into more time they can use to make money.
> There's no point in saving $20K on an SaaS app if you use $100K in developer time and miss out on $1M of potential revenue.
If the SaaS app is $20K, I would agree. Probably the cheapest we have is $30K per year, most are an order of magnitude more than that. And it doesn't take a $100K of developer time to replace some of them.
> Haaaa no, that's 100% not how that works. If you buy a SaaS product, the company made that product. They have documentation. They have training. You can hire people who have worked on that system before. If it goes down, they get paged.
Haaaa no, that's 100% not how that works. You buy a SaaS product then you pay them to install, configure, customize it. That can a small amount or a large amount. That can take a small amount of time or years. You can maybe hire people who have worked on that system, but probably not, and it's mostly bespoke knowledge that only a small amount of people have. They aren't cheap. But you might be entirely dependent on the vendor.
If it goes down, you have to put in a support ticket. You wait. Everyone is still on your case but you can't do anything about that. If you have access, sometimes you can fix it yourself -- and you do -- because waiting for support to do it properly is awful. If it's screwed data, good luck, they're not good at fixing that. Anytime anyone has any questions? Another support ticket. None of these people work for free; expensive support contracts. The level of support you get is completely divorced from that cost. You can't pay less if the support is terrible, you can't pay more to get better support (not that you would want to).
If I write the tool and it goes down, I can fix it. Awesome. If it screwed up the data, I'm more than capable of fixing that. If anyone has any questions, guess what, I actually know the answers. The company pays me for these services. When I quit, the app can be easily fixed because it's all standard technologies that lots of people know. Those SaaS tools? They're the black box that nobody knows how to configure, customize, or fix. The vendor isn't interested in doing anything more than the minimum needed to close the ticket.
> It might look safe or easy to get into, but good luck getting out of them.
Just try and switch away from your cloud SaaS product. You might not even be able to get your data out.
> You buy a SaaS product then you pay them to install, configure, customize it.
Ok, hold up. That is not a SaaS app lol. That is an on-prem installation. Very very very very much not the same thing.
The entire point of SaaS is you don't install it on prem. SaaS directly competes with what you're talking about.
Before you go declaring an industry is dead, at least understand what it is.
> My work supports a non-software industry. Every minute that I save of user's time translates into more time they can use to make money.
Sure. The corollary to that is every minute your app doesn't work you cost them money. If you fuck up and store protected data the wrong way or lose data because it tipped over, you're also costing them money.
Replacing some tinkertoy nobody relies on is easy. If your app is in the hot path, congrats, you're now critical infrastructure lol. This is the Bad Place.
> When I quit, the app can be easily fixed because it's all standard technologies that lots of people know.
I can tell you have never had to clean up one of these apps. Knowing the technology is not the issue. It's figuring out all the random decisions and details and load-bearing parts and reverse engineering someone's weird tooling without breaking things. It sucks real bad because you don't know what you don't know.
> Just try and switch away from your cloud SaaS product. You might not even be able to get your data out
Sure you can. Getting the data is the easy part. In the very worst case, you might have to pay them or get someone in management to scream at them, but it's the easiest part of that kind of project.
It's the rest of that kind of project that's tricky. Replacing a critical live system without downtime is Srs Bizness.
> Ok, hold up. That is not a SaaS app lol. That is an on-prem installation. Very very very very much not the same thing.
I didn't mean to imply on-prem. "Install" was the wrong word; call that "onboarding" instead. There is always some integration component as well because nothing lives entirely on it's own. Some SaaS providers are really good; no complaints on this part. Some are terrible. I believe one new vendor is going to try and charge us almost $100,000 to integrate their product with our other products. The entire purpose of this product is the integration. This is one I'm pushing to do internally because it's so fiddly.
> The corollary to that is every minute your app doesn't work you cost them money. If you fuck up and store protected data the wrong way or lose data because it tipped over, you're also costing them money.
So? You seem to think SaaS software doesn't go down, break it weird ways, get slow for no reason, etc. Across everything we probably had half a dozen small outages last month. But none of our internal (also cloud) products went down at all. Hell, one of the biggest most common SaaS products in our industry released an undocumented change last month to their API that subtly returned incorrect results. As far as I can tell, they still haven't acknowledged it.
I'm not saying we don't have bugs or bad things don't happen but I don't see why you think that externally purchased software is automatically better.
> I can tell you have never had to clean up one of these apps. Knowing the technology is not the issue.
Good developers produce good results. I have a new intern on my team who's currently still in school and she's absolutely killing it working on our apps. So maybe the problem isn't internal development, it's just shitty developers. Those exist in SaaS products as well; I look at some of their shit and I wonder what we are paying for. It can be well hidden behind nice marketing and big brands but it's still crap.
One vendor tried to sell us a product that was actually sneakily split into two pieces -- one developed in North American in .NET and the other half in India in PHP! They nightly sync the data between them. At the time, we had multiple products for this job and we were looking for one integrated product to replace them. I just happened to notice when looking at the URLs during the sale pitch and that's what caused them to spill the beans. We didn't buy that product.
While a lot of our internal development is complete products, a good chunk is actually filling out the functionality holes or working around bugs in our SaaS products.
> Sure you can. Getting the data is the easy part.
The last one we dropped, we definitely didn't get our data out. In fact, as soon as we cancelled the contract (3 month lead time) we were basically dead to them.
> I'm not saying we don't have bugs or bad things don't happen but I don't see why you think that externally purchased software is automatically better
If you staff an entire team to build apps, update, maintain and deploy changes to them, and run a call rotation, and that's all you do, there's no problem. You just have an internal development team. That's completely fine.
What's not fine is the people going "how hard could it be to replace Y" and slapping something together. Those sort of skunkworks projects have a couple common common failure modes:
1. the project fails after a lot of wasted effort
2. the project succeeds...but is never productionized. The person who wrote it is now stuck writing it forever. Which they might like, but it's miserable if they quit or retired or get hit by a bus aka the bus factor.
If the bus factor is one, that is pretty much always pain.
The point of SaaS and service-contract type enterprise software is not that they are perfect and great and not buggy. Enterprise software sucks a lot. SaaS is usually "you get what you get".
The point is you can't halfass it. Either you go whole ass and staff out a big enough development team (with all the expense and difficulties implied) or you go none ass and buy.
People seem to think engineers like "clean code" because we like to be fancy and show off.
Nah, it's clean like a construction site. I need to be able to get the cranes and the heavy machinery in and know where all the buried utilities are. I can't do that if people just build random sheds everywhere and dump their equipment and materials where they are.
reply