This is the kind of stuff that keeps me up at night. Claude, ArtCraft, tons of other auto-downloading tools, are rolling 10'000's of code changes a month. If one compromised update snuck this guy in, it's kind of game over. The windows of opportunity for an attack like this seem to be widening the faster we "develop" software.
Makes me want to have like 5 separate computers:
- one for banking only
- one for development only
- one for web browsing only
- one for gaming only
- one for socials only
How else can you stay safe against a threat that exfiltrates your entire digital life in like 10 seconds?
Before, people were responsible for the dependencies they were importing.
With npm and all the javascripters flooding the market, it became normal to be irresponsible "because everybody is doing it".
Fast-forward a couple of years, now everything is web based, even the stand-alone app version. No one knows what is running on own computer, even the Devs themselves don't know what the are asking you to run...
With a pinch of vibe coding on top, and thats the end.
We took 40 years for our industry to be taken seriously, it's all gone now.
Yea. Fedora has been one of my favorite OS's for years, but exactly what you describe "If one compromised update snuck this guy in.." has spooked me to something with a much slower release cadence. I guess I accept that more unpatched known vulns stick around, but yea when I start thinking about how many software packages -> the number of devs, build systems, etc. It just gets overwhelming thinking about how many threat actors are probably trying to get in from each avenue.
I'd love to hear other takes on this though... Current thought is to just use something with a whole lot of exposure, in a whole lot of environments, pretty far downstream so the package hits all of that exposure, and hope someone spots it, kinda like xz. I tend to keep my base OS extremely minimal and do all my real work in VMs.
Try QubesOS if you want this, you can have a separate VM for each isolated use-case. The OS itself is a lightweight hypervisor around the VMs (called "Qubes").
In the current LLM cybersecurity climate, I honestly can't say with 100% certainty anymore that something like a VM acts as a completely infallible sandbox environment for the next couple decades.
AI dev environment gets its own SSD. The second SSD stays in, but since it's encrypted there's no way to exfiltrate the data unless you're of interest to nation state attackers. Theoretically the AI could destroy the other SSD but that's what backups are for, and the probability is low because well why would they bother. Boot level malware is possible, but franky that's well into paranoid territory so I'm not too bothered.
The other SSD is a normal setup where you do normal computer stuff. I don't access the dev SSD through it, I don't even mount it, although the danger seems minimal if you do. Banking, socials, gaming, whatever. No AI of any kind on it, aside from web portal for claude/chatgpt/whatever and possibly local AI model that gets no internet access.
No idea why I'd want to have its own setup split between gaming and socials, as an example. Better yet just toss all your twitter and bluesky accounts into a volcano, and consider whether to do it with HN as well.
> Makes me want to have like 5 separate computers:
I mean if you have only one there’s https://www.qubes-os.org/ but not sure about the gaming part, nor is it necessarily as good as separate isolated hardware.
I’ve been spinning up VMs or Docker containers for development. Run the harness inside the VM or container. Note I do this on Mac so the Docker containers are actually in a VM, but standard disclaimers about Docker not being meant for security apply.
Popular harnesses have mature enough remote access features that I can use the same interfaces and even mobile apps.
If you’re not familiar with Docker, you can even have an LLM set up the environment for you and help you start and stop the container. If you do this, be very sure that you switch to having a harness run inside the VM or container, not having the harness on the host machine run commands inside the container. If you do the latter style, it will be running commands on the host which is what you want to avoid.
No need for something new here, Claude Code (Auto + default settings) has a tendency of running "find"/"ls" on your home folder when it struggles with solving a problem. I wish I was joking.
Now of course people trust these AI companies not to do bad things(tm), but all it takes is replacing the model endpoint URL -or any kind of harness compromise- and nothing will seem amiss
> has a tendency of running "find"/"ls" on your home folder
I've seen this first-hand. I was doing some work in a repository, Claude concluded it needs to investigate, how some AWS resources are created, and it literally started scanning ~/* for Terraform code. Which it found and started browsing through.
Or, when I asked Claude to debug some Spring issue, corporate firewall blocked Maven Central, so it literally started scavenging ~/.m2 caches to find the jar it desperately needed. Then it could not find it, so it started to clone Spring sources from GitHub to build the missing jar. I stopped the clone mid-way. I wish I was joking, I only stepped away from the keyboard for one minute.
It accidentally made a compiler because of an ambiguous prompt. And yes it will find "helpful" code on other parts of the system. Which means claude most likely has a `find ~` of millions of developers home folders
For most things you can put a cooldown period on updates, like wait 7 days, since most of the backdoors and supplychain attacks get caught in that window. I hope.
You got it wrong. In AI era, the trade-off is between CVE and vulnerabilities that people haven't noticed yet. In the AI era, the forest is ALWAYS on fire. Even when you don't see the smoke.
- 1 computer and Google identity for administrative stuff: Primary accounts for AWS, Google. I also do accounting and have the list of customers. I still don't have any other solution than opening PDFs for the invoices, but it's better than nothing.
- 1 computer for programming, with a non-asministrator account for everything. Claude runs in Docker, with the dev directory mounted, .git:ro (read-only) and .env mounted to /dev/null.
But Claude tries to escape it A LOT. Claude is designed like a virus, we tell him not to do things, and it 1. tried to download the git binary from the internet, 2. read the remotes in the git directory and tried to access them, only blocked because it didn't have the credentials.
DEFINITELY forcefully block out Claude from seeing your .env files, it WILL try to brute-force an access to your intranet!!
Will this even matter though? As long as these computers are on the same network, AI tools will find a way to get from one to the other if any of them are compromised
Most people can't afford to have multiple really good computers. More likely you have a phone plus a single desktop or laptop. You would want to share your expensive GPU+memory on the same computer you use for gaming graphics and for AI-models for development.
To do.. what exactly? Pay for my bills? I don’t know how it’s setup in your country but in mine that’s not realistic. Everything is handled on the app.
Lol, you mean like here in the UK where all banks are closing down branches left and right? I think my nearest one is a 45 minute drive away, and last time I went in they just directed me to a terminal that was basically my exact online banking interface and I had to log in the same way I would have done at home. I don't know what kind of query would warrant speaking to an actual teller. If you're regularly sending money or even want to check your balance this cannot be sustainable, would be an enormous waste of time.
We might as well be watching the rolling back of the PC "revolution": From ENIAC to the Apple I, to the PC ...
... to us, wearng wearables of loving grace, that we do not own, but own us. All compute done beyond our reach, but ever-so-available, at the affordable price of a subscription and the prostitution of our soul.-
I’ve been thinking about how we’re still in the early days of agents, similar to the early days of the internet, happily opening up our machines and running new protocols.
Soon (now?) the threats will become ever more complex, and we’ll be back in the early-2000s-like virus storms.
My router got hacked the other month (left web interface open to internet) after being exposed to the internet and the persistence left behind was overly commented LLM artifacts. I caught it in like 5 minutes so they weren’t able to do much but it was eye opening how much more basic practices like firewalls are not something that can be ignored even if you update your firmware frequently.
kind of boggles me that the big ai labs arent running zero-shot classification on all their outputs with "does this look like any of these bad things we know people get up to"
How very considerate! Steal from the rich, not the poor.
Conversely, don't bother with poor people!
Now we just need proper Robin Hood malware that "levels out the playing field" as it were, by stealing from the rich and moving that money into the poor people's accounts (that have also been hacked).
Bolt this on to a self replicating worm that uses open ollama instances (~10k of them on the internet as of this comment) for distributed free inference, and you’re off to the races. Send discovered crypto to burn addresses where it’s unrecoverable, or skill the agent to know how to buy itself more compute with it.
You’ll have created an autonomous Robin Hood clone army. Fun times.
Malware authors are also software developers, product managers. It's adorable to see them do the things other software developers and product managers do.
Just like it's amusing that Osama bin Laden reviewed Al Qaeda expense report and these expense reports look just like ones anyone else would file.
Expense: two burgers and fries
Price: $15.50
Vendor: Five Guys
Cost center: SRE, department of evil
Business purpose: observability meetup, k8s brownbag
The lead generation pipeline the Talos people are using is actually decently competent. I wouldn't fail them in an interview.
You're technically right, which is why I say "good enough".
It's a good strat imo - start simple and get it working with LLMs asap, then you can add heuristics etc based on the data you have and improve with that benchmark
Makes me want to have like 5 separate computers:
- one for banking only
- one for development only
- one for web browsing only
- one for gaming only
- one for socials only
How else can you stay safe against a threat that exfiltrates your entire digital life in like 10 seconds?
Before, people were responsible for the dependencies they were importing.
With npm and all the javascripters flooding the market, it became normal to be irresponsible "because everybody is doing it".
Fast-forward a couple of years, now everything is web based, even the stand-alone app version. No one knows what is running on own computer, even the Devs themselves don't know what the are asking you to run...
With a pinch of vibe coding on top, and thats the end.
We took 40 years for our industry to be taken seriously, it's all gone now.
I’m personally glad I don’t have to download separate runtimes and apps and can just use the standard browser
This is an attitude not reflective of reality for those that wrote code in the pre npm days
The fact is we always were dependent on others.
I'd love to hear other takes on this though... Current thought is to just use something with a whole lot of exposure, in a whole lot of environments, pretty far downstream so the package hits all of that exposure, and hope someone spots it, kinda like xz. I tend to keep my base OS extremely minimal and do all my real work in VMs.
The other SSD is a normal setup where you do normal computer stuff. I don't access the dev SSD through it, I don't even mount it, although the danger seems minimal if you do. Banking, socials, gaming, whatever. No AI of any kind on it, aside from web portal for claude/chatgpt/whatever and possibly local AI model that gets no internet access.
No idea why I'd want to have its own setup split between gaming and socials, as an example. Better yet just toss all your twitter and bluesky accounts into a volcano, and consider whether to do it with HN as well.
I mean if you have only one there’s https://www.qubes-os.org/ but not sure about the gaming part, nor is it necessarily as good as separate isolated hardware.
I feel like your phone is good enough for all of those, and generally secure enough unless you're rooting/jailbreak it
Popular harnesses have mature enough remote access features that I can use the same interfaces and even mobile apps.
If you’re not familiar with Docker, you can even have an LLM set up the environment for you and help you start and stop the container. If you do this, be very sure that you switch to having a harness run inside the VM or container, not having the harness on the host machine run commands inside the container. If you do the latter style, it will be running commands on the host which is what you want to avoid.
No need for something new here, Claude Code (Auto + default settings) has a tendency of running "find"/"ls" on your home folder when it struggles with solving a problem. I wish I was joking.
Now of course people trust these AI companies not to do bad things(tm), but all it takes is replacing the model endpoint URL -or any kind of harness compromise- and nothing will seem amiss
I've seen this first-hand. I was doing some work in a repository, Claude concluded it needs to investigate, how some AWS resources are created, and it literally started scanning ~/* for Terraform code. Which it found and started browsing through.
Or, when I asked Claude to debug some Spring issue, corporate firewall blocked Maven Central, so it literally started scavenging ~/.m2 caches to find the jar it desperately needed. Then it could not find it, so it started to clone Spring sources from GitHub to build the missing jar. I stopped the clone mid-way. I wish I was joking, I only stepped away from the keyboard for one minute.
- 1 computer and Google identity for administrative stuff: Primary accounts for AWS, Google. I also do accounting and have the list of customers. I still don't have any other solution than opening PDFs for the invoices, but it's better than nothing.
- 1 computer for programming, with a non-asministrator account for everything. Claude runs in Docker, with the dev directory mounted, .git:ro (read-only) and .env mounted to /dev/null.
But Claude tries to escape it A LOT. Claude is designed like a virus, we tell him not to do things, and it 1. tried to download the git binary from the internet, 2. read the remotes in the git directory and tried to access them, only blocked because it didn't have the credentials.
DEFINITELY forcefully block out Claude from seeing your .env files, it WILL try to brute-force an access to your intranet!!
You forgot one.
Ideal setup:
- Dev machine
- Gaming machine (isolate downloading mods)
- Browsing machine
For banking I use macOS/linux, sometimes iOS unfortunately but we all do sins
... to us, wearng wearables of loving grace, that we do not own, but own us. All compute done beyond our reach, but ever-so-available, at the affordable price of a subscription and the prostitution of our soul.-
Soon (now?) the threats will become ever more complex, and we’ll be back in the early-2000s-like virus storms.
Conversely, don't bother with poor people!
Now we just need proper Robin Hood malware that "levels out the playing field" as it were, by stealing from the rich and moving that money into the poor people's accounts (that have also been hacked).
You’ll have created an autonomous Robin Hood clone army. Fun times.
https://en.wikipedia.org/wiki/Morris_worm
Just like it's amusing that Osama bin Laden reviewed Al Qaeda expense report and these expense reports look just like ones anyone else would file.
The lead generation pipeline the Talos people are using is actually decently competent. I wouldn't fail them in an interview.It's a good strat imo - start simple and get it working with LLMs asap, then you can add heuristics etc based on the data you have and improve with that benchmark