r/technology • u/MarvelsGrantMan136 • Jul 30 '26
Artificial Intelligence Amazon accidentally spent $1.8 million using Claude for menial coding task, went 860% over budget — 'catastrophically expensive' coding blunders discovered in internal Amazon AI usage metrics
https://www.tomshardware.com/tech-industry/artificial-intelligence/amazon-accidentally-spent-usd1-8-million-using-claude-for-menial-coding-task-went-860-percent-over-budget-catastrophically-expensive-coding-blunders-discovered-in-internal-amazon-ai-usage-metrics1.7k
u/flat5 Jul 30 '26
"only detected some five months after the issue started happening"
Good thing they have their extremely rigorous interview process, so no completely irresponsible, checked-out idiots are working there.
522
u/Prophet_Tehenhauin Jul 30 '26
Most competitive employees in the world take 5 months to notice 2mil vanish.
Them FAANG hires sure are something
263
u/RulesoftheDada Jul 30 '26
When I was at Amazon there was engineer who was fired for spending nearly a million in AWS to mine like 10k in ethereun.
215
u/CompetitiveSport1 Jul 30 '26
To be fair, that's using company resources for personal profit, which is basically theft. Wasting tokens through a mix of shitty processes and intentional tokenmaxing aren't as easily claimed as fireable offenses
→ More replies (3)41
u/Arrow156 Jul 30 '26
Considering wage theft is by far the most common type of theft in this country, I'm certainly not gonna shed a tear.
→ More replies (1)23
u/Glittering-Quote-635 Jul 30 '26
I'm actually surprised he got caught.
30
u/RulesoftheDada Jul 30 '26
It was over several months. Then it led to org wide review of ec2 instances. Any new allocations had to go via director.
→ More replies (2)10
u/Glittering-Quote-635 Jul 30 '26
Yeh, I’ve pretty much worked at only large tech corporations all my life, and I just know how easy it is sometimes to hide things. Guess he wasnt good at it… :)
→ More replies (12)8
u/pr0crast1nater Jul 30 '26
I mean he used a fuck ton of ec2 lol. I understand having a few nodes here and there. But dude must have provisioned 100s of costly gpu ec2 instances.
→ More replies (1)5
u/greg19735 Jul 30 '26
yeah it's one of those things where it's just simply impossible to hide and be worthwhile.
There's no reason to mine $3 worth of crypto and lose your job for it. Might as well see what you can get away with. which of course makes it even more likely to get caught.
Really, it's just a terrible plan
5
u/pr0crast1nater Jul 30 '26
I will admit that the closest to this I really did, was use my company servers to quickly do chia mining, which is a coin mined via storage. But the way it works is unique in the sense there are two steps, plot creation using nvme drives and then storing them in a hdd and mine.
The plot creation was something which could be done offline and so I was confident nobody will detect any anomaly if I used 5 servers to do it, although it was heavy on CPU. And the mining part after that was super lightweight in cpu usage anyway. I got lucky and mined a few coins in one week, then sold them for 2000$. The price of that coin then crashed and I deleted all of that lol.
In hindsight though doing all this for 2000$ was incredibly stupid. But I did it more for the thrill I guess and I was genuinely fascinated by the weird way in which that coin was mined.
→ More replies (2)69
u/IPlayChessBTW Jul 30 '26
Fired? Bro should have been arrested lmao. He stole one million dollars
→ More replies (1)22
u/Bayo77 Jul 30 '26
Most programmers probably knew. Its just upper management doesnt want to hear it.
130
u/segfaul_t Jul 30 '26 edited Jul 30 '26
I work here and I can affirm that most of the “brilliant engineers” are in fact mediocre H1B applicants who got lucky.
EDIT: The "luck" refers to the H1B lottery system, and getting through the resume screen where both the recruiter and hiring manager are, oops, also on H1Bs from the same country as you. Amazon's technical interviews aren't difficult (at leas anymore), and there is rampant cheating with AI anyway.
→ More replies (23)86
u/MightyKrakyn Jul 30 '26
That’s because of the churn. When you’re such a large company and have to according to policy turn over your employees every 2 years, you run out of quality talent pretty quickly
→ More replies (1)67
u/segfaul_t Jul 30 '26
Lot of good ones left when they announced 5 day RTO
50
u/PurelyLurking20 Jul 30 '26
Nearly all* they got offers immediately for remote positions at smaller companies and I imagine most of them are still chilling in those positions.
Amazon churn now affects all positions with Amazon not just warehouse, it shouldn't be considered on par with Google and Apple, not even close
→ More replies (6)9
u/tooclosetocall82 Jul 30 '26
We tried to hire one but then Amazon said he could stay remote I guess so he declined our offer. I wonder how that worked out for him.
→ More replies (2)10
u/LunaticSongXIV Jul 30 '26
I was a subcontractor and got canned immediately when the rto announcement came through, and I told them that I wasn't driving three hours to the office. The mandate hadn't even come into effect yet. I wasn't being a team player, i guess.
→ More replies (1)14
u/missmeowwww Jul 30 '26
This is wild to me! I work in the public sector and have to review contracts frequently to ensure that deliverables are being met, billing and purchase requests are submitted timely, and that each service provider isn’t going over their allocated budget. We review the numbers and data on a damn near weekly basis. I can’t fathom something not being caught for months!
11
→ More replies (7)4
46
u/MultiGeometry Jul 30 '26
Anyone who says private industry is more efficient than government has never worked in private industry. There’s so much waste and the safeguards aren’t as strong because these companies make profits and get lazy.
→ More replies (2)23
u/revolvingpresoak9640 Jul 30 '26
Amazon is so inefficient. They will spend six months writing and reviewing a doc about what they will do, then give engineering some duct tape and popsicle sticks and call it a launch.
61
u/Minialpacadoodle Jul 30 '26
$1.8m is a rounding error for Amazon.
25
u/FrenchFryCattaneo Jul 30 '26
Just because the company has a lot of revenue doesn't mean you can waste $1.8 million on a single project.
→ More replies (3)→ More replies (8)6
→ More replies (14)33
u/Procrasturbating Jul 30 '26
Oh I promise this was malicious compliance tokenmaxxing.
→ More replies (1)
1.9k
u/ElGuano Jul 30 '26
Accidentally, or did they do what my company did, which is force everyone to use AI for everything, put up a leaderboard showing where you were in the rankings, post up how many submissions and commits were AI-authored/assisted, and say that the people who didn't adopt AI and had the least usage would be let go?
And then made a Home Alone face, gasping, "why is everyone using AI for everything?"
695
u/Brave-Perspective714 Jul 30 '26
Same thing happened when "lines of code written" became the metric output was measured by.
My husband had one he was asked to modify, where he removed over 19,000 lines of code that didn't actually do anything.
Dude who had written it had been a "high-performer" and "distinguished-engineer" who had left years before, but when they started digging into more of his work they found the same thing over and over again. He had basically written one large chunk that did things and then undid them so they didn't affect the output, and copied and pasted it into everything he developed to look like he was a rock-star according to their metrics.
The bulk of my husband's mid-level DevOps career was deleting lines of code and reducing api calls.
583
u/MeatEaterDruid Jul 30 '26
Honestly can't be mad at the original dude. That's some 'change the margins so the paper looks longer'-type shit.
350
u/Brave-Perspective714 Jul 30 '26
Yep, and it worked.
That's what they said they wanted, so that's what he gave them. Just a great example of someone creating the metrics not understanding what was actually meaningful about the work.
173
u/SordidDreams Jul 30 '26 edited Jul 30 '26
Ah, good old Goodhart's law. Any metric that becomes a target ceases to be a useful metric. Half a century old and still a complete enigma to management types everywhere.
56
u/Quaiker Jul 30 '26
Goodhart's Law and Chesterton's Fence should be posted up in every workplace.
12
11
45
u/NRMusicProject Jul 30 '26
When your boss uses ridiculous metrics to judge your work, you minmax the shit out of it. Same thing as purposely doing an hourly contracted job slower so you can make more money.
Nowhere near as witty, but when I was a minimum wage worker at Hungry Howie's as a kid, they were short on boxes and sent me across town to the next nearest one (30 minutes each way) to pick up some more. When I asked how much I'll be paid, they said "you're already being paid to work! Just do it!"
I spent an hour hanging out at the beach with some of my surfer friends on the way back.
27
u/toolatealreadyfapped Jul 31 '26
I was a lifeguard when I was a teenager, and my boss hated to see us sitting there "doing nothing." So she kept adding dumb tasks, until I realized it's not the work she wants done as much as she just wanted to see us be busy.
I kept a bucket of leaves by the lifeguard stand. When I'd see her suburban pull up, I'd jump down, dump the leaves into the pool, and she'd walk in and see me dutifully scooping leaves out of the pool. She'd nod approval, go about her business, and leave. I'd gather all the leaves into the bucket, then climb back onto my chair.
5
u/hippocratical Jul 31 '26
Mine was you could walk to a get a 2 hour coffee break if you carried some thick files and binders under your arm as you walked through a few departments and out a back door. People saw you walking with purpose but never saw you leave the building. I got praise from my boss for being such a hard worker.
11
u/PorkVacuums Jul 31 '26
The key is looking annoyed. No one in corporate wants to get roped into some bullshit that's someone else's problem. So if you walk around with a file or a binder and just generally look annoyed, people just keep out of your way.
→ More replies (1)13
u/rkoy1234 Jul 30 '26
idk seems to me like he understood but just was maliciously complying.
getting an llm to intentionally write meaningless code that looks plausible but doesn't break anything while encapsulating the original goal seems tad more complex than just getting it to do the task.
8
u/Brave-Perspective714 Jul 30 '26
Lol... LLM's were still science fiction in the days I'm talking about.
18
u/cfb-food-beer-hike Jul 30 '26
A little off-topic, but this reminds me of a study done on SAT essays. They found they could hold an essay up from across the room and accurately guess its score based on the length. As a species, we've never been good at evaluating anything lol
→ More replies (1)33
u/systemic_booty Jul 30 '26
Yup, and the problem isn't the student changing the margins or making the periods size 14 font, the problem is the essay requirements being by page length rather than word count OR better yet, the substantive content of the essay itself.
→ More replies (3)→ More replies (7)16
u/RVelts Jul 30 '26
In high school we changed the margins and increased line spacing and the font size of punctuation to make papers longer.
In college we changed the margins and decreased line spacing to fit as much as possible in the maximum 2 page limit.
It seems like high school promoted being verbose, and college wanted you to intelligently summarize in as short of a paper as possible.
→ More replies (1)63
u/CatharsisVoid Jul 30 '26
So glad to be on a team with a director who praises code deletions (cause it's usually tech debt related).
However, we all know that the biggest topic of discussion in manager and senior leadership meetings during performance review season revolve almost entirely around commit count.
You know what started happening? People stopped squashing and cleaning up commits before merging...
36
u/Brave-Perspective714 Jul 30 '26
Yep. My husband is running into it again now with PR reviews. As soon as management gave an indication that they were evaluating some aspect of performance by how many PR reviews someone was completing, they started having more and bigger problems.
Guys were just going in, saying, "yep looks like code" and approving it so they could get on to the next one and pump that metric up. They had to make it so now only *trusted* seniors can approve PRs, but they aren't shifting workload around to free up the time for those guys to be able to do all of the approvals.
Husband is looking for a new job, but everybody knows what the job market for IT is like right now.
→ More replies (1)12
u/snakerjake Jul 30 '26
So glad to be on a team with a director who praises code deletions (cause it's usually tech debt related).
We expect our juniors to write more code and our seniors to delete more company wide
→ More replies (1)13
u/kos-or-kosm Jul 30 '26
Which, at least from my perspective, makes a lot of sense. Get the juniors a lot of experience while they get the functionality working, and then have the experienced seniors refactor it afterwards, making it more efficient.
17
u/zaphodbeeblemox Jul 30 '26
If the quality of my work, nay my character is to be judged by how many words that I pen to paper, then by the divines and the beast himself you will be forced to reckon with the fact that I shall not only superfluously adorn my vocabulary with naught but the most extravagant of phrases but I shall ramble on until all reason has left my prose.
→ More replies (17)12
u/Independent_Bear989 Jul 30 '26
Any company that has “lines of code written” as a metric deserves to go bankrupt, or the people who make that metric should be fired and blacklisted from any job where they work with designing software.
→ More replies (2)143
u/TheGreatKonaKing Jul 30 '26
Yah was gonna say, maybe some clever employee’s leaderboard-gaming script just worked a little too well!
→ More replies (6)111
u/cornmonger_ Jul 30 '26
what an idiotic policy
→ More replies (1)151
u/nox66 Jul 30 '26
The AI craze has shown me that many executives will gladly run their business into the ground for no financial benefit at all as long as they don't have to admit they were wrong and responsible for it.
42
u/b0w3n Jul 30 '26
They couldn't conceive of a situation that they aren't the smartest people in the room, and LLMs are definitely able to outsmart many of them... so they figured that LLMs could universally replace workers.
→ More replies (1)48
u/looeeyeah Jul 30 '26
"I asked Claude if it was a good idea, Claude said I was very smart for asking such clever questions, and once again I am correct!"
→ More replies (4)15
u/CrunchyKorm Jul 30 '26
I'm not going to act like people can't be blinded by financial incentives, but it's funny that c-suite members, of all people, are the primary group that cannot seem to understand the difference between AI being valuable because how how much funding it seems to be getting, and how much money it actually ... makes as a product.
A lot will never admit they were wrong, but a lot more can't understand that AI's main value now is just investor signaling.
→ More replies (17)37
u/PaulClarkLoadletter Jul 30 '26
AI will fire just a shit load of code out of the slop cannon. You want a module that will call an cron job? Here’s 60,000 lines of code.
10
u/Turlututu1 Jul 30 '26
I've seen people at work use AI to vibe code intranet pages or similar and I'm wondering how much bloat there is in the html....
10
u/I_Am_Jacks_Karma Jul 30 '26
just run the command to tell it to reduce bloat duh
(jokes aside this does sometimes work)
→ More replies (2)→ More replies (3)22
u/PaulblankPF Jul 30 '26
Last year I was trying to learn to code for making a video game as a solo developer. I was making simple stuff still like platforming to learn how things work. I asked chat gpt at the time to try to help me write part of it. It was all kinds of fucked up. Every time I pointed out a mistake it would just go “Oh, you’re right! Let’s fix that by changing “__” to “__” and that should fix everything.” Except it had like fucking 10 errors in only a few lines of code. It was worse than me trying to figure it out myself with little experience (I took graphic design and a coding class in high school in 2004).
→ More replies (7)19
u/AnAncientBog Jul 30 '26
It can be a useful tool if you already know a ton about coding and can get very specific about what you ask it to do and bound it with appropriate parameters and are able to check it quickly.
Which is not at all how its being deployed.
→ More replies (3)
376
u/Chrono_Convoy Jul 30 '26
Give me $1.8 million and I won’t code a thing!
104
u/Awesomebox5000 Jul 30 '26
Shoot, I'll code nothing for a cool $900k.
48
u/IvarTheBoned Jul 30 '26
I'll split the difference and do it for a cool $1.35 million, because you don't want to go woth the most expensive or the cheapest options.
→ More replies (5)11
u/Manatee-97 Jul 30 '26
Stop stop. If one person offers to do it for 900k it will create a minefield of low salaries
→ More replies (1)
299
u/gaspara112 Jul 30 '26
In engineering we say “good, cheap, fast. Pick 2”.
AI is when you pick fast twice.
33
→ More replies (10)20
u/Afferbeck_ Jul 30 '26
Employers hate hearing that phrase! Somehow they always think they can pick all three and get mad when they don't pan out.
13
u/SparklingLimeade Jul 31 '26
That just makes me think about how many people genuinely don't understand the concept of a priority system. If you mark everything maximum priority then you're just making things worse for everyone working on those items and especially for yourself.
But they'll never have the opportunity to understand because they are the source of the stink and so they never experience life without it.
79
u/Charlie2and4 Jul 30 '26
Back in my day if you just walked around screaming the words, "Agile" "Reduce churn", and "EBITA" You got a 3 hour lunch break every day.
29
→ More replies (3)21
u/Emmoeeee Jul 30 '26
And just perpetually say you’ll circle back
13
u/Charlie2and4 Jul 30 '26
That's why I like you. You and me are on the same piece of paper, we think outside the box.
Let's you take lunch, then I will take lunch, then we will take lunch.
441
u/Henrarzz Jul 30 '26
Leave it to tech bros to replace perfectly working solution with shiny new but less efficient one
133
u/unity-thru-absurdity Jul 30 '26 edited Aug 01 '26
"...the golden gleam of the gilded surface hides [the] cheapness of the metal underneath..." - a quote often misattributed to Mark Twain describing his 1873 novel, The Gilded Age: A Tale of Today, after which the historical era was named.
→ More replies (2)20
u/The_Rowan Jul 30 '26
That is an amazing quote for today’s America. Thank you for sharing that. Mark Twain is an American treasure.
→ More replies (27)32
Jul 30 '26
[removed] — view removed comment
→ More replies (2)17
u/farhil Jul 30 '26
also sometimes the principal engineers piss me off because they get handed a greenfield project done with like a team of other principals + senior engineers then create a fcking white paper at work thats like “omg we went so fast with only a FEW people (ie principal engineers making north of 500k a year) for this (brand fucking new) project with AI!”
Struggling with this at work too. Like yeah, you put your best guys on a team to work on a project with no technical debt, no juniors to handhold, and no production environments to support... Of course they got shit done.
6
u/StrongExternal8955 Jul 31 '26
Oh don't worry, they'll hand that project to you to maintain further.
47
u/Snorblatz Jul 30 '26
Almost two million wasted but still tracking how long it takes warehouse staff to poop 💩 ok Amazon.
→ More replies (9)14
u/Dajoechi Jul 30 '26
My friend she was a consultant and tracked this for another company right before covid. She just got her masters and her first job was to sit by her desk tracking bathroom breaks. She had her desk stationed right outside the mens bathroom. Got paid 120k starting for that. Living the dream i guess..
125
u/SeeingEyeDug Jul 30 '26
So much compute is wasted on wrong answers. The amount of times I have to reprompt AI to correct errors and the amount of wasted compute cycles for all that fluff on even the simplest shit I’m asking it. It’s insanely inefficient.
→ More replies (29)
216
Jul 30 '26
[deleted]
93
u/InteractionSoft14 Jul 30 '26
Yeah but that was only for 1 project. If issued of that scale pop up all over the place it can get out of hand pretty quickly. Amazon is a big company but it still doesn't make sense to piss away dozens of millions
→ More replies (22)26
u/smartello Jul 30 '26 edited Jul 30 '26
That’s annualized savings I made with twenty five lines Java code change to fix concurrency issues. Was not considered being significant enough to be used in an L5 to L6 promo doc. It just seats in footnotes.
Also, Amazon self hosts models via bedrock so the number in the article comes straight from the ass
9
u/bucolucas Jul 30 '26
Oh yeah, I remember getting laid off right after a big project got turned in. Your accomplishments mean nothing to these people.
→ More replies (8)10
u/CompetitiveSport1 Jul 30 '26
I mean, the article we're commenting on explicitly says that?
the tech giant’s latest quarterly revenue sits at more than $181 billion, meaning these excess AI expenses don’t even account for 0.1% of what it makes in a month.
→ More replies (1)
33
u/elmatador12 Jul 30 '26
“Wait AI costs money??” - These companies it sounds like
→ More replies (3)6
47
u/geldonyetich Jul 30 '26
Ironic that this is happening to the patent holder of the one click buy. Or perhaps karmic?
21
u/Knyfe-Wrench Jul 30 '26
It's even more ironic that it's the company behind AWS, famous for massive charges over small errors.
→ More replies (3)
12
u/Epyr Jul 30 '26
That actually seems like a extremely small AI budget for a company the size of Amazon.
4
u/Joe091 Jul 30 '26
…that’s not the entire AI budget for all of Amazon. It’s for some project, team, whatever.
23
u/HuckledYourBerries Jul 30 '26
If only someone could have seen this coming 😐
4
u/therealchadius Jul 30 '26
It's not like writers switched to writing bloated articles over 100 years ago when they were paid by the words or anything!
→ More replies (1)
10
u/Soulfly37 Jul 30 '26
A buddy of mine works for aws and he is on AI at least 6 hours a day. Most of the time he says it doesn't really help... and might even make the work take longer. But, he is forced to use it.
19
u/SoCal_GlacierR1T Jul 30 '26 edited Jul 31 '26
And yet companies continue to lay off humans in order to justify more AI spending.
26
u/brainrotbro Jul 30 '26
I routinely use AI for menial coding tasks. Why would I offload the interesting stuff to AI?
→ More replies (10)
6
5
u/Kastar_Troy Jul 30 '26
So they were willing to spend $200k on minor code generation which may not work in the end?
How is this all cheaper than just using thinking humans?
30
u/LieNaive4921 Jul 30 '26
Why is this even possible? Can't you cap Claude spending per... whatever you want?
Junk tech tbh
27
u/PuzzleMeDo Jul 30 '26
They probably didn't think they needed to, and they didn't want their developers to get stopped in their tracks in the middle of some urgent task.
When you're a two trillion dollar company, a couple of million dollars isn't that much to waste.
→ More replies (3)4
u/Pr0ducer Jul 30 '26
when you cap spending, and user burns entire month supply in first week, do they sit and do nothing for 3 weeks?
→ More replies (5)
5
5
u/Slackluster Jul 30 '26
How much would they have spent paying human workers to do those menial coding task?
→ More replies (6)
5
u/Numerous-Process2981 Jul 30 '26
I think it’s been very revealing just how unbelievably stupid all these companies and everyone in charge of them is
4
u/Logintheroad Jul 30 '26
Then fired 2k employees blaming it on AI. /s.
But also not really, I like in a tech town and people are losing their jobs to "ai" every day. Except the CTO and CEO, they are still getting theirs.
5
u/karma3000 Jul 30 '26
It's going to be a bloodbath when tokens are finally priced in order for chatGPT / claude to make a profit. Ie they will cost 5x to 10x what they cost now.
→ More replies (1)
11
u/laughy Jul 30 '26 edited Jul 30 '26
This is clickbait garbage. They cherry picked a few events that went over budget to make it seem like companies are so, so regretting the use of AI (hint: they are NOT). The article plays the normal tactics of linking to their own articles to argue a point the linked article isn’t making. Example: “as spiraling AI costs made them think twice…” - no where in that linked article does Amazon say or imply they are “thinking twice” about AI.
Come on folks, we’re better than this. It’s ok to be against AI, but stop trying to kid yourselves that there’s some sort of reckoning for tech companies coming. There isn’t.
→ More replies (6)
4
u/TheComplimentarian Jul 30 '26
Ironic, because I've seen people do this over and over with AWS. I worked with a company that had a really slick automated build process for cloud infrastructure, but they were running over budget, so I got a gig to come in and figure out what was going wrong.
They had a really slick build process, but they didn't delete shit, so every time they'd done a new deployment and pushed out a new application stack, they'd left an orphaned one just sitting around burning cash while doing nothing.
I was pretty well-paid for fixing it, but the amount they paid me was a tiny fraction of what they'd wasted by not vetting their own workflows, and this is pre-AI. The did this to themselves by not asking the most basic questions.
4
u/Snorblatz Jul 30 '26
This is why everything sucks now. Apps, websites, whatever people used to code by hand.
→ More replies (1)
4
u/Wander715 Jul 30 '26
I'm convinced that the future of software development is not pure vibe coding, it's too expensive and falls apart at the seems for large complex systems. I think a healthy mix of manual human produced code and AI generated code is probably the middle ground that most people settle on eventually as a good tradeoff in terms of speed and quality.
4
4
u/jakgal04 Jul 30 '26
Amazon makes over $1.67 billion in revenue per day, I think they'll survive.
This would be like us common sheep paying $5.00 for a soda on tap instead of buying a bottle from the fridge for $1.50.
→ More replies (1)
4
u/RedlandRenegade Jul 30 '26
I work in finance and right now, it’s all going to shit.
AI is an empty ballsack of lies but they’ve invested so much in it. They don’t want it to fail…yet it will.
→ More replies (1)
4
4
4
u/Radical_Coyote Jul 30 '26
Is that number correct? $1.8 million doesn’t sound like very much for a company the size of Amazon
4
u/fourtyTHEdeuce Jul 30 '26
Oh look the really expensive tool no one was really trained on but we've all been told to utilize to the max capacity so we can cut personnel is blowing up in our face. Neat
6.6k
u/lazyhustlermusic Jul 30 '26
They gamified it with a leaderboard and incentivized waste.
Now they're surprised pikachu when unrestrained usage generated an unrestrained price tag.