r/devops 2h ago

Ops / Incidents I built a safety tool for the AWS Console that switched itself off for 6 seconds every time you used it

0 Upvotes

I wrote a browser extension that adds a "you're in PROD, are you sure?" confirm to destructive AWS Console clicks. Capture-phase listener cancels the click, shows a dialog, replays the click if you confirm.

Trouble: some actions re-render their menu between confirm and replay, so the replayed click hit a detached node and did nothing. My fix was a short bypass window after confirming — for a few seconds, clicks pass straight through so the replay works.

The window wasn't scoped to the action you confirmed. It was global. So for ~6 seconds after confirming any destructive action, every other one was unguarded. Confirm a Lambda delete, click Terminate on an EC2 instance three seconds later, and it just goes.

A tester reported it as "the popup stops appearing sometimes." Not a UI glitch — the guardrail was switching itself off, on a timer, every time it ran.

Fix is one line — scope the window to the confirmed action:

// before
if (Date.now() < bypassUntil) return;
// after
if (Date.now() < bypassUntil && rule.label === bypassLabel) return;

The lesson that stuck: a bypass is a security control too. I wrote mine as a UI workaround, so I reviewed it like a rendering bug, not a security decision. Anything that turns your protection off — even briefly — deserves the same scrutiny as the protection itself.

Anyone else hit this class of bug — a temporary exception that was broader than intended — in auth caches, feature flags, rate-limit bypasses?


r/devops 2h ago

Discussion Blacksmith.sh GitHub runners

2 Upvotes

I signed up for blacksmith about a week in I’ve got a $7 bill already. I’m also getting you’ve hit 80% of your free mins and you should add a payment method.

I’m seriously confused after being extremely impressed by how fast their runners are. Now I’m wondering what the heck is going on.

I tried to reach out to them but need a payment method to contact billing and no reply on the support@ email.

I’m happy to pay as the service is good but really surprised how they work/do business. I feel I could easily be in for future bull shocks.

Has this happened to anyone else.

Looks like I’m not the only one hit by this - https://forestwalk.ai/blog/surprise-blacksmith-costs/


r/devops 3h ago

Discussion Implementation is getting faster but the org around it moves at the same speed as always and DevOps is in-between.

3 Upvotes

We all know that large organizations tend to have a lot of meetings, bureaucratic processes and are in general slow moving. That was true for a long time and I experienced that myself, having worked in multiple different companies from startups, scaleups and large corperations.

However since everyone now adopts AI and implementation speed is undeniable multiples faster. I mean it's not just implementation, but also a good part of DevOps. But the rest of the organization moves in the same speed as before. Even if the company uses AI in the other departments (my company does basically throw AI at everything, I guess you know what I mean), they still can't competet with the productivity change in software development.

The blockers are now decisions, requirements and communication. That's not completly new, but now its >95% of the time. I implement features in hours that would have taken days before (I am not a pure DevOps Engineer). And it's not the review process itself that blocks us (We use AI in reviews as well, because otherwise it would be impossible), it's the normal company processes and pace.

Don't get me wrong here, I kind of enjoy it a bit, because I can use a lot of time for learning, building stuff that I think is worth improving, but I think we have a general problem here that is basically systematic to every large organization. And I see that so called "AI native organizations" (what ever that means), will probably overtake tech companies in the long term.

I have a really good comparsion, because I am working with a startup as a side gig and my main job is in a mid-sized tech company. In the Startup we (2 devs) built a product that would have taken several months by a 5-person team in the past and the reason is mostly: decisions and communication.

Do you see that in your company?

My conclusion is that we need to make engineers owners of the feature/product + infra, otherwise we won't see much productivity gains and this means also that management layers need to be cut, because they are the bottleneck currently. The idea of centralized DevOps will is going to die I guess.


r/devops 5h ago

Discussion Anyone else interested in connecting with DevOps folks from different parts of the world?

19 Upvotes

I recently came across a post on Reddit from someone looking to connect with other DevOps professionals. I reached out, and we ended up moving the conversation to Discord.

It's been pretty nice having a place where we can talk about things we're working on, exchange ideas, troubleshoot things, and share what we've learned.

But the conversations haven't really stayed limited to DevOps - which is actually what I've enjoyed most.

We sometimes end up talking about things like:

  • How DevOps/tech work differs between countries
  • Differences in workplace and engineering culture
  • How people got into the industry and how their careers have developed
  • Tools, practices, and technologies people are using
  • Interesting things we're currently learning or building
  • And occasionally just completely random stuff 😄

I'd like to meet more people from different parts of the world and have more of these conversations.

You don't need to be a DevOps expert or have 10+ years of experience. If you work somewhere around DevOps, SRE, cloud, infrastructure, platform engineering, sysadmin, or software engineering - or you're simply learning in that direction - feel free to reach out.

We're currently a small group, and we'd like to keep it that way rather than turn it into a huge server full of people who never talk.

If you'd be interested in chatting and getting to know some other people in the space, leave a comment or send me a DM. Happy to connect.


r/devops 5h ago

AI content How to create Planning, Design, Development, Testing, Deployment, Maintenance Pipeline? Please suggest sources

1 Upvotes

Hi everyone,

I recently watched a podcast featuring a senior developer from EPAM, where he explained the different phases of the SDLC. He mentioned that he built a pipeline covering the entire software development lifecycle—from planning and design to development, testing, deployment, and maintenance—with each stage flowing into the next.

I really liked that idea, and now I'm wondering how to build a pipeline like that myself. Does anyone have any good resources, guides, or examples to learn from?

Thanks!


r/devops 11h ago

Discussion CV Review — DevSecOps / Cloud Security internship

Post image
0 Upvotes

I'm a student targeting DevSecOps / Cloud Security internship opportunities.

I've tried to position the CV around DevSecOps rather than generic DevOps or cybersecurity.

I'd appreciate feedback on:

  • Is the DevSecOps positioning clear?
  • Are the projects strong/relevant enough?
  • Is the Skills section too crowded?
  • Does anything look unnecessary or exaggerated?
  • What would make you reject this CV at first glance?
  • If you were a recruiter, what would make you reject this CV?
  • What would you change first?
  • - Is the CV ATS-friendly and likely to be parsed correctly by applicant tracking systems?

r/devops 17h ago

Tools Jenkins Tutorial recommendations required

0 Upvotes

So mu company has asked me to learn Jenkians. Which tutorials or playlist would you suggest to learn jenkins? Please give your suggestions.

Also, how much time does it require to learn so that I can start writing basic CI/CD pipelines.


r/devops 18h ago

Tools Do you build your own pipelines?

23 Upvotes

I work in a small team that deploys some internal products. No big user pool or database for me! We build small web apps that automate inner processes, and some scripts and jobs. Most of it runs on k8s, the apps and the jobs. We mostly run everyting on Azure. And everyting is managed by code, obviously, so we've build GitHub actions pipeline that live with the code to deploy our infra via bicep and the code via a test-build-deploy-promote pipeline, started on push.

But I've seen some platforms that propose full CI/CD as a services, and it feels like managing the pipeline yourself these days is a bad thing. To me, deploying a solution, especially in the containerization era, is simple; Run the tests, build the image, deploy the workload. Add some customization related to the app if needed (specific parameters, logging or testing jobs, etc). But is managing this code yourself a bad practice? I have to say, it does make for a lot of duplicate code in a lot of repos. The pipelines are very similar.

Should we always aim to use a standardized CI/CD platform? What tools do you use for CI/CD? How different do you handle it from a small project to a more important service?


r/devops 20h ago

Discussion How to optimise this billing??

Post image
0 Upvotes

Hii Guys!,we built an MVP.

Offered the services to three specific clients and got a good response, we are planning to advertise and bring more signups in the next two months

Initially the server bill was roughly 5-8 dollars but now it's increasing so wanted to check

-how do you optimise your server billing?? -Should I move the infra to AWS,will that be cost effective??


r/devops 21h ago

Career / learning Do we need to sacrifice holidays?

0 Upvotes

If I move to devops should I accept that I won't get to celebrate weekends and new year's?

How has been your experience?


r/devops 23h ago

Career / learning Moving to DEVOPS

0 Upvotes

Hey guys,

I recently got the opportunity to move to a DevOps team, and I’m looking for some advice on how to prepare.

I have around 5 years of experience working in infrastructure and systems. Most of my experience is with:

  • VMware/vSphere and virtualization
  • Windows and Linux administration
  • Networking and troubleshooting
  • NetApp and storage
  • PowerShell scripting and automation
  • Monitoring, backups, and general infrastructure operations

I also have some beginner-level experience with Docker, Kubernetes, Git, GitLab, Argo CD, Ansible, and Terraform. I’ve built some labs and worked with them a little, but I definitely wouldn’t consider myself experienced with them yet.

I’m starting the new DevOps role next month, so I want to use this month to prepare as much as possible.

For people who moved from infrastructure/sysadmin into DevOps, what would you recommend focusing on first?

I’m mainly looking for a good Udemy course, YouTube course/playlist, or structured learning path that is practical and focused on skills I’ll actually use at work.


r/devops 1d ago

Career / learning Help on picking which one to choose between data engineering and devops to transition from QA

6 Upvotes

Hello! As the title says, I'm a QA engineer and I'm trying to transition to a new position. Right now, I don't know whether to pick Data Engineering or DevOps. Regarding my skills, in my free time I'm learning Linux, Python (along with some MySQL), CI/CD, Docker, Kubernetes, and currently playing around with Azure. I'm asking because both seem interesting, but I don't know which one to choose to learn further.

Edit: also learnt some openshift and helm chart, because it was requested at my job


r/devops 1d ago

Discussion Anyone else nervous about what coding agents can actually run?

0 Upvotes

Using Cursor a lot more with tools enabled. Love the speed. Don't love the part where the only thing between "delete this" and it happening is me watching the terminal.

Is anyone doing something more solid than prompts + hope, or do you just keep it away from prod/cloud entirely? Had any close calls?

Genuinely just curious how people are handling this.


r/devops 1d ago

Career / learning CKA additional resources

5 Upvotes

Was preparing for CKA and took Kodekloud CKA course. Currently doing mock exams then moving into the ultimate CKA mock exam series.

However I have been reading that since 2025 there is a lot more emphasis on new topics like:

Helm, Kustomize, CRDs, Calico/Flannel, Cluster Upgrades/Installation, CNIs, CRIs, that I havent seen much in the labs/exams there or are really basic. I think the one that is most frequent is Helm.

What can I do to improve in these areas? What other new areas I missed (I am a bit familiar with Gateway API but recommendations on what to expand on this is also welcomed!)

Thanks!


r/devops 1d ago

AI content How do you solve long-term memory in AI automation workflows?

0 Upvotes

I've been thinking about AI automation lately, and I'm starting to feel like long-term memory might be one of the biggest problems.It's not just about making AI capable of controlling a screen. The AI also needs to remember what it's supposed to do.

There are already quite a few ways for AI to control screens, like OpenAI Computer Use, Claude Computer Use, Gemini Computer Use, as well as various hybrid approaches.The way these systems maintain memory seems to rely more on things like structured actions exposed by apps and the keywords being used in the current interaction. Personally, I don't think this approach works that well.

I also don't find this kind of screen-control approach particularly convenient.If I could use a hardware board to control the entire screen instead, that would make much more sense to me.Basically, you plug a hardware board into the device's USB port, and let the hardware capture the phone's screen and then control the device through USB HID.

I think this approach is pretty interesting because the AI doesn't necessarily need to know what API each app has, and it doesn't need a separate integration for every app.

It just needs to be able to understand what's happening on the screen and remember what it's supposed to do.So I feel like memory is actually the key problem here.

Are there any existing solutions or approaches that I should look into?I'd really like to understand how people are solving this problem.


r/devops 1d ago

Ops / Incidents Brain fart moment this morning

32 Upvotes

If you have a small managed K8s cluster with a single node pool and you would like to change the SKU of the VMs in that node pool, make sure to double check how your Terraform provider handles this change

Maybe on AWS or Azure or whatever, this is a graceful operation where the new nodes are provisioned first and the old ones are drained before being deleted

On OVH however it will destroy the existing nodepool first then provision the new one, and during this process your cluster will have zero nodes and whatever was running on it will be down

Thankfully this wasn't a production cluster. At the end of the day I'm just a dev who takes care of devops because nobody else will. In hindsight I feel like I should have seen this coming, but I guess most things work automagically nowadays and it can be a surprise when something does not


r/devops 1d ago

Discussion Getting started with DevOps with no IT background?

0 Upvotes

Hey all, I’m looking to seriously change my career path but I am genuinely unsure where to start. I come from a Business Management background, and am looking to get into the IT field, specifically DevOps. I realize that will not be an easy transition by any stretch of the imagination, but life needs to change, and this is my change.

My question is, where and how do I even begin? I’ve heard conflicting answers from peers and family, some say that a degree is absolutely necessary, while others say that it’s not necessary, and that companies will be more interested in experience and/or knowledge of the topic. Is a degree truly needed in today’s world, or am I able to be self taught and still land a decently paying job? If so, what topics would you recommend I get a good understanding of?

Thank you in advance!


r/devops 1d ago

Discussion do I need a devops person or a sysadmin?

70 Upvotes

here is the context: Mid size (1000 users) company that has been trying to do in-house software dev (outsourced to south east Asia) with poor results.

Company IT side is fairly mature, heavily virtualized, hybrid on prem/cloud, but a bit weak on the Azure side. Existing sysadmins are pretty solid.

The dev is, well a shitshow. We are presented with projects with no infrastructure requirements, the devs dont really seem to know what they need. Currently they are doing everything in Blazor after we begged them to move to cloud. Last few attempts to develop mobile apps has been an abysmal failure. Im dumbfounded by what I see (and I've never coded). Coding is all shoved into a single .dll, no comments, devs dont seem to know that we need to use HTTPS, dont understand key vaults (we caught them passing passwords in clear text). we had to fight with them to implement change control, and they dont seem to understand our efforts to move ahead with pipelining.

My question for you is, do I need a devops admin? Can a more senior devops admin also be involved in reviewing code? infrastructure planning?

not sure how to move forward


r/devops 2d ago

Security How do I protect my IP for on prem/byoc deployments

0 Upvotes

I was giving a bit of thought, I can't just deploy my services directly to a customer's vpc as via root access they can just see my code. Be it docker running on an ec2 or eks. If I dont go ahead with the control/data plane split, what options do I have to protect my code? Will i have to go to the OS level and build my own AMI etc with attestations and key verifications and stuff at each stage, or use something like nitro conclaves or nvidia confidential compute or something along those lines? What is the way?


r/devops 2d ago

Career / learning Undergraduate Thesis Subject

6 Upvotes

Hello everyone,
I'm starting my 4th year in Computer Science and I'm looking for a DevOps/Cloud related thesis subject. I want to do this thesis in collaboration with the company I'm working at, but they told me to find some possible ideas before we decide to move forward with this.
I'm pretty new to DevOps and Cloud, I filled an intern position at the start of July, so I would like to avoid the very complex subjects.
Thank you.


r/devops 2d ago

Discussion github windows self hosted runners (on k8s). Is it supported?

1 Upvotes

trying to find a ref architecture that is widely recommended for deploying windows runners on k8s. From what I read ARC doesn't support it still? if not what are my options?


r/devops 2d ago

Tools Built this tool to simulate real world scenarios and incidents in local kubernetes

Thumbnail
github.com
0 Upvotes

Hey everyone,

When I learnt kubernetes and completed CKA, I didn't instantly get a chance to work on a real project. Later, i got a job and learnt a lot, but I felt that there was a need for a tool where we can simulate real world scenarios, we can learn what kind of scenarios and incidents come in real environments.

For this purpose, i built a tool which does exactly the same. You can simulate real world scenarios, use kubernetes tools like opencost, keda, grafana, argocd, traefik. Learn how you can use it and find out different commands to help you learn.

Do check this out if you are interested in this.


r/devops 2d ago

Career / learning Built a small 3-node Kubernetes homelab

Post image
477 Upvotes

I recently set up a small 3-node Kubernetes homelab using Dell OptiPlex Micro PCs running Talos Linux.

The main goal was to have a local cluster where I can learn, experiment, break things, and test tools without relying on cloud infrastructure every time. I’m planning to use it for Kubernetes networking, storage, observability, GitOps, security, upgrades, and general experimentation.

Small setup, but already a very useful playground.

Curious what others here are running for their Kubernetes homelabs.


r/devops 2d ago

Discussion How do you handle CI/CD credentials? Using GitHub Actions made me realize static encrypted secrets aren’t very safe.

74 Upvotes

After I first set up a deployment pipeline, I would simply drop DB passwords and API keys into GitHub Secrets and feel completely safe because they are encrypted. I recently went through a security breakdown on GitHub Actions that showed me that it could be a mistake to think that way.

The main issue is that an encrypted secret is still a static, long-lived target. Once the workflow finishes, that credential stays active indefinitely. In the breakdown, I saw a few default behaviors attackers look for, like teams forgetting to revoke access after a job runs or lacking the audit logs to even know when a key was used.

The proposed fix is shifting to dynamic orchestration where the pipeline generates a short-lived token at runtime and revokes it the second the deployment finishes.

If you're writing deployment workflows, how do you handle this, do you just use GitHub's default storage, or are you injecting temporarily credentials to avoid leaving static keys exposed?


r/devops 2d ago

Discussion i'm struggling with the evidence side of secret detection

0 Upvotes

i'm working on a decision-making problem where an agent gets a possible leaked api key and has to decide whether it's actually a prod credential and then either ignore, verify or remove/rotate it.

right now i'm using repo context (file path, variable name, surrounding code), key format, git history, whether the code is active/reachable, deployment context, and last_used_at from the provider's admin api.

the problem i'm running into is that most of these signals can tell me "this probably is a real secret", but don't tell me much about whether it's still live or already revoked.

for people who've dealt with secret leaks in actual devops workflows, what do you normally check before deciding what to do with a flagged key?

i'm especially interested in evidence you can get without actually authenticating with the discovered key.

disclosure: this is my own personal project, not a commercial/product promotion. just looking for technical feedback.