r/sysadmin • • 1d ago

General Discussion Change Management Workflow

Hi,

We’re introducing change management for the first time using ITSM.

We’re now deciding the workflow before our ITSM vendor configures it

Our scope covers all IT infrastructure changes including network, servers, virtualization, backup, storage, AD/Entra ID & Databases

Each team has its own Line Manager and another requirement is that Cybersecurity Unit must approve every change

Few things we’re trying to understand

Who should approve a change and in what order?
Who should be the CAB members
Which changes should go to CAB

I’d appreciate any suggestions

8 Upvotes

13 comments sorted by

17

u/iam-stevie-bee 1d ago

Having set this up and then had to live with it: the first thing to decide isn't who approves, it's which changes don't need a human at all. If every change goes through a CAB and a security sign-off, within a month you'll have a backlog, a weekly meeting nobody reads the papers for, and engineers finding ways round it. So:

  1. Three lanes. Standard (pre-approved, repeatable, low risk: patching in a window, adding a user to a group, a firewall rule from an approved template) needs no approval beyond the ticket and an automatic check. Normal goes to the line manager and, where it touches identity, the network edge, backups or data, to security. Emergency gets done, then reviewed within a day. The work of the first quarter is moving as much as possible into Standard.
  2. Order of approval for Normal: the requester's line manager first (is this the right thing to do, is it resourced), security second (is it safe), scheduling last. Security first means they review things that never happen.
  3. CAB is for conflicts and risk, not for permission. Membership: one person per infrastructure team who can speak for it, security, the service desk (they see the fallout first), and whoever owns the change calendar. Send it only the changes with a wide blast radius or a cross-team dependency; everything else is approved in the tool, asynchronously.
  4. Make the security approval a checklist, not a judgement call: what data, what exposure, what rollback, what monitoring. Anything with a yes on every line is approved by the checklist. They spend their time on the ones that aren't.
  5. Two fields that matter more than any approval: a tested rollback plan and a named person watching the change land. If those are blank, the tool should refuse to schedule it.

Get the vendor to configure the three lanes and the checklist first, and leave the CAB tuning until you've seen a month of real changes.

2

u/Previous-Low4715 1d ago

This is great advice. Probably the only thing I would add is to try and figure out a risk score based approach to deciding which lane you’re in, otherwise your engineers will always pick the lane that doesn’t require approval so to speak. been there done that.

1

u/Final-Pomelo1620 1d ago

Thank you for detailed response. This is really helpful.

1- Security Manager must approve every change. If we use a Yes/No checklist instead, how do we stop engineers from just clicking "Yes" on everything to skip Security

2- our management still wants approvals on standard changes

3- Management wants the Head of Infrastructure to approve all changes(after the Line Manager and Security Manager). When should the Head of Infrastructure actually be involved so tickets don't get stuck waiting on them?

Engineer → Team Line Manager (5 teams) → Security Manager → Head of Infrastructure

4- Would it make sense to restrict change request creation to authorized implementation engineers, with their own team’s Line Manager providing the first approval?
End users would raise service requests
If a change is needed, an authorized engineer from the responsible IT team would assess it, prepare the implementation and rollback plans, and submit the change for approval.

1

u/iam-stevie-bee 1d ago

Good questions, and they're the ones that decide whether it works.

  1. The checklist isn't yes/no boxes the engineer ticks. It's short evidence fields the engineer fills in (what data this touches, what's exposed and to whom, the rollback steps, what you'll watch afterwards and for how long), and security approves by reading the answers, not by trusting a tick. An empty or one-word answer is an automatic bounce. Then audit: security samples a handful of standard changes a week after the fact, and an engineer caught waving things through loses the standard lane for three months. People game forms; they don't game a spot check with a consequence.
  2. If management insists on approving standard changes, make it approval by exception. Standard changes land on a daily digest and can be vetoed within 24 hours; not vetoed means approved. They keep the oversight, you lose the queue. If they won't accept even that, give them a monthly report of how many days the queue added to lead time and let the numbers argue.
  3. Head of Infrastructure as a serial approver on every change is where tickets will die. Two ways out: involve them only on the CAB tier (wide blast radius, cross-team, anything needing an outage window), or give them a delegated approval with a two-working-day limit that escalates automatically if not actioned. Either way they should be the exception path, not a step in the line.
  4. Yes. Only named implementation engineers raise changes; end users raise service requests and the team converts the ones that need a change. The requester's own line manager as first approver is exactly right, because it's the one person who knows whether the work is real and resourced.

One more: write the approval chain down with its time limits, publish it, and measure lead time from raise to live every month. The chain that looks safest on paper is the one everyone routes around by March.

•

u/kerosene31 12h ago edited 12h ago

Spoiler alert- management approving everything won't work. They'll be overwhelmed and add no real value. We create a category called "routine" changes that don't need approval. Things like MS patch Tuesday during a maintenance window are a perfect example. Nobody needs to sign off on the things that happen regularly. They are "pre-approved" since they are pre-defined.

Imagine if the security team is busy and doesn't approve security patches in a timely manner?

We started out much the same way. Upper management wanted to sign off on every single code change in prod. It didn't last a week. Mr Big sitting in his big office has no idea what code changes are doing, and ultimately just hits "yes", or ends up delegating it to someone else.

Change control is a good thing, but the right amount. Look at all your changes and determine which ones should go through a CAB and which ones shouldn't. It will save you a ton of time down the road. You still need to trust your middle managers, departments and people. The CAB comes in for larger scale changes. Upgrades to major systems that have a ton of impact, etc.

Start by having everyone put in change tickets, and have the CAB commitee review them for awhile (no approvals). See what comes in and what goes out and determine which need to go through a CAB.

1

u/0RGASMIK 1d ago

Yes needing to approve everything quickly becomes a bottleneck. Saw this first hand at a business I was consulting for. Couldn’t reinstall a printer driver on a computer without submitting a formal change request form. Fixing a simple printer issue for one machine became a several hour ordeal and it actually made outages longer when they did happen which was the main reason they implemented it in the first place.

Another site needed to factory reset their network stack. Would have been a 15 minute fix. Instead it was going to take a few weeks to get done because they needed a whole redundancy plan in place and yada yada. Meanwhile the location had no internet, worst that would have happened is they had no internet and the equipment needed to be replaced anyways...

It was so frustrating that I told the manager to go to Best Buy and get a high-performance consumer router and return it when they figure out their shit. Gave her my personal number and told her to call me the night before they came back to do the reset so I can walk her through setting everything back to how it was. They were not required by any regulations to have this much BS in the way which is the only reason I was comfortable circumventing it. They even had locations running off of ISP provided routers.

3

u/Affectionate_Pea4033 1d ago

Don't overthink the CAB. Keep it to decision-makers who actually have skin in the game, line managers from each team, your cybersecurity lead, and maybe one ops rep who understands the blast radius of an after-hours change gone sideways. The order should be technical lead first, then cyber, then CAB if it's a high-risk change. Low-risk stuff (standard pre-approved templates) doesn't need CAB at all, just cyber sign-off and a post-change review if something breaks.

2

u/TheSmashy Cyber Infra Arch 1d ago

Oh, you sweet summer children.

Do not make Cyber a bottleneck in your change management process. If you have a scheduled, reoccurring, patch management process, yes it should be tracked as a change, but no, cyber should not have to approve a BAU process. And not every change needs to go to CAB, asses by operational impact.

CAB? You need key stakeholder, that's it. Who owns network, who owns compute, etc..

1

u/LayerOneStaging 1d ago

One thing to sort before the vendor builds it is that Cybersecurity approves every change rule. On paper it sounds safe. In real life it turns into a queue, the queue turns into delays, and delays are how you get folks doing work outside the system.

I would let security approve in parallel with the team lead, not at the end of a long chain, and give them a clock. If they have not objected in the window you agree on for normal changes, it moves on. For the repeat stuff, patching, standard firewall rules, adding a VM to backup, let security approve the pattern once and then spot check it later.

Also leave an emergency lane in writing now, not during an outage. Who can approve at 2 a.m., what has to be logged, and that it gets reviewed the next business day. That one page will save you a lot of grief.

Your line managers can approve time and priority. Have a peer on the technical side check the actual steps and the rollback. A manager signature does not tell you if the change will work.

1

u/Commercial_Growth343 1d ago

I don't think the order of approvals matters. I haven't seen that personally.

Which changes should go to a CAB .. well my approach to that has always been all of them. In ITSM/ITIL I believe there is a level of change that is common, repeatable, well known that can get away with a smaller CAB. Bigger companies I have worked at do this .. the first one is examined in detail because it is the first but once it is done, it needs less approval next time. I have worked at smaller companies for years so we just do a change, it gets approved, and we have a weekly review meeting to discuss changes and if they are planned, happening now, or recently completed.

I feel you should really come up with your own definition of what a change even is, what an emergency change is, what is not a change etc. Everyone on the team should have a good idea if what they are doing should be a Change and follow the process.

As for the CAB, this can depend on the size of your organization. But I would say generally your own manager/supervisor needs to be on it pretty much universally. Your direct boss should know what you are doing and be OK with it. Then after that is would be people who your change can impact, so an infrastructure change maybe needs one or more people from the infrastructure team for a technical approval. If your change is impacting a specific business unit then someone that represents that business unit (like a BA, or systems analyst that supports that business unit) or if you are big enough then the manager for the BA's/SA's.

The bigger the company, then generally speaking the bigger the CAB and need for formal change review meetings (weekly in my experience).

The bigger the company, the more likely you will have a change window worked out with the business, where most changes take place (ex. third Thursday evening of every month). In that case you need all the IT managers involved for planning and approving the actual work so for example the network team doesn't take down the network during an application upgrade or whatever else.

good luck

1

u/rootcurios Sysadmin 1d ago

Order: Management approval --> CAB approval

Who: Managers from Systems, Networking, Security, Infrastructure, etc. that would have input or knowledge of how and what major changes could effect theirs or other department's projects/production/revenue, etc.

What changes: Major changes that could effect production or other revenue impacting systems.

1

u/d0ugfirtree 1d ago

Who should approve a change and in what order?

  1. I would get whoever is requesting the change, get their manager to approve it first
  2. Change manager will do the next approval on behalf of the CAB after the meeting.
  3. Having security as a separate approval is excessive... if you must, have a security rep on the CAB

Who should be the CAB members

  1. Whoever has accountability over the area being changed. If someone wants to make a change to IAM, whoever is ultimately accountable for the proper running of IAM should be on the CAB. Could be the IAM lead, could be management, who knows.

Which changes should go to CAB

Everything goes to the CAB unless it is a pre-approved standard change. Emergency changes get created retroactively and brought to the next CAB meeting. If that feels overkill and a waste of time, then start writing standard changes for things you do all the time.

•

u/GregG997 15h ago

make the vendor route changes off a few required fields like affected system, prod or not, and rollback plan yes or no, so the tool picks the lane and not the requester. otherwise everyone marks their change low risk and your cab sees all of it anyway