Day in the life of a Lead Principal Information Security Engineer

Whenever I get the “what do you do” question, whether that’s from a stranger at an event, or someone making conversation it’s typically varying somewhere between “I’m in cybersecurity” all the way up to “I help shape the strategic vision, translate that to tactical action, and lead a team of engineers toward the goal”. Granted those are gross simplifications but I thought it might be nice to step through a day in my role at my current organization to see if it’s something you might like to pursue.

The question which arises for you, dear reader, is “Would I want to be a Lead Principal Engineer?” and to that I say, what we always say – “It Depends”. However, let’s go through how my team and organization works through our days on as much of a “standard” day as possible, even though those are few and far between.

First things first

I like to set my day up around what I’ve found to be successful: I can think harder and get more tasks done in the morning. Therefore I like to try to get on an hour or two before most of my team (they’re typically in Pacific Time while I’m in Mountain) and undertake the following schedule as best I can:

  • Organize the day – take a look at meetings, tickets, and messages to see if there are any fires that require immediate attention. Look at spillover from the prior day/week and re-prioritize as appropriate. I like to put these into buckets – needs done, good to do, nice to have done.
  • After my day is organized the most important thing is to get something done or clear progress made – no matter how small that might be. I tell my team and believe wholeheartedly that if you aren’t getting victories and feeling those wins, you’re going to get frustrated, exhausted and eventually burnt out.
  • From here the day varies but is usually meetings – which I’ll be clear, I hate. I love seeing my team and interacting but meetings frustrate me to no end because there are typically too many, without clear goals/outcomes, and can easily be sidetracked or hijacked if not managed properly. We’ll touch more on meetings in a bit.
  • I do my best to block my lunch out and a little time for a walk or some time away from the screen – even if it’s only 15 minutes, it’s vital to disconnect and reset for the afternoon.
  • After lunch also is pretty meeting heavy but usually around more creative work and problem solving – which serves my mindset and rhythms better.
  • End of the day rounds down with a few meetings and if not, is used to set up the next day and reflect on what was accomplished. Sometimes it’s also slamming the laptop shut to run out and pick up the kids or it might be staying late if an incident is popping off.

If I can stick to the above it’s a good day – however, as we know in security and IT in general, systems, networks, and attackers have no courtesy to think of you and your schedule. We categorize the stuff that distracts us and is unplanned work as Thrash. I may do another post on how we tried to work in Sprints/Agile frameworks and failed miserably at some point because I do believe our lessons were translatable and relatable.

What kind of work gets done

This might require somewhat of a history lesson but when I joined I was essentially the second security hire which meant a few things – I’d have to get up to speed and hit the ground running in the worst recruiting terms possible, but also we’d be setting up the scaffold of not only what to work on, but how to work going forward. With this in mind we decided to embrace the Center for Internet Security Critical Security Controls as a great way not only to achieve a foundational framework adherence but also introduce concepts regarding security to the larger organization. With this in mind, we set out to score ourselves on the controls and outline the work to move the needle on those controls.

So with all that context and exposition out of the way, what does a typical work item look like? Let’s just say we’re going to look at Control 1 – Inventory of Enterprise Assets. If we just assume that we don’t have anything in place the first thing I do is look at the policy stack – does anything say “you shall have an inventory of all enterprise assets”? If not, start drafting a new one. This falls into my wheelhouse because I’ve always had above average communication skills and I truly like to flex them. If my day includes a policy review and writing I’m going to look at everything – every word, letter, comma – everything and be sure that the policy states the desired result, without being too verbose, but also being ambiguous enough for us to address technical and more detail-oriented items like technology type, implementation type, instructions for deployment down the line into Standards, Procedures, Guidelines, etc.

Some might question – does policy work fall to a Security Engineer? If you’ve been reading or talking to me in any capacity you’ll know my response, say it with me;

IT DEPENDS!

In some organizations you’ll have a compliance or GRC function that oversees policy review, development, approvals and more. In our organization as an example, we’re smaller and we do have compliance functions but the InfoSec team is much closer to and has more experience with policy development so we just take care of it. Will that stay true forever? Probably not – but it’s fantastic practice for myself and my team as at a lot of other organizations this would just be handed off to another team and the most exposure InfoSec might have would be reading or linking to the policies when breached.

What else? Well if we’re still working on Control 1 the next step is to determine if it’s implemented and on how many systems – none, some, most, or all? For a lot of teams determining if you have a complete asset inventory is a job that’s never done – so we have to go out to all the teams that touch assets or have them under their purview and have conversations which usually lead to “do you guys want read access?”. From there we can determine how many assets we think we have, how many are in the varying inventory/control systems, and start to at least understand some of the numbers. But even this process is broken down into smaller tasks and processes – I’ll ask the questions about how large of a margin of error are we willing to accept, what do we know we don’t know, what don’t we know, and what should we know but we can’t prove?

Now so far we’ve talked about the tasks I might do on a given day but they’ve been largely fluid – not “go code this thing” or “run vuln scan on X”. There’s a good reason for that – I’ll never say that I’m above certain work but when you reach a certain level you also have to be comfortable giving away work (delegating) and allowing your team to learn, grow, and do the job in their own ways. A lot of my work at my level now is more about being given a problem or statement and breaking that down into solvable chunks. Sometimes they’re massive issues or have quite substantial business impact – sometimes it’s “hey we’re looking to bring on a new system – what do you think?”. So as you start to come up the ladder in experience – regardless of if you stay an individual contributor or go the management route – you’ll start to be consulted as a resource and advisor internally.

So if you’re keeping track so far the kinds of work that get done for a position like Principal Engineer are;

  • Largely consultative, strategic planning, and breaking projects into achievable tasks and milestones. This includes research and delegation to proper teammates for completion of the work.
  • Meetings – these start taking more and more time so my best advice is be very deliberate about your meetings and be ok saying know.

So you might wonder – do you do any task level work? And that answer is yes, but sometimes it’s a matter of taking that work instead of delegating it. Checking in across your team is vital to understand their workloads and what else they can take on without overloading them. Sometimes this does mean you get stuck with the less-than-thrilling work, but that’s ok – or at least you should be ok with it because if you delegate all the trash, your team will know.

A Word on Meetings

I’ve talked a bit about meetings and while I write this I’m actually reflecting quite a bit on meetings and strategy surrounding the calendar and going back in November I know I’m going to be more on purpose. I recently saw a fantastic tweet from organizational psychologist Adam Grant;

I love this idea of categorizing and tagging meetings (I’m going to do this directly on my Outlook Calendar) with clear intent of the outcome. If we go into a decision meeting, we must reach a decision – if it’s a learning meeting – what are the learning outcomes? If it’s bonding – we can be a bit more loose but being sure to democratize time and ensure everyone feels welcome and heard. Lastly, if a “Do” meeting we’re going to have a desired outcome and work toward that outcome. I like that there is not any sort of “Brainstorming” category because I believe, as I’ve seen mentioned, that brainstorming on a call is counterproductive – the loudest voices are heard and the consensus falls to the first couple of ideas or to the idea the boss proposes. I’ve even tested this by challenging ideas and noticing how quickly someone flips their thinking to mine, which is not always correct!

Days to weeks, months, years

If you’re like me, time seems to be a fleeting concept and is entirely too easy to get away from me. I’m not the most organized person so I have to be focused and on purpose about how and where I spend my time. That being said I’m also not willing to compromise my family health and strength for a job – thankfully I now work for a place that respects and expects that themselves.

The other piece that isn’t touched on here is Thrash – that unplanned work that just seems to come up. I like the term thrash because it can come off a bit volatile but that’s what this work is. It’s taking our attention away from the project and longer term work toward maturity in order to answer questions, investigate escalations, or potentially help guide conversations for the business regarding security. Thrash is unfortunately a necessary evil in cybersecurity/InfoSec – every time you have a plan it’s going to get shredded so the best you can do is have a few plans to call audibles and be able to adjust.

What Does the Future of This Role Hold?

Always tough to say, however, my desires are still relatively unclear and change on a weekly basis. I have said a few times I don’t want to be a CISO because they tend to have short lifespans, however, it’s a natural progression in the management track at some point. I could stay in engineering and building which I do enjoy, but may need to make a very clear decision to abandon the management and higher level managerial duties that I’ve inherited. There may also be room for expansion into a formal program management, architecture, etc. but for now, I’m very happy to continue building a program and maturing a mid-size organization and working with a damn good team who I love seeing grow alongside me.