Q&A with Kelsey Hightower Part 1

As some of you may already know Kelsey Hightower hosted Office Hours at Netris booth at KubeCon Europe 2023. We talked about the future of infrastructure and when to and when not to use Kubernetes; Kelsey has shared his perspective and given career growth pieces of advice.

Kubecon attendees were excited to meet and greet Kelsey Hightower and, of course, they had questions. Here we gathered some of the most interesting interview questions for you.

Question:

What are the best lessons you learned when you started out In tech?

Kelsey: Best Lessons I’ve Learned? Customer service. Everyone thinks it’s technical, right? Most people wanna learn how to code, support DevOps, and platform engineering, but most engineers have no customer service. They don’t know how to talk to people. They don’t know how to inspire people around them. So when you’re an engineer, let’s say you have the right answer. Like, “Hey, we should do it this way,” and no one believes that’s the right answer, so you need another skill. You gonna have to be able to convince them. You gonna put the keyboard down and say, “Hey, this is why, this is the right way of doing it. We’re currently doing it this way. I think this is better, and here is how this works out for all of us.” And at that point, you need your fellow engineer to believe in what you are talking about, and then you can get back to the code. So, I think that’s the most valuable thing I learned that being an engineer is only part of the problem. The other part is being able to inspire people to believe it. I think my favorite quote I learned back then was: “Some people have ideas, and some Ideas have people.” Right? And that’s the way you do it.

 

Question:

I am kind of new to this, and I am currently learning, so I hope I can learn something from this book (Kubernetes Up and Running: 3rd edition). Do you have any tips for beginners?

Kelsey: Tips for beginners. I mean, when you say you’re a beginner, what does that mean?
Attendee: Well, I saw it; I read something.
Kelsey: But do you have any experience of, like, Linux?
Attendee: Yeah. I do.
Kelsey: Okay, so you’re probably not an absolute beginner.
Attendee: Not an absolute.
Kelsey: To me, I think for anyone that has a previous experience, how much of it carries over? So if we look at Linux, the majority of Kubernetes requires Linux to work, right? So, that’s just the thin layer on the top. So, I would say, as a beginner, give yourself credit for the previous experience and then use it when you’re learning a new thing. Because I think with Kubernetes, people think it’s this brand-new revolutionary thing. it is not. It’s mainly using all the Linux capabilities with a few missing pieces on top. So, I would say the first thing I would do, especially when we wrote the book. If you look at the first few chapters, we are trying to remind people of the things that were missing but how important is the rest of the stack. Right? The Kernel is still important; even though containers and Kubernetes do all this work, at the end of the day, they still give it to the Kernel to do the rest, right? Logging is roughly the same. HTTP is roughly the same. So, my advice would be, give yourself credit for the 80% you already know, and then when you look at the new standards, answer the question, “How does this relate to the old stuff?” and you will get your answer faster.

Question:

How do you keep your knowledge up to date? New projects like every day. Kubernetes is evolving quite fast. How to keep up? How to be on top of it?

Kelsey:  All this stuff is probably just the remix of old stuff, right? Service mesh, load-balancer with HTTP proxies, different config, different programming language. But when I look at Envoy, it looks a lot like Nginx. It moves the same type of traffic. It has a different way of configuring? Sure. But the name is roughly similar. It deals with HTTP headers. You can write plugins. Instead of Lua, you use something like WASM, okay. Instead of configuring what a file you can use gRPC, okay. But the fundamentals are roughly the same. So, when I walk in all these booths, I say, “What do you do?” they say, “Cloud-Native, blah, blah, blah, authorization” Okay, I get it. Just like the old thing, next. And so most people. You roughly gonna find anything that’s really new. Because if it’s really new, no one could ever adopt it. You would have to be learning everything from scratch, so most of these things are better user experience, pick any of these projects next to us. “We eliminate latency in your…”
Okay, still networking, L2, L3. They are probably putting some agent and getting some logs, looking at data, and then they gonna re-configure MTU settings to make sure everything matches. I guarantee that’s probably how it works․ And if I’m wrong, it’s gonna give me the delta, right? So I would probably say, just learn the fundamentals. When you think about encryption, just learn the fundamentals. How does TLS work? Why do people use OpenSSL versus BoringSSL? These are fundamentals. And then I would  just
go and look at all the projects and say, “Hey, what’s your way of implementing the fundamentals?” “Oh, I like it that way. I will use it.” So that’s how I stay up to date. So, just mostly noise. Find a person who actually knows what they are talking about. And a person who knows what they’re talking about uses it as historical context. Like I was talking to WASM the other day. Everyone says, “WASM is going to change everything.” I said, “Great. What is missing from WASM today?” “Oh, no sockets, no drivers, no threads.” “And then what is gonna be you are working around?” “Modules, components.” “This sounds familiar. Have we done this before?” Yes. In other languages, in other contexts, we have done this. So, I find that person. After an hour and a half of asking good questions with historical context, I feel my knowledge gap is closed, and I am not ashamed of learning it in public. So, I may say, “I understand it this way,” they say, “That’s wrong.” No, this is the right way of do it. Great. And then, I shrink my knowledge base over time. That’s it. And I think of it as new. I just think of it as a remix of old things. 

 

Question:

As a senior engineer, what can you suggest to junior ones to focus on or specify in the next 3 years?

Kelsey: I like that. In the next three years.
Attendee: I mean, I am interested in collateral orchestration? But Kubernetes is the huge portion, and I don’t want to just dive in and try to learn all these stuff at the same time.
Kelsey: So, when I was starting my career, I just go super deep, with everything that’s in front of me. Because in 20 years you want to have a breadth of experience, but you don’t want to be shallow. You want to be deep as well and so some jobs… I was doing tech support, and I thought about, how do you manage a ticket queue very effectively. You haven’t answered in two days? Closed. Oh, we keep seeing the same ticket over and over, I’m writing the shell script. And so you really figure out how to do test automation quickly, and then I move on to another job, and now we’re managing networking gear. I need to learn everything about Juniper switches. How to back it up, how to update the firmware, how to automate the backup process, and the restore process. Why does IPv6 different IPv4? Why are subnets done this way? How do you deal with the situation when you run out of IP space, and they need to consolidate it. That’s going super deep, right? Why do MTU settings matter? What is the relationship between the MTU setting on a switch and the network card on the Linux OS? How you change it? Again I’m going deep. And when I join companies like Puppet Labs, we need to automate the first two things. And now I can use the depth over here. And so for me, if you’re going to be using Kubernetes, go deep. Why? Because you’re going to go deep, you going to learn about schedulers, and then packing. You going to learn about Linux namespaces, you going to learn about L3 networking and different limitations. And as you go deep the next job you have, you going to be here to carry over some part of that distributed system. So there’s fundamentals in Kubernetes. Forget Kubernetes for a second, but a lot of the distributed system stuff that people learn in college and PhD in theory you get to learn in practice. And so to me, I would say go as deep as you can in the thing that’s in front of you, it will prepare you for the thing in 3 years, right? And then you can choose. And I would say in the early days titles mean everything, right? It means everything. Because it tends to be your own boundaries that you will be comfortable with. I’m a Linux administrator and you going to rise to that occasion. They give you a promotion to Linux administrator level 2. You are going to rise to that occasion. And you have enough guidance on where to hung on your skills. And then later on you’ll realize that the title doesn’t matter at all. Because you going to have all the skills necessary on the team you’re on. Can you write code? That might be helpful. Do you understand how I operating system work? That might be helpful. So that would be kind of my guidance: so pay attention to the thing in front of you, don’t get too distracted. If Kubernetes is in front of you and ChatGPT is over here, what you could do is pause and get stuck in say, “Should I learn ML or should I learn Kubernetes? And then you just waste time being paralyzed. What you do is say, “All right, we’re doing Kubernetes right now, I’m going to go deep. I might run at ChatGPT’s smaller model on Kubernetes. I might figure out how to train something using a TPU attached to a container running on Kubernetes. But I want to stay focused,” right? Versus someone that’s been distracted that’s so shallow they don’t have anything that they can carry forward.

Attendee: So you say, don’t fear to deep dive into what’s in there right now?
Kelsey: Yeah, I gamble with my career. When golang came out, I gambled. I’m like, “You know what? I like this Go thing. I am gonna learn how compiler works; I wanna learn how it does concurrency, how it does memory management. I’m going to write tools in it. Can I replace Python? I was at Puppet Labs, and everything was in Ruby, and I switched to Go, and when I couldn’t write Go there, I chose to leave to a different company where Go was being accepted. Boy, did that work out well for me, right? And the same thing with Kubernetes. When it came out, people thought it was going to die, right? Because remember Swarm and Mesos had the lead. So I made a gamble that I’m going to learn everything, so I started contributing. So I think it’s okay to bet, but make sure you rub fundamentals on it. You don’t want to be the Kubernetes person. You want to be like a distributed systems engineer that happens to be using Kubernetes. So if another company is using Nomad, you say, “Hey, there are some things 80% the same.”

 

Question:

So, do you know any sources that provide good quality mentorship, for example, from people like you or someone with proper knowledge?

Kelsey: No, I mean, for me, my whole career I’ve always built organic relationships, right? Because not everyone that’s smart is good at being a mentor. and I think for me, it’s always been: do my homework and then reach out, and I noticed that it works well. Like what I’m studying about WASM, I’ll do my research, and maybe I’ll make a comment out of it, and all the WASM people open up. And then willing to have conversations and deep dives. And now, that community is starting to be part of my network in some ways. And so to me, I think that’s been the way I’ve done it. And so, if I need 10 mentors, I need 10 people that I can call with questions after I do my homework. So for me, at this point in my career, I just prefer access to the network. So, sometimes like for example, I contribute to open policy agent, and then meanwhile would ask same question. So hey, I’ve given something first. The contribution was small one but then it turns to open up access to having my questions answered about “why they do it this way?” “What came before did they thought was missing that it deserved the whole new project?” And so, for me personally, I just like my network to be strong, and it’s a symbolic. I want it to go both ways. So I want to make sure that I have something to give because, typically, it is easier to ask.

 

Stay tuned for Part 2.