Q&A with Kelsey Hightower Part 2

As some of you may already know Kelsey Hightower hosted Office Hours at Netris booth at KubeCon Europe 2023. We talked about the future of infrastructure and when to and when not to use Kubernetes, the future of Kubernetes and much more. 

We are excited to share the second part of the interview here with you about “future talks” and predictions.

Question: What is your opinion on quantum computing in terms of Kubernetes? Do you see it as a, you know like viable platform when it comes to scaling?

Kelsey: Quantum? This is like talking about spaceships and cars. You’re not gonna drive to the moon, right? There’s a whole different paradigm. We would hope no one tries to run a container on a mainframe. Why would you? Or on a quantum computer.

Attendee: But still, you will have workloads…

Kelsey: You have workloads, but there is no reason to do what we did 40 years ago, right? You have a quantum computer. You hope you have a better programming language, a better way to do computation. Do we really wanna write it in Java? When you wanna better programming language tuned to that environment. It’s like an objective C and IOS. Do you really wanna put Java on your iPhone and Haskell? Unnecessary.
The good thing is with a mobile device, we got a real SDK, so you can access the camera, you can do payments, you have a gyroscope. We’re not gonna reinvent that for every programming language, alright? So, hopefully, if we get a quantum computer, we can get a really powerful SDK. Do you wanna serve a website on a quantum computer? You probably wanna break Bitcoin encryption keys, right? You wanna do something only the quantum computer can do, so I would expect no one in the early days who try to run a container on a quantum computer because the only thing you gonna use it for is something you can’t do with a traditional computer. Plus, it would be too expensive to run an API on a quantum computer, right? You would’ve wanna only use it for something that benefits from right on that.

Attendee: Yeah, of course, I was just asking from the perspective of scheduling and stuff like that, but I completely got your point.

Kelsey:  I would imagine the schedule will be built into that, right? Because it would be so expensive. And maybe it will look like a mainframe will have time-sharing built in, right? Run this job, I can afford about this much and have that do everything. So, my guess is it would be built in. The interface will be a little simpler. It will take a snippet of code and give you the result back in some format that you can use. That’s how I expect it to play out. 

 

 

Question: What are the best lessons you learned when you started out In tech?

Kelsey: Best Lessons I’ve Learned? Customer service. Everyone thinks it’s technical, right? Most people wanna learn how to code, support DevOps, and platform engineering, but most engineers have no customer service. They don’t know how to talk to people. They don’t know how to inspire people around them. So when you’re an engineer, let’s say you have the right answer. Like, “Hey, we should do it this way,” and no one believes that’s the right answer, so you need another skill. You gonna have to be able to convince them. You gonna put the keyboard down and say, “Hey, this is why, this is the right way of doing it. We’re currently doing it this way. I think this is better, and here is how this works out for all of us.” And at that point, you need your fellow engineer to believe in what you are talking about, and then you can get back to the code. So, I think that’s the most valuable thing I learned that being an engineer is only part of the problem. The other part is being able to inspire people to believe it. I think my favorite quote I learned back then was: “Some people have ideas, and some Ideas have people.” Right? And that’s the way you do it.

Question: How would you start building a data analytics platform based on Kubernetes?

Kelsey: Data?

Attendee: Data analytics platform.

Kelsey: Data on Kubernetes? Kubernetes shouldn’t make a difference. That’s my answer. How would you build it on VMs?

Attendee: I would build with AWS or Google Cloud.

Kelsey: You would pick some VMs, and if you wanna do data analytics platform, you gonna make some decisions. How you gonna ingest data? What do you wanna store? And how do you want people access it, right? Those would be the primary things you would ask. So Kubernetes becomes a very small part of that equation. Do you wanna use Kafka? You wanna use Snowflake, BigQuery? Fine, that’s the thing for most people. You can go on and build it from scratch, but Kubernetes is an implementation detail. So, I would answer the question, regardless of Kubernetes, how would you do it? And if you have a good answer with VMs, it’s probably the same answer for Kubernetes. And if I was doing it, I would probably use BigQuery and skip everything else. I mean, seriously, I don’t need a Rube Goldberg machine to put data in place when the Query is over. So, get the answer for VMs and then just do a little bit of work to get the same solution on Kubernetes if it’s helpful. And what you might find, you are probably fighting Kubernetes more for data kind of workloads than the value that I think it would bring. So, my answer: figure it out for VMs, get a good answer there, and then the same answer probably works for Kube.

 

Question: How do you think, what will happen next? Kubernetes will die, Assembler will be a replacement, something else? Do you see a trend that is rising?

Kelsey: You want Kubernetes to die.

Attendee: No, no, I am asking if there is an option.

Kelsey: I am saying you probably want it to because…

Attendee: No, I am getting money on this, so no.

Kelsey: Yeah, the motivation, but the thing is, someone is going to replace it.

Attendee: And the tool that you are seeing that is rising that could replace Kubernetes?

Kelsey: So, all the problems with Kubernetes, someone is working on it. So when the Docker came out, there were a lot of people were there identifying problems with a Docker. Only one container at a time is a problem. The networking model was very limited in early days of Docker; that’s a problem. Docker didn’t have a scheduler. You’d run it on one machine. If the machine goes away, what happens? You gotta fix that. So if you collect all the problems wrong with Docker, you end up with Nomad or Kubernetes. So let’s take Kubernetes; what’s wrong with Kubernetes today? It’s pretty complex, right? People are out here trying to recreate a whole cloud inside of Kubernetes. Do you really need to run Prometheus inside of Kubernetes?
And so I think what will happen is people will realize there are parts of Kubernetes that you can extract, and you don’t need Kubernetes. If you just wanna run computes, right? In GCP, we have cloud run. There is no Kubernetes anywhere, not underneath it, not hidden, it’s not there at all, but we have parts of Kubernetes API. We have the podspec in a form Knative. So we gonna extract out the useful part and get rid of the part we don’t need. Someone somewhere is collapsing Kubernetes stack down, like K3s, for example. You look at the implementation. Do you really need all those moving parts, or can you simplify it. So I think the next thing will come to solve the issues we have. Do you really wanna write yaml for the rest of your IT career?  Probably not. So my guess is someone is gonna figure out a better way to declare what we wanna do. And when that day comes, we will make progress. And progress means that we have to leave this behind. I don’t think it goes anywhere for 20 years, but it won’t be a whole conference of people so excited to talk about Kubernetes this much. 

Question: One of the things that I think is important as IT members, we know everything about tools and basic stuff, right? But how would you address this kind of tool and innovation in the business model? How did you address them with people who’s not technical? For example, how do you sell Kubernetes to someone that doesn’t know?

Kelsey: I wouldn’t try to sell Kubernetes to someone in business. Let’s say you wanna open a flower store. You’re not gonna sell concrete to florist. I am selling flowers. I just need a building with windows to sell my flowers. I don’t wanna talk about innovation in drywall. Not important. Just give me a safe building. Maybe we can talk about square footage, maybe we can talk about better advertising and positioning my flowers so that they attract people as they walk by. That’s interesting conversation. But to go and say, “Hey, we wanna replace the concrete.” It’s like, I’m pretty sure there have been investments in concrete, but I don’t care. So, if you’re running a real business, you don’t care about Linux nor Kubernetes, so you gotta sell something else. Let’s say I have an e-commerce site, and it goes down one hour every day, and we are losing sales, and I am in business. You’re saying, “Hey, I can make that no longer happen. Instead of one hour a day, we may able to get it to one hour a month.” If I am in business, that’s enough. We have a deal. If you literally can make it not go down for one hour per day, we have a deal. I don’t care what you use. You can use Nomad, you can use Kubernetes, you can use Lambda, as long as it’s not down an hour every day. Now you may know, “Oh, Kubernetes will do some restarting resilience, repeatability, blah blah blah.” Do we get our one hour? Yes. Because if you bring Kubernetes and it’s still down one hour a day, doesn’t matter. So to a business person, I tend to not talk about the tools. I tend to say, “Hey, I am the expert here. I understand what tools are necessary to make this work, and I understand the problem, and so when I go pick the tools, you have to trust me. I am the expert on the sight. You are the business owner, so I trust you that that’s a real problem that needs to be solved, and so I would use the right tool.”  If you call someone to fix your house, they look around and say, “Oh…” and reach to their toolbag. Do you really care what tool they pull out of the toolbag? You don’t care. You say, “Hey, whatever you have to do. Let me know when you’re finished.” Right? So I think the same way when it comes to our tools. Talk about the issue, and you, as your team, will going to solve the issue. And we will probably use the most efficient tool for the job. 

Question: If you wake up tomorrow, and there will be no cloud, no CNCF, no Kubernetes. Will you retire, or will you choose a different…?

Kelsey: I am gonna retire before that.

Attendee: Tomorrow morning?

Kelsey: Not that early. But soonish, I won’t be doing as much longer. If there was no cloud, no Kubernetes, I am just helpful in that context. But if you gave me another context, like balancing someone’s financial system or budget or my own budget. I am gonna be just as creative. I am gonna study the history, I am gonna study the treasury rates, I am gonna study the interest rates. I am gonna wonder why stocks go up and down, and then I will be in that mind, or I will be in that mode, using the same mindset. So, I think for most of us technologists, it just happens that we chose to work on tools, but most of my career was not working on tools. Before working at CoreOS or Puppet Labs, I was in financial services, dealing with banking problems, right. And I just happened to reach the tools to help me solve banking problems. So if there is no cloud, that would be wonderful. Maybe we would all just go outside, can we throw mobile phones in there too? And maybe we would go outside again, right? So, to me, I think… I think it’s just true for most people in technology. If you gave them anything else, we would probably figure out how it works, try to make a better version of it, and then use that in our day-to-day professions. So, yeah, I don’t know if… I don’t really look at Kubernetes too much anymore. That’s been true for four to five years, so I would say all this stuff is temporary, but I am gonna enjoy it while it’s here. 

Question: Following up on the question about the future of Kubernetes, where do you see Kubernetes like in five years from now?

Kelsey: I mean, it will still be here, like, Linux is thirty years old, Mainframe is older. They are both still here. Do you wanna be working on it is the question?

Attendee: I would say yes.

Kelsey: But to me, that would be a sad answer because how much growth will you have in five years? I hope you grow so much in five years that you won’t care about Kubernetes. For what? Does it mean that Kubernetes is less valuable… that you’ve grown so far that Kubernetes is not enough for what you’re trying to achieve. I don’t write Bash, not because Bash is bad. Bash is not the tool for me to work on the kind of prompts that I have, so there is Kubernetes, and then there’s you, and ideally, you’re growing at a faster rate than Kubernetes is, because at this point, Kubernetes is so popular in production, it can’t change very fast. There are people who have been in this industry for a long time, and if you talk to them, some of them, they have 20 years of one-year experience. They only know Ubuntu. That’s it. They know it well but nothing else, so if you’re asking them to solve a problem, they only know what Ubuntu can do, no other solutions. So, the way I would look at it is you can set a goal for yourself. “ I am gonna continue to grow,” and then maybe one day you look back at Kubernetes and say you might be the person who replaces it, right? When Docker came out, everyone thought they gonna do Docker for twenty years, right? And then eventually something came, and now the Docker is underneath. It’s kinda buried now because there is progress. So that’s what I would hope. That’s what I see for myself. I see progress, right? So Kubernetes will be fine. 

 

Question: What do you think about how the rising AI tools and technology will influence the cloud-native era or our jobs or our tasks?

Kelsey: I mean, we are having this debate on Twitter. A lot of people are excited about using ChatGPT to generate yaml files for Kubernetes. So, if I see that, I am like, “Why are you having anyone generate yaml files?” Because now the thing as guessing, right? It is trying to predict the yaml file, so give me a Kubernetes manifest to deploy this container. And it’s like, “I think you want this.” And if you don’t know what you’re doing, you’re like, “Hey! It looks right, kubectl apply, it’s working?” Now you look at it as an expert and say, “Hey, there’s this sidecar, and it says “steal my data,” why is this sitecar in here?” “I don’t know. ChatGPT did it.” So you’re guessing.

Attendee: I can see your point, but I can’t see an example that will happen or happened in the past. ChatGPT, for example, generated a freaky sidecar inside the pod.

Kelsey: Yeah, I guess where I am getting is ChatGPT can only guess and approximate. It may accurately generate a working manifest. It may not be what you want, and how would you know if you want it or not? You have to look at it. So, on the positive side, it’s good. It’s like autocomplete, right? If I’m typing, I don’t know what to do; auto-complete steps in and gives me the default value. If I don’t know what the default value for CPU allocation is or CPU limit, I might just accept it, but it may cause problems in production. So ChatGPT is something that can give you the first draft. I think that’s pretty good. Give me a working manifest, but now I need to go step through it. But in production, we don’t want to guess. We want to know. I know I’ve given you four GBs of RAM. That’s all you have. That’s all I want to give you. I don’t want to guess that’s what you get. So I think in that the opposite of a generative AI would be like basic symbolic programming. You get 4 GBs. Now the user… I don’t want them to ever have to write yaml; that would be sad. So what do you do? You say, “Hey, listen, just give me a small key-value file, name of the container, do you have health checks? If so, what path? How much memory do you think you want? Maybe you want 8 GBs, but I will still give you only 4. But I would like to know what you think you want. It’s only have three Fields: container image, where is health checks, and how much memory do you want. That’s it. Very simple. And I take that and I can just use Helm or a Kapp. Do I really need AI to generate a template?

Attendee: For generating, I don’t think. Because you can go to documentation and copy past the…

Kelsey: Or ChatGPT can give you the first draft. As a team, you want to know what each of those fields mean because you’re the administrator. You’re supposed to be responsible. You’re supposed to be knowledgeable on what’s in there, so if you want to get started with Stackoverflow, copy and paste on Stackoverflow, so people copy and paste from the docs. Or you can copy and paste from  ChatGPT. What’s the difference? It’s fine, but you want to make sure that the thing you choose to standardize on makes sense. And then you parametrize it, and so now when you go to your user, you’re not asking him to create yaml, so there’s no need for ChatGPT to generate anything. I’m asking you to make a decision. What container do you want to use explicitly? We don’t need to guess. Which one do you want to use? And so I only want to give my user a prompt that says, “Just tell me the things, only things necessary, no need for yaml.” So when it comes to infrastructure as code and config, I think we should probably make sure they were only presenting interfaces and APIs that people can actually use.

Attendee: And the do you see some specific tasks that could help us or…

Kelsey: Well, so AI is gonna be good at things like, “Should I restart this container?” You’re running Java, and every 12 days or so, you get an out-of-memory exception. Every 12 days, right? This much traffic flows, your HEAP looks like this, and every 12 days at slightly different times, it crashes unpredictably. And so, you could say, “Hey,  I’m happy to restart this container between the hours of 1 a.m. and 3 a.m., but only do it when it’s necessary.” Now what a decent model, everything I just explain to you as our model of the world when the world looks like this. “I predict this time to restart. But I’m only going to do it at 1 a.m. to 3 a.m.” So if I’m using some automation that’s like self-healing automation, I’ve trained it on historical crashes. This thing is going to be pretty good at predicting when the next crash will come, right? Because all the data elements match, and it’s like, “Hey, you are in a state where you probably need to restart all of these.” “Why? there’s nothing wrong yet.” “I know there’s nothing wrong yet, but all the signals are clear, and for this other app written in Ruby, I’m seeing something similar, even though not Java, it is looking something similar, but that’s what models are good for, right?  They are good for proximation, and we train them. So I imagine everyone is using their crash data to make this model better. And so one day in the future, if someone has a crash detector for Kubernetes, not the thing will have real-time access to everyone’s cluster, but we have the historical information so we made a decent model. And now you can have it watch your cluster, and when it sees something, it can flag and say, “Hey, these three pods are going to crash in the next 2 days, 98% percent accuracy.” That’s a good use of AI, right? It takes the model, it tries to make its best prediction and then you can decide as a human to take action. I like that. That’s a good model. 

If you missed the part 1 about career advice, you can read it here.

Stay tuned for Part 3.