Press "Enter" to skip to content

ICYMI: Release to Production, part 2

The phrase “release to production” conjures a very specific set of thoughts and even emotions for folks who live, breath, and work with technology. Some of those thoughts and feelings are positive, while others are fraught with conflict. At the same time, those of us who are active in our religious community experience a different kind of “release to production” – releasing our children to the production environment of our faiths, whether that is teaching abroad, missionary work, or adult religious education that takes our young adult across the globe. And like our IT-based production release experiences, we watch our kids transition into chaotic systems, where parental observability is minimal even as the probability of encountering unknown-unknown error types grows. This week we continue the discussion from the last episode, where Leon and Josh to look at what our IT discipline can teach us about how to make this phase of the parental production cycle easier.

Kate: 00:00 Welcome to our podcast where we talk about the interesting, frustrating and inspiring experience we have as people with strongly held religious views working in corporate IT. We’re not here to preach or teach you our religion or lack thereof. We’re here to explore ways we make our career as IT professionals mesh or at least not conflict with our religious life. This is Technically Religious.

Leon: 00:24 This is a continuation of the discussion we started last week. Thank you for coming back to join our conversation.

Josh: 00:31 You know, there’s a moment in my and my childhood that I think accurately reflects my approach or how I got into IT. And I really wished that I could have had somebody at the top of Devil’s Run with me who could say, “Look, now young Josh, this is not a good idea.” So you have to picture this. It’s this, the largest hill in my neighborhood and Ontario and it is a, a run that goes down, hits a flat top and then goes down again, uh, into this grassy meadow before there’s a highway. And there are trees that are grown, that are grown in across the path. And here I am, I’m probably seven years old, right? This is the 80s. There are no bike hel, there are no like bike helmets. No. And I’m on my BMX, right. Uh, no suspension. It’s not like I was, you know, dry, uh, riding a mountain bike with, you know, eight inches of travel on the front end and three in the back. Like this is teeth chattering. And I friends and I of course are at the top of this run. No one locally is Devil’s Run. And I’m like, I can totally do this. And so I set off down this hill and about, oh, about a third of the way I realize I’m in trouble. Not only are my fillings rattling out of my teeth, right, but I, I’m, I’m losing control. And then I hit the middle, this, this flat top and I’m like, oh, I can. And then I hit the second part of the hill and I’m flying down.

[chattering noise]

Just the chatter the entire way I get to this meadow. And I realize that there’s a fence coming up. Cause that’s the only thing between me and the highway. I slam on my brakes. If you’ve ever slammed on your brakes and a grassy meadow, you do not stop. You just slide. I crashed headlong into this chain link fence. The next thing I remember was my friends standing over me. “Hey, are you okay?” I don’t know how long I was out. They, they walked down, um, Devil’s Run after seeing my spectacular run. I wish, I wish that someone had said, hey, you know, you should probably take a different route to the bottom of this hill. But I got there really quickly and that’s kind of like my IT career. I got to my it career really quickly. I’m only 40 (ahem!) something and I’ve been in it for 21 years and there are a lot of people that are a lot older than I am that, you know, they did the, they did the traditional route and didn’t get into the right t career until they were 25 or 26.

Leon: 03:10 Sure. Right. And Yeah, the subtitle for that. And for some of us, perhaps the epitaph on our tombstone will be, “Mistakes were made. Lessons were learned”, and that’s…sure!

Josh: 03:25 There was a lesson to learn in that?

Leon: 03:26 Yeah, there probably was. There’s a few, few functional lessons. So in the category of “Mistakes were made. Lessons were learned.” And really it was the, the impetus for this entire episode, uh, my son is, uh, actually about to be on his way back from Israel. Um, I’d like to point out that he left last Wednesday, uh, for a year of yeshiva and he’s coming home. Uh, it was, uh, a challenge from start to finish. He got there and things were not as he expected. Um, I will, I will say that he is a fairly resilient kid. He’s done traveling. He’s, you know, he’s been to a high school yeshiva program that wasn’t at home. Yes. There was some, um, some homesickness that was certainly involved. Yes. There were normal dorm shenanigans that occur in every dorm situation, you know, “Hey, I want that spot. You can’t have that spot!”, You know, that kind of stuff. But, uh, there was also a set of circumstances that were not related to that. They also were not of the caliber of civil unrest, mind you, but, um, just a lot of things that didn’t match the set of expectations that he had going in. And by the time we got the better part of a week in, he was so miserable and so unable to, to change his frame of mind that nothing was going to work. And we also realized that everything that we wanted for him, everything that we, and by we, I mean my wife, myself and him, were going to get out of this experience at yeshiva wasn’t going to occur. And even if it did occur, it wasn’t worth or greater than the challenges he had faced already and the challenges that we’re clearly still going to occur while he was there. So we made the decision this afternoon and, um, got a ticket and he’s, he’s gonna fly home tomorrow. So this is effectively the same as you know, catastrophic failure and a rollback in, you know, in production. That you have your change control window, you have everything plotted out and things simply nothing installs or deploys the way you expect it to. And you find yourself at 2:30 AM with three more hours on the change window to go saying, hmm, no, we got to pull the plug, roll it back and we’ll try again some other time, but we have to sit back. So that’s, that’s the story. Um, so let’s, you know, so let’s talk about this in an IT context.

Josh: 06:11 You know, the, this is a tough one, right? Because in in the IT context, if we have to roll back our change, the change management folks are going to ding our change scores, they’re going to say, “Hey you, you failed to deploy”, and I will say that my current employer is changing that mindset. There is no longer a, a ding for rolling back a change because it means that you did your T-minus an you were executing and you recognized that this is not going to work the way that we wanted it to work, so we’re going to roll it back because we want to protect the customer. Or in this case, you want to protect your, your child. You.

Leon: 06:54 Right.

Josh: 06:54 You’re like, “Look, you being at yeshiva is just not going to work. We’re going to bring it back, we’re going to re-plan, and then we’re going to redeploy and probably in another direction.” I mean, we’ve talked about this before, right? Sometimes you have to walk the path that you were not supposed to walk only as far as you needed to so that you could realize it’s not the path and then come back and walked confidently down the path that is the correct path for you. Much like me pursuing my, my law degree just was not going to happen.

Leon: 07:24 But you never sat there and said, “What if? What if?” No, you, you had enough of the what if to say no, no, no. I know. I know what that one was. Yeah.

Josh: 07:32 I also wanted to be a stockbroker. I made it, I made it to the first class of my, or the first lesson of my first two classes and I was like, no, I do not want to be a stockbroker. Did you know the stock brokers have to do math?

Leon: 07:46 Yeah, yeah, yeah. You’re doing a lot of it.

Josh: 07:48 I didn’t, I just thought they made money. It was weird. I had the things you don’t know when you’re 21.

Leon: 07:53 Yeah. Yeah. This is difference between counterfeiters and stockbrokers.

Josh: 07:56 Oh, weird.

Leon: 07:57 There’s a joke there. I, I’m not good enough for financials to be able to finish that joke. So feel free to like leave a comment on the podcast about like how the, what the punch line for that would be. Um, so, so when we think about rollbacks to production, I think in it context it happens. But then we think about, all right, what can we do? Not just to make sure that this thing doesn’t fail in the rollout, but how we can change our, uh, it culture, our it processes so that rollbacks occur less often. And you brought up a really good example, um, you know, as, as a true DevOps aficionado, you’re going to invoke the holy name of Netflix.

Josh: 08:36 Of course, right? Netflix. Netflix is “The” company when it comes to, hey, let’s break things. Uh, they introduced this idea of chaos monkey and it was actually built on a platform that allowed for this continuous, continuous deployment model and they would inject these problems. Uh, so the idea was that they wanted to see what would happen when there was a quote unquote random failure. Uh, so they, they developed this, this platform that was shut down an instance and did it impact us? And if it did, oh, that’s interesting. Why did it, ah, let’s go investigate. And so they would do the root cause and then resolve it. Um, chaos monkey has gone a step further now. Uh, one of the, one of the inventors of that methodology and of that platform has developed a platform called gremlins and gremlins are, are, I mean, they’re exactly like those little evil creatures that were in the bad eighties movie that we watch over Christmas of the same name. Right? They’re like, “Terminate an instance? Uh, no, no, no, no. We’re going to rail the CPU in your box. Oh, we’re going to fill a volume. Oh, we’re going to steal your swap memory!” Um, that whole idea of let’s, let’s inject some failure into that deployment model just to see what happens. That’s what I think is really important.

Leon: 09:57 Um, so, and just to clarify for folks who may not be familiar with the term or as familiar with IT, this isn’t just breaking things for the sake of breaking things. This is breaking things to then see what the effect is and build a product that is more resilient to these random breakages. And not only that, but to build teams that think in a very particular way about what could go wrong. Um, just to extend the Gremlin concept even further. I heard that, I’m not sure if it was at Netflix or somewhere else that the chaos monkey visits the humans.

Josh: 10:32 Oh boy.

Leon: 10:33 Um, right. And does not inject chaos into them. But what it does is, uh, somebody will show up at somebody’s desk right before a release to production and say, you are really, really sick right now. You have to go home for the rest of the day with pay, but you have to go and the person will say, “But, but, but I was part of the release team.” Like I, we know, we’ll let you know who, what happened tomorrow. Good luck. You know, this has been a visit from your friendly neighborhood chaos monkey. So you know, what happens when a particular person isn’t available. And of course, again, the, the, you know, the next day they go back for a postmortem and say, well, because you know, Sarah wasn’t there. Um, we didn’t have somebody who do that. Oh, that’s really interesting. Um, since I know that Sarah is very committed to being able to take vacations and actually be sick, sometimes we probably ought to figure out how we can have some redundancy in our human processes. Um, you know, so that that doesn’t affect us when it matters.

Josh: 11:33 Um, I just want to point out that the last week, my team of three, I quickly became a team of one and it just, you know, PTO and a great opportunity to go out and meet with a vendor and suddenly Joshua is running solo. Um, right. We survived, although we like to inject chaos just for fun.

Leon: 11:53 I’m not saying chaos isn’t entertaining when it occurs to other people. [Laughter] It’s was it Mel Brooks who said the difference between comedy and tragedy, if you fall down a manhole cover, that’s comedy. If I get a paper cut, that’s tragedy. So yeah, the chaos monkey is great, but uh, it’s, you know, when it happens to somebody else.

Josh: 12:15 So how would we apply this then, Leon to, to our families, right. Uh, I think I have some experiences that my family has gone through in the past decade, right? My daughter was diagnosed with scoliosis, uh, had back surgery. She’s got, uh, 21 inches of titanium rod on each side, or sorry, 16 inches of titanium rod on each side of her spine. Uh, 21 or 22 titanium screws in her back. And, um, she did a, uh, Trek which, and Mormon, uh, in Mormon culture, I wouldn’t say theology, Mormon culture. Um, they reenact a, a pioneer journey, so they have handcarts and they, they drive them. She did that. Uh, 3.5 months after back surgery. I mean, my son on a mission, um, you know, was diagnosed with Tourette’s, which made conversation very hard. Now he’s out doing missionary work and loves to talk to people. Um, and then on my own, my own family, right. I, I suffer from depression and I, my, my work toward getting promoted happened to coincide with a really difficult depressive episode. Um, so I mean, I, I, for me and my family, I think that those experiences have taught us this. And I do love baseball. So this, yes, as the baseball analogy, when life is throwing you curve balls, just swing away. I mean when people look at those things, we were like, “Oh, well, you know, Josh, you know, he, he has, he has depression”, but when you swing away at those curves and you, you know, you pull one out of the park, uh, for me that, that, uh, allowing that chaos into our lives, uh, it allows, it allows the acknowledgement that, “Yeah, I don’t have control over this thing, but I am still going to take an active, active role.” But I mean, how do I take that and how do we instill that into my kids. Obviously I, I’ve done it, but I, I mean, I don’t know how I did it.

Leon: 14:13 So I think you’re, I think the analogy is good and I think the point is good that Netflix said, look, failures are going to occur. So, the only way we can get better at them is to keep experiencing failures and keep on growing from them. But we’re not going to wait for the failures to happen to us. We’re going to actively seek those failure modes. Now that doesn’t mean uh, again, quoting Barbara Collaroso, uh so, uh, class if you don’t cross the street. But if you don’t look both ways before crossing the street, something bad could happen. “Jenny, go run and show them!” Like you don’t do that. So, but as you, you can’t watch, but I almost almost made him spit coffee out of his nose. Um, or whatever you’re drinking probably wasn’t coffee. Um, so I think that finding situations for our family to go through with which are less than perfect, which are, um, perhaps a little fraught or have the potential to be a little bit fraught, whether that is, you know, moving, I mean, just simply, you know, moving, moving to a new, a new home to new school, to new city, I think that causes a lot of families, a lot of, uh, concern. What are we going to do to the kids? We’re going to teach them how to move. We’re going to teach them that things change. Um, I think moving every month is probably a little excessive, but I think we can look for those as opportunities, not just as challenges. Um, if you know, somebody in the family speaks a different length, is able to speak another language. I think having that person insist on speaking it and allowing the rest of the family to adapt to that. Um, I’ve seen families where one parent speaks only one language. The other, so the family I’m thinking of, he’s from Spain. She is from Switzerland. They speak all the languages, they speak together, so they each speak Spanish, French, German or schweizerdeutsch in English. They speak all of those. So Dad speaks to their daughter in Spanish only. Mom speaks to the child in schweizerdeutsch only. And they speak to each other in front of the child in French only.

Josh: 16:22 Oh my goodness!

Leon: 16:23 So that the kid understood that there were different modalities to speaking. When I’m talking to Daddy, I have to use these words. When I’m talking to Mommy, I have to speak up these words and if they’re both here, I have to use these other words. But it taught the child to sort of mental resilience. Um, that I think is admirable and I’ve seen families do it that way. I’ve also seen families do it where dinner time is, you know, Spanish time. We’re just at, at this meal, this is the only language we are going to speak. Good luck. You will not damage your children doing that. Uh, pulling from my own personal experience. Uh, the way that the Adato family takes vacations is relatively unique. Uh, I’ll try to tell this briefly, but um, when we go on vacation, we don’t tell our kids that we’re going, we don’t tell them where they’re going. We don’t tell them how long they will be gone. What I mean by that is that my wife and I plan the vacation like you do, but we don’t tell our kids anything about it. It’s done completely in secret. And then on the day that we leave, we usually, uh, wake them up around three o’clock in the morning. A lot of our vacations are driving vacations. So we wake them up around three o’clock in the morning and sing to them [singing] “We going on on an adventure. We’re going on!” As the kids got older, they would swear at us more as we did that because we were waking them up and then they know what’s coming. We’d load them into the car in their pajamas and we start driving. About 20 minutes in, we’d start giving them clues and we’d continue to give them clues, uh, for the next four or five hours. When they were young, this kept them occupied and out of the “Are we there yet?”-mode since they wouldn’t know are we there yet? Cause they don’t know where there is but the clues were not particularly helpful. For example, we were going to Boston and some of our clues was like the little magnet tiles. You know there was a one and a two… One if by land two if by sea? [groan] There was also, there was a Minute Maid orange juice container that was cut out in the shape of a guy for the Minute Men. And, uh, we also had a little stone and a matchbox car… For Plymouth Rock. Yeah.

Josh: 18:33 Oh..wow….

Leon: 18:33 Since I said we were going to Boston, all of that made sense. But since we didn’t, they, they sometimes got really obscure. Like we went to Israel, we did take a trip to Israel and one of the clues there was a model airplane with holes drilled in it. Um, which did not mean as my son announced loudly to the plane that we’re going to crash. [Laughter] Um, instead I was, I was trying to make a joke about the holy land, so… Right!?! Right. So these clues, these clues are not easy or, or helpful clues. They’re really just obnoxious clues that keep the kids paying attention. So, but the point is, is that as we go on vacation, like a whole vacation experience is one of guessing and trying to figure it out and, and having fun with it and learning to enjoy the uncertainty of it. Um, because at the end of the day, I think that’s the part that we as parents and also I think, uh, you know, we as young adults who were failing in different ways and, uh, our kids who are young adults and failing in particular ways, I think that’s the challenge is “Wow do we face and, and address uncertainty?” How do we, you know, “I thought it was going to go like this and it’s not, and now what am I going to do about it? Um, you know, I don’t know what to do. This wasn’t part of my, my game plan. So now what?” And sometimes the answer, the right answer is rollback come home, regroup. You know, sometimes that is the right answer. Sometimes the answer is, you know, sidestep. Okay, so lawyer isn’t going to work, but you’re not going to not work. You’re not going to not do something. So, alright, how do I take a side step into another career? I think that that’s what it comes, uh, comes down to, is facing that uncertainty and having strategies for when those uncertain moments crop up.

Josh: 20:32 Thanks for making time for us this week to hear more of Technically Religious, visit our website, TechnicallyReligious.com, where you can find our other episodes, leave us ideas for future discussions and connect with us on social media

Leon: 20:46 Test in Dev? Not me! I test in prod. What could possibly go wrong?

Josh: 20:51 Narrator: Apparently, a lot. Nobody was surprised.

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: