Talking Postgres with Claire Giordano

CLAIRE: 00:00:04
Welcome to Talking Postgres, a monthly podcast for developers who love this database. I'm your host, Claire Giordano, and in this pod we explore the human side of Postgres databases and open source, which means why do people who work with Postgres do what they do and how did they get there? I want to say thank you to the team at Microsoft for sponsoring today's conversation. And today's guest is Gwen Shapira. Gwen is the co-founder and CPO at Nile, which was founded about four years ago. And Nile is best known for making Postgres multi-tenant. Prior to Nile, Gwen worked at other data companies like Confluent and Cloudera. She has a Bachelor of Science in Computer Science and Statistics, lives in Silicon Valley, Northern California, not too far from where I live, although we've never met in person around here. And Gwen has worked in and around databases for over 20 years now, although not always with Postgres, which is something we're going to talk about. Gwen is the author of two technical books about data, and she's also active in the Postgres open source community. Welcome, Gwen.

GWEN: 00:01:15
It's so exciting to be here Claire I've been listening to your podcast for a couple of years now. It is the best way to learn about the community.

CLAIRE: 00:01:27
I am so flattered to hear you say that. I know that a lot of people who work with Postgres, I want to say a lot, I guess I'm thinking of Melanie Plageman in particular, she will often listen to recordings of Postgres conference talks while she's out for a run as another way of learning through osmosis. But I do think podcasts are nice because you're not missing out on any of the video because there is no video and you can just listen and do whatever it is you're doing. So I'm so glad you've been enjoying it. Today's topic is how I went from Oracle to Postgres with a big NoSQL detour, where I, in this case, is not me, but it's you. And I thought we could just start with your origin story as a developer, because that's where we always start.

GWEN: 00:02:16
Yeah so the beginning was kind of normal I think I was doing a degree in computer science over in Tel Aviv. I needed to make some money to pay for the degree. So I got a part-time job as a developer. And the first year, I was basically programming first Visual Basic and in Java. But the guy I really admired on the team was the Oracle guy. He knew everything. By the way, just in case Hanan Hit is listening, you had a lot of influence on my life. So he was so good. And he knew everything. That was the thing. Developers knew their programming language. They knew algorithms. This guy knew where to place blocks on the spinning disk so they would be on the fastest sectors. And the correct adjustments of the buffer sizes on the network. And what is the latency if the nurse bridge between the CPU and the memory is under congestion. You can give him, you know, I'm seeing extra 85 milliseconds of latency and he will tell you, oh, your bottleneck is probably over there. Go take a look. So I had just tons and tons of admiration to this guy who just seems to know absolutely everything. And being young and impressionable, I wanted to be just like him when I grew up. So I started taking database classes. First, it was just like courses that a local company gave. And then, you know, it was reading book after book. And there were some blogs at the time. So I also started following database bloggers. And that was kind of also my introduction to social media, what social media was back in maybe 2005. And then when I finished my degree and started working full time, I got the database role. That was kind of how it all started.

CLAIRE: 00:04:30
I read somewhere, though, that when you finished your degree, you were offered a job as an engineering manager for the web team. Am I misremembering that? But you wanted to become a DBA instead because you would learn more about databases. Do I recall that right?

GWEN: 00:04:45
Oh yes it was so yes I think I was actually leading the team for about three months or so but yeah it was not for me and I was also kind of managing people who were many years older than me. Yeah, it was it was a good experience in the sense that my own manager was really good and was super supportive and I learned a lot from him just in how to be a good manager but I also learned that if you have basically zero experience, you're not going to be a good manager. You have to have some level of credentials. You have to know how to earn the respect of a team. So that was first a not great experience as a manager. And I backtracked within weeks, if I recall correctly. The next time I tried being a manager, I was more experienced and it worked a lot better.

CLAIRE: 00:05:52
Okay, so when you got started with databases then after you got your degree, which database were you working on initially? I know it was not Postgres.

GWEN: 00:06:02
Yeah I know it was not. Postgres was extremely young at that time, we're talking about year 2004-ish so there was not a ton of Postgres community and there's honestly MySQL was kind of the LAMP stack, but everyone knew that if you're building a serious product, you have to go with Microsoft, IBM, or Oracle. And we started with Microsoft SQL Server. And again, at the time, it was still fairly newish and caused us a bunch of headaches. But honestly, the reason we moved to Oracle from that, also in fairly short order, was that the DBA really admired was an Oracle expert. And always a good bet to use the technologies that you already have people who are experts on.

CLAIRE: 00:06:56
Yeah, there's a classic phrase that I've always loved, which is the best tool is the familiar tool or something like that. People pick what they know.

GWEN: 00:07:05
Yeah, pretty much. They are in their comfort zone. They solve issues faster, so everyone around them feel a lot better. So yeah, that's basically how I landed on Oracle. I think it was Oracle 9 at the time. And I stuck around all the way to Oracle 12. And then that was enough Oracle for my entire career.

CLAIRE: 00:07:30
Got it. Okay, so Oracle 9 through Oracle 12, I don't speak Oracle release numbers, so I have no idea how much, how long that time period was.

GWEN: 00:07:40
Oh, wow. Okay, let's see. I think 2004, I think, was Oracle 9. And then all the way to 2012, I believe. I think Oracle 12 was in 2012. But, yeah.

CLAIRE: 00:07:57
So almost a decade, eight years-ish.

GWEN: 00:08:00
Yes. By the way, Oracle is kind of famous for gaming the version system. So the first version of Oracle was either Oracle 2 or Oracle 3. And it was intentionally not version one because nobody would use version one for the production database, right?

CLAIRE: 00:08:20
You know, when I graduated from college, I actually, you know, as one often does is you interview with lots of companies, you don't know if anyone's going to make you a job offer, you kind of wait and see what's going to happen. And I did get a job offer from Oracle. But I didn't go there. I went to Sun Microsystems instead. But I think I would probably know even more about databases had I gone to Oracle back in the day.

GWEN: 00:08:44
If they did kind of try to get you back by acquiring Sun Microsystems a few years later.

CLAIRE: 00:08:50
Well, more than a few years later. But yes, I had left Sun at that point a few years before the acquisition. So it didn't happen. Never got an Oracle paycheck. Okay, so you worked with Oracle. And when you were working with Oracle for that eight-ish years, what kind of role did you have? What kind of job? Were you a user? Were you a developer contributor? Tell me.

GWEN: 00:09:16
Yeah, so basically I was, DevOps was not really a thing, but it was kind of a DevOps role. So we basically owned the production, large set. I think we were like six people who owned hundreds of Oracle databases across the company. And then it was really automating, managing it and answering complaints about why is my database slow? Why did my report not show up on time? Why are the numbers all wrong? You know, the usual.

CLAIRE: 00:09:54
Okay, so you built some diplomacy skills then answering all these complaints. Is that in addition to your technical skills?

GWEN: 00:10:00
You know, diplomacy has always been very challenging for me. I think it's a bit of a culture thing. I tend to be direct and also known to drop F-bombs where, you know, the storage does not behave appropriately and gives me 50 millisecond latency for trying to read something. So I don't know if that was the point where I developed the diplomacy, but when I left, at that point it was HP, when I left HP and started doing consulting, I very quickly had to learn diplomacy because the one rule of consulting is that it's always the consultant's fault. Nobody who was ever hired by the company could break the database. If anything breaks, it has to be the consultant.

CLAIRE: 00:10:54
I thought it was always the intern's fault. I missed that. Okay, I'm going to change it to the consultant's fault.

GWEN: 00:11:02
So you kind of have to learn how to navigate, you know, document every decision that was taken and make sure that it was all approved by the appropriate channels. Figure out who is the person actually making the decision and who are people with a lot of opinions. Figure out how to work with the people making the decision to influence the decisions. While you are a consultant, you have no formal power in this organization at all. All you have is hopefully good advice. If you do a good job in testing and benchmarking, then you also have some numbers on your side, but that's about it. And you still have to try to make your customers successful.

CLAIRE: 00:11:48
Okay, so I want to dive in more to those years between Oracle and your work now at Postgres. But first, let's fast forward to you work with Postgres now. You're at a company called Nile. Tell me what you do, what your job is.

GWEN: 00:12:03
Yes.

CLAIRE: 00:12:05
What is Nile?

GWEN: 00:12:06
Yeah so we founded Nile more or less four years ago it was all people who left Confluent in the beginning. And originally it was not supposed to be a Postgres company. It was supposed to be kind of a SaaS for SaaS company, building a platform. We had some Kubernetes inspired ideas and we had a lot of customers' conversations. It didn't really end up with a product that neither we nor anyone else really love. But from all those conversations, we discovered that everyone we talked to was using Postgres. And we're like, if we want to really solve problems for all those companies that we're talking to, let's look at what Postgres is helping them with and what it isn't. Why are they even using Postgres? What can they do with Postgres? And very quickly, I found out that they can do with Postgres literally everything. It was amazing. And it had all those extensions. And it was reliable for them. And they built queues on Postgres and vectors on Postgres. And everyone loved it. And it was so, I don't want to disparage other databases, but it's very rare that developers actually love their database that much. So I was kind of like, okay, we need to get some of this love, essentially. And when we looked at the things that were difficult for them, it was always things. It's like, you know, we have hundreds or thousands or hundreds of thousands of customers, and we either put them all on the same Postgres with this customer ID column, and then later we try to do some sharding, or we start putting everyone on one database, and then we try to figure out how to scale it. But there was no really good tools for managing this architecture that literally everyone has. Everyone has a lot of, every SaaS company, every B2B, it has so many customers on Postgres. And very, there's great Postgres tools. We love the ecosystem. That's what drew us into it. But there weren't tools for these things that everyone is doing. So we're like, okay, we know what we have to build.

CLAIRE: 00:14:29
When you mentioned that Postgres can do literally everything, the word everything stuck in my mind. And I was trying to think of where did I hear that recently? Where did I hear that recently? Coming in June at this year's POSETTE, it's the fifth year of the virtual conference my team organizes, there is a talk titled The Rise of Postgres as the Everything Database, which just reminds me of what you're saying.

GWEN: 00:14:55
Exactly that. There is also a book, I think, Just Use Postgres, which basically just goes over all the everything you can do in Postgres in case you have a Postgres and you feel like you may not be making the most out of it.

CLAIRE: 00:14:55
And I actually have a copy of that Just Use Postgres book down the hall. Dennis Magda sent me a version of it, and I think he even autographed it, which is kind of cool.

GWEN: 00:15:19
Dennis is really, really cool.

CLAIRE: 00:15:21
I know. I know. And let's see, who's the speaker on this, the rise of Postgres? Varun Dhawan is giving that talk. So yeah, it is popular. Let's face it. And you built your company around it, which is kind of cool. How long was that journey from starting the company and having these Kubernetes-inspired ideas to kind of drilling down on multi-tenancy and Postgres?

GWEN: 00:15:47
So I think it took us a year from the point where we started the company until we decided that, yes, we want to build multi-tenant Postgres. And then it turned out that building multi-tenant Postgres is actually the hard part. And we had a lot of iterations of different ways of doing it. And I think we are now landing on the architecture we really love. But again, Postgres is really large and it has huge surface area. And I think everyone wants to be as compatible as possible. So it was definitely a large journey to understand how to hook into Postgres correctly. You know, the things that everyone who tries to contribute to Postgres knows. You try to change one thing, you break five other things somewhere else. It's very interdependent, very interconnected. If you try to make a change, let's say somewhere around visibility maps, maybe, or you try to hook into the way the query planner work, you risk having bad plans. You risk maybe seeing data that should have been isolated. So you need to be very careful about how you're doing things. And obviously working with amazing Postgres experts on our team was the key to that, because I showed up knowing a lot about databases in general, but not about Postgres and not even that much about internals. The only open source database, relational database that I worked with was MySQL. And I didn't do that much internals for it, a bit on the storage layer, but that would be about it.

CLAIRE: 00:17:40
Okay, because your years at Oracle, you were really focused on the user side of things, right? You were responsible for all of those servers, which I actually think is great experience [yeah exactly] for someone who wants to be a chief product officer, right? It helps to have stood in some of your customer's shoes. Obviously, all customers are unique and different, and so you can't understand them all. But yeah.

GWEN: 00:18:05
Yeah and it's also the inverse I mean if you love technology but also really love talking to people helping them use the technology best. Today, there is a bunch of roles. I mean, you can be in developer relations, you can be in developer marketing, you can be in the kind of professional services roles, you can be a product manager. Back then, it was basically professional services or product. DevRel is a more recent phenomenon. So, yeah, that's kind of, I think my inclination was always toward working closely with the people who use the technology. And that's kind of how I landed in all the roles that I landed in.

CLAIRE: 00:18:54
Okay, but I'm still really curious about how you spun up on the Postgres technology. You said a minute ago that it helped that you were surrounded by these deep Postgres experts, which I totally get. But what else did you do? Everybody learns differently. So I know that the way you learn won't work for everyone, but I'm still curious.

GWEN: 00:19:15
Yeah definitely so actually I was very lucky that I was not the only person who moved from Oracle to Postgres a lot of my friends did it and some of them did it earlier than me some of them did it more or less at the same time so the first thing I did was reach out to my friend Jeremy Schneider who used to be my teammate.

CLAIRE: 00:19:41
Who I am trying to get on this podcast. He had me up at the Seattle Postgres Users Group that he organizes, co-organizes a couple of months ago. And anyway, he gave me a soft yes. In other words, a possible future yes without a date commitment. So I'm working on him. And if you're listening to this, Jeremy, just know that I haven't forgotten.

GWEN: 00:20:03
Yeah Jeremy we definitely need you on this podcast it's it's vital so yeah and he so he helped me at all so you probably know he has his happiness chart which is like a small gif with all his tips on how to not I wouldn't say how to succeed with but definitely how not to shoot yourself in the foot so I made everyone print it out and hang it next to their desk and look at it several times daily.

CLAIRE: 00:20:32
Is this published? Can we find it online somewhere?

GWEN: 00:20:35
Yeah if you search for Postgres happiness, happiness hints I think it's called, [I'll put it in the show notes for this show] then yeah this is so important for everyone who is. Getting started you it's again it's not even close to everything you need to know it's just that it's some hints and those hints are very and it's very short so you can memorize all of it and just avoid stupid mistakes. It will make you look better in life in general, I think. So he gave me that. He gave me two books with internals to read. And then he also suggested basically a bunch of blog posts. I think the Postgres FM podcast showed up in there. So yeah, he basically sent me a list of read all of this and obviously the docs are pretty good so he was like yeah you should read the docs but you know most people just say they read the docs you have to really read the docs and then he I think he actually also recommended your podcast as more like this is not where you're going to learn the technical skills but the community is important so I think that's also where I got your podcast from.

CLAIRE: 00:21:57
Well, thank you, Jeremy.

GWEN: 00:22:00
So anyway, he dumped all those resources on me. And then, you know, you find the mailing lists, which also I haven't asked questions, but just reading them can be helpful. Searching, it's years and years of mailing list. You learn a lot just by searching the history. So I think that's more or less where we got started. And then, oh, the most important bit, Postgres is open source. I could read the code. And it's actually so readable and so nicely written. And so many helpful comments and the various readme's and the readme on the transactions part. It actually explains how transactions work. I mean, it's amazing as one would expect from a readme. So it sounds like faint praise, but honestly, a lot of open source code is a mess. And the fact that this community manages to keep readable, well-contained, well-structured code is great documentation for freaking 30 years with so many people. There's the core contributors, but then there's just a lot of drive-bys who show up and make one contribution and then kind of go away. And the code did not devolve into a large mess of spaghetti. It's amazing. I was so impressed. *laughs*

CLAIRE: 00:23:30
You mentioned 30 years so I just want to like segue for anyone who's listening and doesn't realize it this year is the 30th anniversary of the founding of the Postgres open source project now you may have heard that the technology itself is going to be 40 years old and that's also true and it's not like a off by one error or a typo or something so 40 years for the tech 30 years for the open source project itself. And then I know that you're involved in organizing this year, the annual PGConf.dev, which is the annual Postgres Development Conference. It always happens in Canada. This is, I think, the third year since the conference kind of rebranded and changed its approach and vibe and all of that. It's happening in Vancouver in May. And I know I'm going to make sure to drop a link in the show notes to the conference t-shirt this year, which you might be thinking, wait a minute, we're talking about technology. Why does a t-shirt matter? But it's really cool because they made it celebratory in honor of the 30th birthday.

GWEN: 00:24:36
The t-shirt and the stickers are the most important part of every conference and I'm going to die on this hill. I mean, you can listen to talks on YouTube and you learn a lot from them but the vibe of the conference the being there is something different and then going back home and you have a t-shirt and every time you sit in your closet or wear it, you remember all those interactions and the great vibe. And, you know, some days my Postgres conference t-shirts motivates me for showing up and kind of doing a better job for the community.

CLAIRE: 00:25:16
Well, we're not doing video right now, but I do have on my PGConf New York City sweatshirt and underneath my PGConf.EU t-shirt from 2023 from, I think, Prague. So, yeah, I'm with you. We, I will say also that the hallway track at PGConf.dev, last year in Montreal in particular, was just amazing. I always thought you can't miss the sessions. You got to go to the talks. And I remember just sitting in the hallway talking to people for a couple hours straight.

GWEN: 00:25:46
Yes, I definitely also missed a lot of sessions that I had to catch up with [With the video] because just the people who show up at those conferences are so amazing. You grab someone in the hallway and you talk to him about, I don't know, foreign data wrappers. And it turns out that this was the guy who created foreign data wrappers and he knows everything. And then you turn around and you talk to someone about scaling Postgres. And it's like, oh yeah, I work for ServiceNow and I have 200,000 Postgreses. It's just so amazing. I also wanted to mention a lot of my contribution to the community. It really started, so one of the Postgres experts that I learned a lot from is Julien, I think, Rouhaud would be his last name.

CLAIRE: 00:26:37
Who works at Nile, right?

GWEN: 00:26:39
Yeah, the best way to learn from people is to hire them.

CLAIRE: 00:26:43
He's also a Postgres open source contributor.

GWEN: 00:26:45
Yeah, he has HypoPG, which is amazing.

CLAIRE: 00:26:49
And I know he listens to this podcast.

GWEN: 00:26:51
Seriously?

CLAIRE: 00:26:52
He has sent me his he's one of the people, you know, a lot of people listen to the show and don't. They don't think to reach out to the creator or the host or the team behind it, because why would they like? But anyway, Julien did. He sent me a note one day. It says he listens to it while he's in the car driving. And that was kind of cool. So you were saying Julien.

GWEN: 00:27:13
Yes, so one day he told me, Gwen, you know, we're getting a lot out of the community. I think we should think about giving back. And I'm like, well, we're a startup. I mean, it's not like we can go and be a big sponsor and all the cool things Microsoft may be doing. But one thing I can do is just go and help out. So I grabbed Melanie at the conference and said, look, you're doing an amazing job. It's the best conference I've ever been at. How can I help? And to her credit, a year later, she's like, hey, can you help us with getting logos from some sponsors? It's always been a bit of, you know, not hard work, but you just have to send a lot of emails and be on top of it. And I'm like, sure, I can send emails and be on top of stuff. That's usually my thing. And then, you know, something that I learned, I think I heard it from Stacey. She said, if you do a good job at something, people will ask you for more. So this year I'm organizing a bit more.

CLAIRE: 00:28:17
Yeah, and I know the team really appreciates it. And, I helped a little bit with social last year and this year I'm on the Tuesday planning committee, that Robert Haas is leading. So we just look at the submissions and speaking of which there's a CFP closed deadline coming up for the Tuesday and submission. So I'll make sure to include that in the show notes as well. But that organizing team is really knocking it out of the park. And a lot of the hackers and contributors will show up. And there's a lot of planning that happens for the next release. A lot of negotiation, a lot of discussion, trying to figure out if people are going to get the support they need to pull things off. So, okay, so we talked a bit about how you spun up on the technology of Postgres. And we'll share some of those tips in the show notes. But I skipped over all your years on NoSQL. And I guess what I'm curious about, we can go backwards and walk through them. But also, how has working on Oracle and working on various NoSQL databases influenced your work on Postgres? I'm just curious whether you approach problems differently than maybe someone who's worked on Postgres their entire career. How does that NoSQL experience inform how you think about things?

GWEN: 00:29:43
I'm appreciative to Postgres and relational databases in particular, in general, just how much stuff they do for us. And what I mean is that it's, you know, when you are on NoSQL, there is no transaction isolation. You have to worry about what exactly are the guarantees and where they're going to fail. You have to be aware of a lot more when you're coding. And a lot of times the way I felt about it, and I know a lot of my teammates felt that way too, it's more like you have building blocks of a database and you kind of have to build your own. So let's say you have Kafka. Nice way for a database person to think about Kafka is a write-ahead log. You basically have one change after another, after another. And then you put maybe some kind of a key value store in front of it to kind of cache the latest value. So you have faster querying of those things. And then there are different ways to put SQL on top. So there is open source projects for that as well. So you kind of pick and choose different building blocks and you kind of end up building your own database in the process. And if you are a large company or you are very skilled distributed systems teams, there's a lot of benefits for this kind of disaggregation. And it does give you a lot of understanding on appreciation of the low level mechanisms. How does one build a transaction from building blocks? How does one handle consistency with a write-ahead log recovery, catching up when things fall behind? So caching, so you kind of get very good understanding of a lot of primitives that later served me very well as you go and work inside the database. And you also get a lot of appreciation for, you know, the communities that make it so that when you don't want to think about all those primitives, when you don't want to build your own database from scratch, when you just want to install something and have it just work with the expected guarantees, these things exist. And it's kind of nice that you have all those different options on how you want to run your production operations.

CLAIRE: 00:32:19
Yeah, I think when we were exploring different titles for this episode, one of the ones you proposed was how Postgres helped you appreciate Codd. So I don't know if you want to explain to people who might not be familiar what Codd, Codd, stands for and how that connects.

GWEN: 00:32:39
Yeah, so, Codd and I would say Jim Gray, those are the database pioneers, right? A lot of what defines a relational database, relational algebra. Things like foreign keys, the idea that you split things into tables so you don't have duplicate data. If you change things in one place, it's enough. You can efficiently query it. ACID is a very important primitive. All those things are basically what makes life easy for us as a developer. It tells us, hey, you don't have to do all this work. You don't have to figure out how to not duplicate data and how to get consistently correct information whenever you're querying and how readers don't block writers or they only block writers when they actually need to block writers. All those things are basically things that they figured out for us. So I don't have to figure it out again and again. And this has actually been kind of a major thing in my career. I am so pissed whenever I have to solve the same problem more than twice. It just, I feel it's morally offending. My family has, you know, German roots and wasting someone's time is like the height of disrespect was the way I was brought up. And I have no patience for databases who waste my time. And so, you know, I did a lot of consulting and you talk, you solve the problem and then you consult someone else and they have the exact same problem and you have to solve it again. And this is so annoying. So at some point you're like, we have to fix the database so nobody will ever run into a problem or that they will have better tools on how to resolve it themselves. So I don't have to go and solve it again and again. So yeah, a lot of the things I did was basically I ran into the same problem three times. Now I have to go work for a company or start a company that will solve this problem once and for all.

CLAIRE: 00:34:55
You're reminding me and this isn't the first time I've remembered this out loud but I was talking to Simon Willison once and I was thanking him for his altruism with his Today I Learned blogs and all of his blogging where he shares his kind of, problems and solutions and things. And he's, Oh no, I don't do that for other people. I do that for me. I do that for future Simon. Because he doesn't like to have to solve the same problem twice either. So when, when he does figure it out, his blog is a way of documenting it and capturing it.

GWEN: 00:35:29
And also finding it again in the future. One of the most annoying things is that I solved the problem and I don't remember how I solved it.

CLAIRE: 00:35:36
Yeah, yeah, yeah. So, which actually brings us to the topic of blogging. So as I was researching, getting ready for today's episode, I found a quote from you online that said, blogging connected me to like-minded people. And through this network, I found work that I loved. You could say that blogging created my whole career. So, I, tell me more about that? Where did that quote come from? How did it happen?

GWEN: 00:36:06
Yeah, so basically I went early in my career as a young database person working in HP at the time. I was by far the most junior. Let's say I had two, three years of experience. Everyone else around me had like 20. And obviously as the most junior person on the team, nobody ever listens to you. You think you have a clever solution for something, but everyone's like, no, no, let's do everything the old way. And so I, and you know, I already told you, I started my learning databases by reading people's blogs. So I was like, I know I'm going to go and write down all my ideas and maybe other people will read them and listen to me, even if nobody at work listens to me. And that's pretty much how I started writing and documenting. And very similar to what Simon said, you solve a problem and you write it, and the next time you run into the same problem, you Google it up and you find your own blog with explanation on how to solve the problem. And sometimes people show up and tell you, no, when you're solving it the right way, the wrong way, or this is a better way to solve it. Or, you know, I mean, adding an index is nice, but as you said, it's going to lock up your table. So you kind of learn from people, you learn a lot from the interaction. And back then, the social media, which was basically the blogosphere, it was such a friendly community. And everyone was helping. Everyone was answering questions. A bit maybe like the Postgres mailing list. Everyone is not there to score points. You're not going to market your way. You're not going to become a famous influencer and make a lot of sponsorship deals from that. People are there to technically help each other. And I appreciated it so much. And basically it was because I met all those people who also had blogs and they started talking about this conference. So I also went to this conference. And then you meet the people in person. And you can tell them, hey, I started this blog because nobody listens to me at work. And they can say, well, we're hiring. Do you want to work with us? And if they're really cool people, then I want to work with them. So I try to interview and maybe find my next job. Because everyone wants to work with people they really admire. And now I show up at those conferences. I want to hire people. And I hope to be a person that they admire enough that they will want to come and work at night.

CLAIRE: 00:38:54
Got it. I take it that's the requisite plug for the fact that Nile is hiring and people should check out your website. Right. OK, so the other thing I discovered is that there was a blog you wrote, I don't know, sometime within the last couple of years, and you called it your most popular blog ever. And it was about transaction isolation in Postgres. And I just want to know, why is that your most popular blog?

GWEN: 00:39:23
I don't know. Ask the people who all went and read it. I have absolutely no idea. I mean, I knew that transaction isolation is a tricky topic. One of the reasons I wrote the blog is that I always get confused. There's so many different scenarios, different interleavings, the vacuum, MVCC. There is just so many moving parts. So I, yeah, I had to write it down just to make it clear in my own head. And apparently I wrote it down in a way that maybe other people also appreciated and made it clear in their head. And maybe they shared it with some of their friends and it became popular. But it was definitely, I didn't expect it to be that popular, to be honest. Especially since I'm not the first person to explain how transaction isolation works. I mean, because it's confusing, it is quite a popular topic in general. You see, you know, I'm going to be in PGConf and I'm sure I think there is like three different talks about vacuum and how it's going to evolve in the future and how to use it now and all those things, because it's something that everyone really, really cares about.

CLAIRE: 00:40:42
So you just said you're going to be at PGConf.dev, is that right?

GWEN: 00:40:46
You cannot really be on the organizing committee and not be there, right?

CLAIRE: 00:40:50
Okay, yeah, I guess that makes sense. Okay, cool. So because I do know you work for a startup, so it's not like you're going to all the Postgres conferences that happen around the planet?

GWEN: 00:41:02
No I wish I could.

CLAIRE: 00:41:02
Got it. Well, then I will look forward to seeing you in person in Vancouver. Okay, so I'm still feeling badly about the fact that we skipped over your 10 years of non-relational data systems between Oracle and Postgres. Is there more to talk about in that space in terms of maybe lessons you learned or failures that have stayed with you and informed how you do your work today? And were you doing product work at that time or were you still doing either consulting or managing servers as a DBA?

GWEN: 00:41:48
That's funny, I did everything. It was about 10 years. It included product, it included services. It included engineering. It included managing an engineering team. So it definitely included bits of everything, which prepares you very well to the job of a founder, which literally means doing bits of everything. And I think if I had to just summarize in short, the biggest thing I learned would be to use the right tool for the job. And the right tool to a very massive degree would be the tools that the people you work with understand really well. It's not the only consideration, but it has to be pretty high up there. Among other reasons is that if you don't understand something very well, then you are likely prey to marketing and salespeople. They will show up and tell you things like, our database has zero latency and you can do active, active replication with no conflict ever. And it's all immediate. And if you don't know why this cannot possibly be true, then you are going to make really bad choices.

CLAIRE: 00:43:08
Got it and in that 10 years where were you working which was it?

GWEN: 00:43:16
Yeah so it was part of it was my work at oh now you're making me say the companies that we're both struggling with. It was Pythian for I think about a year or two and then Cloudera for two years and then it was Confluent for seven years.

CLAIRE: 00:43:37
Okay, got it. You just had such a varied career in terms of your technologies, but it's all in databases. So, I guess that one person, we have to make sure to get his name spelled properly, so I can add it in the show notes. Who was the person who knew everything about how the data blocks were stored on disk while you were at university. I want to make sure to include it, but yeah, he did influence you and your career path.

GWEN: 00:44:06
Very very much so yes.

CLAIRE: 00:44:10
Now the other thing I found online is that you sometimes, I don't know if you did this a lot, but I found a riddle that you posted.

GWEN: 00:44:20
I saw you liked it last night and I was like, this was, how did you even find it?

CLAIRE: 00:44:27
I was just researching and you know how it is when you're researching and searching. You just run into all sorts of stuff. So, the riddle I ran across and maybe you post a lot of riddles about Postgres. I'm not sure, but you posted this question of what happens when you add and remove a column from a table in Postgres 2000 times. And a lot of people like responded and resonated and were curious about that fact.

GWEN: 00:44:55
Yeah, it's kind of silly, right? I mean, you don't think that a lot of people need to drop and add a column 1,600 times. Yeah, for me, a lot of what I do with databases is driven by my curiosity. I just, you know, questions pop into my head and I want to know the answer. So I really like doing a quick test and just try and find out. And then if the test is unexpected, it turns into a riddle. And I go and do a deep dive to find out. Because if I do a test and I get a result I didn't expect, it means I did not understand the system correctly, right? This is the rule of benchmarks. If you do a benchmark and you get the results that you expect, you're doing marketing. If you get something unexpected, you learn something. This is the key. You have to figure out something. You're doing tests to understand your system better. So in this scenario, I learned quite a lot about how Postgres actually deletes columns. And why is it so fast? Because, spoiler alert, it doesn't delete the columns. They keep hanging around there. They're just kind of hidden. And yeah, that was definitely time well spent. I learned something, I improved my skills, and I found out that a lot of people like learning things and improving their skills. It's something that I feel like more people should be doing is just to do a small experiment, especially when there is an argument. This is something I learned while consulting. I cannot argue with a customer. If I am in the argument, I already lost. This is the second most important rule of consulting. If you're arguing with your customer, you already lost. And it doesn't matter what is the outcome. But if I can join them, I can do an experiment with them, and then we will find out if it's that way or the other way, then it's a win-win. I learned something. He learned something. We're now collaborating rather than arguing. It's fantastic. So yeah, I feel like whenever a question pops into your head, you should do an experiment. If it's unexpected, you have to learn what you didn't know that caused an unexpected result. And then you have something you can publish in your blog or on social media, or you have something to share with the world. If it's a big thing you discovered, you submit a talk to your favorite Postgres conference, and then everything is even better.

CLAIRE: 00:47:33
So, in terms of sharing, you're touching on social media, and I know prior to Microsoft acquiring Citus, I credit Twitter with contributing to my landing the job at Citus Data. Just because I was connected to people at RedMonk, Steve O'Grady, and then he connected me to Craig Kerstiens, who then I was following on Twitter. And, you know, Twitter was part of all of these relationships. And since the world is fragmented in social media, I'm actually not sure where to look. I have accounts on Mastodon and Bluesky and Twitter and LinkedIn. And I'm just curious, as you share these things, these riddles, these blogs, where are you finding the most engagement with people in the Postgres world?

GWEN: 00:48:27
Honestly, the last year has been a bit depressing. I'm posting like you, on Twitter, on LinkedIn, some on Bluesky. But I feel like there was something about the blogosphere in early 2010s. And then later, the ways that a lot of technology, Twitter kind of, or X kind of became the technology hub. And it's not quite there yet. Between seeing more of the algorithms and you see from your friends and a lot of really good people just kind of getting disgusted and leaving it's just yeah I haven't found my new place and you know I keep joining discords and I hope that you know by joining enough discords I will find my people I will find the place where I can share and learn and grow with people but at this point nothing really sticks for me. I don't know. What about you? Do you find anything that I'm missing?

CLAIRE: 00:49:29
I mean, I think I'm getting the most engagement on LinkedIn these days, but Tristan Partin just mentioned on the Discord, the live chat for this recording, that there are a lot of Postgres people on Mastodon. And I do think that's true. And I think Mastodon though, it's very temporal. And so you have to be on there enough to like catch people at their times of day when they're paying attention. So the way that I'm using Mastodon, which is kind of once in a blue moon, that's not really working. So I think to connect to some Postgres people, I probably have to be more are present. But LinkedIn is the best I've got right now to reach people. It's just so different than the way it was a couple of years ago. So we're all adjusting, I suppose. There is a Postgres hackers Discord now, which I assume you have joined, right?

GWEN: 00:50:26
Yes, yeah. It is a really nice place and yeah, it has really nice people. It's very focused though on people doing work inside Postgres. So it never seemed like the right place for stuff like what happens when you add and remove a column for Postgres because everyone there already knows exactly what happens some of them even wrote the things that makes it happen.

CLAIRE: 00:50:51
Yeah, but I do think that if you are an aspiring Postgres hacker, someone who wants to contribute to the open source project, but is maybe new to it, the monthly hacking workshops that are effectively hosted on the new Postgres hackers Discord. And when I say new, we're coming up on two years. So it's not that new anymore. But I think it's a great place to learn.

GWEN: 00:51:10
Yes totally.

CLAIRE: 00:51:15
It complements the mailing list.

GWEN: 00:51:17
And you can ask questions and you get more real-time answers. I also want to point out Andrey Borodin had a series on hacking Postgres. He just shows how he does different things with his browser and he puts it on YouTube. And I found it a good place to learn internals as well.

CLAIRE: 00:51:38
I know that he did a series of hacker trainings at a previous PGConf.dev as well. Do you know if he's doing it again this year?

GWEN: 00:51:48
I remember something about the workshop, but I don't know if it's that. I know one of the sessions I'm really looking forward to is the real-time patch feedback session. Robert has got together some of the top committers to basically give people feedback on their suggestions, patches, etc. It looks like it can be a very exciting session.

CLAIRE: 00:52:17
Yeah, that is happening on Tuesday. And what's cool, this year, Tuesday is very, very different. So in past years, Tuesday was full of half day workshops and people, a lot of people felt like, well, I don't need to show up at PGConf.dev until the Wednesday. Wednesday and Thursday are full conference days. Friday is the un-conference, right? Which is where the schedule gets created in the morning of, you know, by virtue of voting and everything. But yeah, the real time patch idea evaluation is happening on Tuesday. And shout out to, let's see, both me and Greg Burd were facilitating another panel on Tuesday called, which the idea I think was Melanie Plageman, but Unexpected Successes and Epic Failures by Postgres Committers. And it's a roundtable with Alvaro Herrera, Daniel Gustafsson, Peter Eisentrout, and Thomas Munro. So successes as well as epic failures.

GWEN: 00:53:15
I would. I would so much love to hear it. Isn't the best story kind of how did things go terribly wrong? There was also the io_uring retrospective last PGConf, which also made for a very good conference talk.

CLAIRE: 00:53:32
Yeah I think when you think about patterns of conference talks the learn from my failure is always one of the most popular patterns and I don't know why it is psychologically. Are we all just, do we get glee out of seeing, hearing about someone else's failure? I don't think that's it. I think it's that there's an appreciation for someone being vulnerable and being able to make themselves vulnerable, share what went wrong. And you can self-identify because stuff goes wrong all the time for all of us.

GWEN: 00:54:03
I think failures also make for better lessons. There's this, it's the age of lesson of like the heroic struggle, right? Every good saga has this heroic struggle. And I think like, as if things, if you just say, okay, I did this, this, this, and it succeeded. I didn't learn that much. If I said I tried this, it didn't work. And then I had to look into that and that didn't work. You just, you have so many more lessons.

CLAIRE: 00:54:33
I'm looking at the Tuesday schedule right now, and there's an onboarding new community members to Postgres. Bruce is giving a talk, what's missing in Postgres, which I think is cool too. Similar to failure. It's like what people like to figure out what's missing, what's wrong, what could be added.

GWEN: 00:54:51
There is even a talk from a MySQL guy on what Postgres should learn. [Ooh, okay.] It's going to be spicy. I think this PGConf is going to be very spicy.

CLAIRE: 00:55:13
Okay. Spicy is good. Okay. So I want to make sure that we keep the promise of the episode title. How I went from Oracle to Postgres with a big NoSQL detour. Is there something I was supposed to ask you about that big NoSQL detour? Or I mean, at the end of the day, I'm just happy you're in Postgres now. I'm happy you're at Nile. I'm happy you're contributing to the community. But for your journey.

GWEN: 00:55:36
I feel like we covered it quite well. We did lessons learned. We did what I appreciate. I think it's...

CLAIRE: 00:55:45
Okay. We didn't talk about 30 days of 30 inks though.

GWEN: 00:55:48
How did we miss that?

CLAIRE: 00:55:49
Okay. What in the world is this? I learned about it about five minutes before we started this episode. I've heard of like writing November or like no, no shave November. Like I've heard of other things, 30 day, habits that people form. What in the world is 30 days of 30 inks?

GWEN: 00:56:08
Okay so you have to first understand that when I am not a technology fanatic. I am very old school and I like stationery and I especially like fountain pens and the thing with fountain pens is that you can fill a fountain pen with any ink you want you it's not like you buy a pen and then it comes with one ink and you're stuck with it and so you can buy a lot of inks in many many colors and with shimmer and glitter and all kinds of stuff and I have a collection of 160 inks. [One six zero.] 160 and in April which has 30 days I decided to enjoy my collection by using a different ink color every single day so I start my day by drinking my espresso and then filling my fountain pen with the ink of the day and then I write with that ink and pen for the rest of the day.

CLAIRE: 00:57:11
That's very cool. You know, I read somewhere once that there's something about pen and paper and processing of information with pen and paper. Is that something you've studied or can teach me more about?

GWEN: 00:57:26
I haven't studied it, but that's absolutely how my brain works. So if I do a conference talk, I will sketch out every single slide on paper before I even open PowerPoint. If I am trying to wrap my head around Postgres architecture, I will draw a lot of boxes and put function names and file names in different boxes so I can kind of know how to navigate. If I need to solve my own architecture and write code. I also, before I even open my LLM, I draw boxes and kind of get my head around how I want it to look. And then I can do a much better job of writing the code, instructing an LLM to write the code, review the code later because I know what is supposed to happen. I have this nice diagram right there. So yeah, I'm very much a pen and paper kind of person.

CLAIRE: 00:58:24
Yeah, when I'm stuck, I actually can think pretty well keyboard in hand. I learned to type at a very young age. And so it's natural for me and I type very fast. But certainly if there's diagrams involved or pictures, I'm not good at using a keyboard to create diagrams. Other people are. I'm not. I don't use Visio. And that's when I have to get pen and paper out. Or if I'm stuck, if I really just can't, I have a writer's block or a creator's block, the only way out is pen and paper.

GWEN: 00:58:57
Yeah it just uses a different part of your brain.

CLAIRE: 00:59:01
I am looking at Jeremy Schneider's Postgres happiness hints right now. Let's see. Checksums, huge pages enabled, updates and upgrades, logging. These are the sections. So he has created six different categories, if you will, of these happiness hints, which I think is a great name for tips and far more memorable, far more likely to be like rediscovered or easy to find on search results. You know what I mean? So, checksums and huge pages, updates and upgrades, logging. Then there's multiple physical data centers is the fourth category. There's an alarms section. This is really cool. I like it. And so you basically went through and made sure you understood each and every one of these.

GWEN: 00:59:52
I started by just following them. And then actually my co-founder pointed, I don't even remember which one, but he pointed at one of them and said, I don't buy it. I don't believe in that limit. So obviously the next thing is, because one thing the happiness hints don't tell you is what will break if you pass any of the limits, right? They just tell you don't, but they don't tell you what happens if you do. And as I mentioned, I'm a very curious person. So yes, next thing to do is to, what happens if you have a table above a certain size? What happens if you try to create an index? What happens if you try to dump it? What happens if you try to make a large update? How long does it actually take? How good is the optimizer? How accurate it is? What happens if I have 50,000 schemas in the same database? It turns out that even if they're all empty, PG dump can take hours. So you kind of run into, okay, now even if I decide not to listen to one of the hints, I know what to expect. It's not like I'm just following blindly. I know what trade-offs I'm making. And that's what engineering is all about. You're making the right trade-offs again and again.

CLAIRE: 01:01:12
I mean, it's interesting, even in the work that I do, and I, you know, spent the first 16 years of my career in engineering, mostly in the kernel group, but now I do community work, right? And, but it's all about trade-offs too. Like every day, there's decisions and choices that we have. [Yes. Yes.] To make and trying to figure out, well, by what criteria are we going to make this trade-off? What criteria matters the most? I actually am a big hater of pros and cons lists because I find them to be very misleading, right? Because you can list the pros and you can list the cons, but you might be missing the most important criteria. [So true.] So I always structure my trade-offs as a matrix where it's like a report card in school and every row is like whatever the criteria is that we're assessing. And I try to look at my choices, my trade-offs by those things. And I draw an empty matrix first, where I just focus on, do I have all the right items in that criteria column?

GWEN: 01:02:10
That's such good advice. I mean, you get the criteria right, and you're much more likely to make the right choice, essentially.

CLAIRE: 01:02:18
And then my next trick is after the decision is made, people are going to criticize it, disagree, dislike it, try to undermine it, whatever, whatever. And so I always share that decision matrix like internally so that people can see why. Oh, oh, I get it. Oh, I hadn't thought about that aspect of it. And it gets rid of some of the churn. Not all, but some. I am so glad that you joined us today. Before we leave though are there more riddles can I go hunting online to find more of these riddles or was that just a one-off that I happened to discover last night?

GWEN: 01:02:58
It's probably not the only one, but I don't do them super often. It's, yeah, it's an inspiration thing.

CLAIRE: 01:03:08
One of the things that you mentioned Postgres FM earlier one of the co-founders of the Postgres FM podcast is Michael Christofides, who is a fabulous human being. And he's been doing Postgres puns lately. In the worst, nerdiest way.

GWEN: 01:03:21
No it's hilarious in the worst nerdiest way possible but it cracks me up.

CLAIRE: 01:03:31
And it's a little embarrassing that I get so much enjoyment out of them. I don't know why. What is it about puns that just pull people in? Even if they make you groan inside. They're still cool.

GWEN: 01:03:40
Do you know where we store all our bad database jokes? [Where?] In the database. [Thank you for that.] Sorry, I think I'll now see myself out.

CLAIRE: 01:03:57
Well, we covered some really interesting ground today. I'm so glad that you've joined us for today's podcast. Thank you so much, Gwen.

GWEN: 01:04:04
Thank you so much for having me. It was a really pleasure.

CLAIRE: 01:04:08
Now for those of you who are listening, if you like today's episode and you want to hear more of these Talking Postgres shows, you should subscribe on Apple or Spotify, YouTube or wherever you get your podcasts. And please tell your friends because word of mouth is gold in the podcast world. People are far more likely to enjoy and discover a show that a friend of theirs has recommended to them. And if you leave a review, even more people will discover the podcast. You can get to all the past episodes and get links to subscribe at TalkingPostgres.com. And transcripts are included on those episode pages on TalkingPostgres.com too. And we do a good job to try to make sure that the transcripts are useful. I know a couple of people who don't listen to these shows, but they do read the transcripts religiously. A big thank you to everybody who joined the live recording and participated in the text chat on Discord.