Talking Postgres with Claire Giordano

How I got started as a developer (& in Postgres)
===

CLAIRE: ​ I want to welcome everybody to Path To Citus Con, the podcast for developers who love Postgres, where we discuss the human side of open source databases, Postgres, and the many brilliant Postgres extensions. This podcast is now available on all of your podcast platforms. You can get to past episodes and get links to the various platforms at aka.ms/PathToCitusCon, all one word. Transcripts are included on the episode pages on Transistor, too. And I want to say thank you to the team at Microsoft for sponsoring these community conversations about Postgres. I'm Claire Giordano, one of your hosts.

PINO: And I'm Pino de Candia. Today's topic is "How I got started as a developer and in Postgres."

I'll start by introducing Andres Freund. He's a Postgres hacker. Member of the Postgres core team. Works at Microsoft, and has been working in the PG community since 2008.

CLAIRE: And I would like to introduce our second guest, Heikki Linnakangas. He is also a Postgres hacker. To be a Postgres hacker means to be a Postgres developer and a committer and contributor.

Heikki is a co-founder of Neon, another database company, and he's been working on Postgres since 2003. Welcome to you both. So the topic for today, of course, is how I got started as a developer and in Postgres, and we wanted to start with the developer side of things and hear from both of you about: how did you become a developer? When did you become a developer? Why? Andres, do you want to go first?

ANDRES: I think I'd had at least three, four attempts at becoming a developer, as probably many guys in that time, I was impressed by games at first and then tried to write a game and then discovered that I actually don't enjoy that at all.

And then stopped and then started again. And at some point, some people asked me to help them write some website with some CMS and I've tried that and I learned a lot of programming and about how to not write a CMS because that didn't work. And later I was doing my civil service military service replacement thing.

And during that I was asked to write software for storing clinical study data. And that's the first one that worked a bit better, I would say. Not particularly well, but it worked. And yeah, that got me started as a developer and soon after also as a Postgres user.

CLAIRE: Now, how old were you when you were trying to write games or working on a website or ultimately doing the project with clinical study data?

ANDRES: I tried to figure that out at some point. I lost all the data from that time, but I think it must have been about 15 when I, the first attempts. And then I think later I started doing like working around computer stuff, like repair, building them for people and teaching seniors to use computers that it must have been like when I was 17.

And then the first time I was paid and then didn't succeed was also around that time. And then I think when I actually did like a clinical study data thing, there was about 20.

PINO: Okay. What happened in between? When you stopped, did you just stop cold or did you keep studying some coding?

And what convinced you to get started again? If you'd sort of had a bad experience, what got you back?

ANDRES: A long time ago. I don't think I really stopped doing stuff with computers. I did like a lot of sysadmin stuff. I ran... I did sysadmin stuff for like a bunch of networks that were somewhere between 30 and a hundred computers or something, and that involved occasionally some scripting and so on.

So that was programming adjacent enough that I think I had like continued exposure. Because of that, I also learned on the job in a way, I didn't like focus on learning things. But now that you actually, I had completely forgotten that, but I also did some attending computer science. There was a program to attend computer science at university while being in school.

And I think I did that. I don't quite remember what years that was. So I guess that also helped me understand some more.

CLAIRE: Now, I first met you back in 2017, I think, and if I remember correctly, you do not have a formal four year university training in computer science, so you learned on the job? You're self taught?

ANDRES: I'm largely self taught. After that civil service year, it wasn't quite a year, but like close to a year, I worked for a few months and then did attend university, but I worked half of the time, attended university, and it turned out to be a bit too much to do both of those. And I think I did the first few semesters of a computer science and math degree. And then decided that didn't quite agree with my personality and my priorities and focused on working full time. It perhaps did not help that the database class talked about Postgres and I had by that time done a few Postgres contributions and the professor was not necessarily always accurate when talking about Postgres and I could unfortunately tell, and that perhaps didn't make me feel I was learning all that much. I'm not sure that was quite accurate, but I think there was, I'm not sure I would be as good as I am now if I hadn't done the theoretical computer science part at university, because that is the thing I think that was the hardest to pick up on a day-to-day basis, just programming, many of the other things were easier to learn and adopt compared to the theoretical computer science part.

PINO: I have to ask if you, sorry to interrupt, but I have to ask if when you were in that class and they were talking about Postgres and you knew the internals better, did you speak up and say, ah, it's not that way. It's like this. Or did you... and how did it go?

ANDRES: How annoying it must have been.

PINO: Oh, you can see it in a different way. I mean, that's participation.

ANDRES: Probably both views could be held.

CLAIRE: Have you ever gone back and talk to that professor again?

ANDRES: No, I didn't think he was an all that good professor, so I don't feel like bad for, you know, annoying the professor, I might have been annoying to my fellow students, could be true, I don't know.

CLAIRE: But it's interesting that even though you didn't complete the degree and you decided to dive full time into being a developer and sounds like into working on Postgres, you still credit some of that theoretical learning from those classes as a positive.

ANDRES: Yeah, I think the programming aspect, I didn't really need at that point to learn that in university because I had done more programming than we did as part of that.

And I had done a bit of database internals work and other larger programs. So I don't know, the manual labor type aspects I had a good handle on it, but like understanding complexity theory and stuff like that. It's just harder, I think, to get a good handle on without like some other theoretical background.

CLAIRE: Okay. All right. Let's switch to Heikki for just a moment. Heikki, can you tell us how you got your start as a developer?

HEIKKI: Well, so I started programming already as a kid. I have a 10 years old brother who got his first computer when I was probably around four or five years old.

And so I was watching him typing in programs to Commodore 64. And I was of course, looking up to him and also wanted to learn. And that was really fascinating. So I started by just following and looking at what he's doing, following his footsteps. And I think by the time I was around 10, I got my own first computer, which was an Amiga, and I started to learn BASIC on that.

And then I started, moved on to C programming. And Andres' story just reminded me that I also wrote a game when I was around 15 with a friend who did the graphics. It was a Ninja game. It's probably still out there somewhere. If you have an Amiga emulator, you can probably find it. So that's how I started the programming, but moving on to developer, I got my first job as a developer when I was 19, I think, at the local software firm and consultancy just a mom and pop, small mom and pop company, literally it was a mom and pop running that.

And they were gracious enough to hire me. And that's where I learned SQL. They put me on a crash course on learning SQL. And I did some Java programming there and learned Delphi and learned about databases, which I didn't have any experience before. So that was my first touch with databases.

They used Firebird or InterBase, it was called back then, and Sybase and some other databases there.

PINO: That's really jumping into the swimming pool.

HEIKKI: Yeah, that was, but that was a great experience. I already knew how to program when I, when I joined that company. But they helped to learn SQL and it was a nice company as a for first job because they also had other people there who were learning to program and learning to be software engineers, so they took a chance on a 19 year old with no degree at that point, of course, and so forth.

So that was very nice.

PINO: Can we go back for a second to your experience with your brother? I was asking myself, was that your first programming buddy, your pair programming buddy?

HEIKKI: No, I don't think I ever programmed with him. I just always watched him and he explained how things work.

And I remember him staring at the disassembler working on some demo code or something. But I don't think I ever did programming with him.

PINO: Okay. Well, that has all the elements of nostalgia, you know, programming a Commodore 64 and your older brother. I love it. It could be a part of a, of a movie.

HEIKKI: I think that was a pretty common experience. Like not, I mean, it was more common at that age or era that the people actually had their first Commodore 64. And, I know my brother had several friends who also learned to program which I don't think is that common nowadays, or maybe people just have different paths with web development nowadays or something.

But that was more common back then, at least here.

CLAIRE: Do you mind if I ask, did your brother ultimately become a developer too or did he go off into a completely different industry?

HEIKKI: He did. He did do programming he worked for Nokia, for his own networks and stuff.

CLAIRE: All right. So A lot of people who listen to this podcast are into Postgres and they either use it, or they they contribute to it themselves. And so they're probably curious about how you got your start in Postgres, particularly.

So what happened? What was the trigger? Was there a single moment or and let's start with you this time, Heikki.

HEIKKI: So, when I got started with Postgres, I was working at a different company at that point, a few years later, after my first job. So at work, I was writing Java code with the Db2 database.

And I started to look around open source projects to work on. And the first big contribution I made was two-phase commit to Postgres. And the story behind that, is that I was actually on paternity leave with my second child and she was a good sleeper. Like, that was a very easy time, but I had a couple of months of feeding the baby and then watching her sleep. And I had a lot of spare time on my hands. So I started to look around for open source projects to work on. And at this point I was a big fan of the relational model and databases as a user of databases, but I hadn't worked on any of the database internals.

So I started to look at open source database projects. So I looked at the MySQL code base, you know, how do these things work under the hood? And then I found Postgres and that was a lot nicer to work with. So I started to poke around and just read a lot of the source codes and then the Postgres source code is pretty nice to read.

There's a lot of comments explaining stuff and a lot of history there at that point already. So I quite enjoyed that. So the first patch I wrote was for, or the first major patch was for two-phase commit to implement two-phase commit in Postgres, because I noticed that was missing in Postgres at the time, and that was something I was using with Db2 at work coincidentally at the same time.

PINO: Why did you... Two-phase commit is, oh, I'm sorry. Please go ahead Claire

CLAIRE: Why did you find Postgres nicer to work with specifically?

HEIKKI: I can't explain it, I guess. I looked at the code base and it was a pleasure to read. It was understandable and, you know, I could find my way around it.

CLAIRE: Is that still true today?

HEIKKI: I like to think so. I hope I haven't made things worse over the years. But I like to think so. There are probably other good code bases around that are equally good or, you know, but I still enjoy working with Postgres. So of course, by now I know all the nooks and crannies and all, it doesn't look as clean and nice to me anymore because I know that some of the things that might look that they're nice and clean are actually dirty hacks I put in five years ago or something.

But yeah, I still hope that it's approachable to new developers.

CLAIRE: Pino, you were going to ask about 2PC.

PINO: That's right. I was going to ask about 2PC because it's a pretty technical topic and you know, sort of foundations of computer science-type topic.

Was that very challenging?

HEIKKI: Well, you know, I didn't know that back then. I was a self-taught programmer and I didn't know how hard it should be. But, yeah, I didn't find it that particularly hard. But of course, the thing that happened, that happens to a lot of people who write their first patch to Postgres is that when Tom Lane committed it the patch that was committed didn't look very much like the patch I actually submitted.

But I was glad about that anyway. I'm proud of the first patches.

CLAIRE: Awesome. So one of the things when Pino and I were preparing the set of questions that we wanted to ask you is we realized that we don't even, I don't even know all of the scope of the types of contributions, the types of projects that you each have worked on. Like you just told us you've worked on two-phase commit, Heikki, as your first project. And I know that Andres has been very much involved in asynchronous IO and direct IO in recent years, but I'm curious if you can each give us examples of the types of projects you've worked on and then we need to go find out more about Andres' start in Postgres too after that university class.

But first, types of projects.

Okay, you're not both answering at once. You're both being respectful of each other. Okay. Andres, go first, please.

ANDRES: So my first contribution to Postgres? It's kind of related to the question about how it got started with Postgres.

CLAIRE: Perfect.

ANDRES: I wrote some program using Postgres and it had a very, very bad schema design.

And because of that bad schema design, the Postgres planner had a really difficult time planning my queries and would often not finish ever. So because it was using the Genetic Query Optimizer. So my first contribution was making something in the Genetic Query Optimizer faster, but it didn't actually address any of the actual problems because it just made like, I don't even remember what, something faster but didn't fix the algorithmic issue.

And I think I sent a patch to approve something, and then Tom Lane rewrote that part of GQ within, I don't know, two days or something and completely obsoleted my patch. And I think there were a few more changes like this where I submitted something and it turned out to be fixing the wrong problem.

And I think one of the first things that I then got merged was... Making random numbers in Postgres less random in a way by allowing to initialize them to some value. So that GQ could be made reproducible, so which was important for testing and stuff like that. I think after that, I...

PINO: Oh, sorry, Andres, do we have time for a quick detour to what is Genetic Query Optimizer?

That sounds fascinating. It's a newbie question. Is that still used?

ANDRES: It's still used sometimes. In the end, Postgres tries to optimize in which order to do joins. And if you do that exhaustively, it is very, very expensive. It's potentially exponential. You can't do that if you have very large numbers of joins, just because of the cost.

So after a while Postgres tries to use some approximate solution. And I think back at that time, the en vogue approach for that was genetic optimizations, which was basically trying to mutate something like a prior guest approach and then see whether it is better than something before.

And that's the Genetic Query Optimizer and has different approaches and a lot of those different approaches are if-defed out. It's all a bit odd. I think the limits at which point it gets used have increased over time. But it's still used if you have tables with queries with lots of joins.

There's some GUCs for configuration variables that say after one, when it's used to join collapse limit and something else another GUC that controls when it gets used.

PINO: Okay, that's fascinating both for the for technical reasons and because it's sort of a bit of history and trends in computer science.

Thanks. Thanks for that.

ANDRES: It's interesting. By the way, it works, kind of, sometimes but if somebody had the energy, rewriting it from scratch would probably be a good idea at some point.

CLAIRE: It's interesting that you both mentioned Tom Lane as the reviewer for your first patch submission. And nowadays you're both committers, so you both are probably the first reviewer for other people's patch submissions.

Maybe you could spend just a moment and explain that whole process. I mean, basically any contribution before it goes into Postgres has to get reviewed by somebody who's a Postgres committer and/or other contributors. Is that right?

ANDRES: Yeah, it has to be reviewed by somebody. It doesn't have... often the first round of review, isn't a committer just because of scalability issues, like given the amount of attention committers have, and then eventually the patch needs to get merged and that has to be done by a committer because we are the only ones that have the right to push to the repository.

And I think it does, one thing that has changed since at least the time I started contributing and now is that, I think back then, the time until your patch got reviewed and potentially merged has increased a fair bit just because the product has grown a lot and the patches have gotten a lot more complicated.

So I think that doesn't make it harder, but just get started today than back then.

HEIKKI: In principle, the process is very simple. You write a patch and you attach it to an email and you send it to the hackers mailing list, and then it gets committed by a committer. In practice that that rarely happens so easily, but there's always some back and forth and depending on what the patch is, there might be, you know, you might need to rewrite it.

And what Andres is alluding to is that often what happens is that no one pays attention to your patch and then it gets harder to do. So yeah, in principle, there's very little barrier to entry, but getting someone's attention to review your patches is always a problem.

Yeah. Even for committers, it's a problem to get the attention of others to review your patches.

CLAIRE: Got it. So part of it is. Are you saying that part of...

all right, i'm just gonna go ahead are you saying that part of being a successful contributor these days is not just architecting and programming and coding your work properly, but it's also building the relationships and the trust and communicating effectively with the other contributors and reviewers to make it happen?

HEIKKI: All of that helps a lot. I think it actually really begins with picking the right problem to try to fix or solve. Like, if you get a lot of contributions that are kind of not interesting to a lot of people, or, they might fix your problem, but it might cause other problems. And it's often hard to, as a reviewer, those are the most difficult ones because, I mean, what can you even say on those things?

I mean, I can see why someone wrote the patch because it fixes their problem, but you can't really accept it either. Or maybe even the worst ones are features that no one just pays attention to. And if it's a problem that someone is facing, it's certainly a problem for them, but it might not be a problem that others are acutely feeling.

CLAIRE: Yeah. That's one thing about Postgres. There's no, there's no pre-agreed upon roadmap of what's a priority for the next release, right? It feels very bottoms up in terms of what happens. Am I correct in that perspective?

HEIKKI: Yeah, that's right. There is no agreed on roadmap. People submit patches on what they decide to work on.

CLAIRE: Okay. And if you submit a patch based on something that nobody else cares about, you're potentially going to have a harder time getting it reviewed. Is that what you're saying?

HEIKKI: For sure. Yes.

ANDRES: Those are, if it's clearly something that nobody else wants, it's kind of not too bad, like the harder cases is where like everyone thinks, I guess I can see a point, but it doesn't really interest me.

And if everyone thinks like that, then nobody has the heart to say we want to reject this, but also nobody will ever go around and actually do the work to integrate it or review it. And I think those are the ones that tend to linger for a long time, which is not great.

PINO: I could see how that just would create frustration.

A hard no is always better.

CLAIRE: So last week PGConf NYC happened in New York City, obviously. And I was talking to Melanie Plageman, who has been a guest on the show previously with Thomas Munro on a similar topic, how you got started as a developer and in Postgres. And she's also a Postgres contributor and she just chimed in on the chat with a question of: I'm interested in how both of you decided to stick with Postgres and seek a job where you were basically paid to work on Postgres patches full time?

So let's talk about that. Heikki?

HEIKKI: That's an interesting one. So, the idea of contributing to open source and I've always been a big fan of open source. So from a experiential point of view, so I wanted to find a job working on open source and on Postgres. So I happened to know this advertisement from EnterpriseDB back in the day on the Postgres jobs mailing list, that is from Simon Riggs. So I replied to that and I got the job. And I think Andres has that in common. I think Andres, your first full time professional Postgres job was also with Simon Riggs a few years later, but with his private company? And actually looking back, I think a lot of the committers and contributors you know, have that in common that they've worked for him at some point.

ANDRES: I don't think it was quite the first job, but it depends a bit on your definition. I did a couple of years of consulting on my own around Postgres, where it's like Postgres interacting with the operating system, interacting with the application code. So it was part of it, but not the whole. Then I, at some point did that, but indirectly via 2ndQuadrant and then worked more and more for 2ndQuadrant.

I definitely got a lot of my deeper, more time developing time when working for 2ndQuadrant first.

CLAIRE: But what made you decide to stick with Postgres and, you know, do it full time? How did that happen for both of you?

ANDRES: I think it was one step for me. Like initially I just did consulting type stuff and occasionally I would find Postgres problems and then would look into them and sometimes they were bugs or some minor missing feature. I think around the time I got started, one of the big features that was being worked on was AuthStandby and, I did code contributions to that, but I did a lot of review of it and I'm not sure that I had that much useful stuff to say because I was pretty new, but I had actual potential users for it.

So I looked into that, and reviewed it a bunch and then put it into production, found a bunch of problems. And so there was this interaction between what people that are paying me wanted and the Postgres community. And so it was interesting to continue down that. And I think at 2ndQuadrant, it was kind of similar.

I didn't do full time development at the start. I did a bit of consulting and support of staff and over time, more and more development work, but I think I never had a full time development job until like many years later.

CLAIRE: When was that?

ANDRES: It must have been like 2016, 17, 18? 2018, I think. I think at some point, I joined after 2ndQuadrant, I joined Citus, and there my Postgres development time was officially 50% and then it was not always easy to find the time for that 50%. I think around that, and I was the only one working at Postgres on Postgres itself at Citus, which was when I went to EnterpriseDB and there I had a full time Postgres development job.

I think that was the first one. I think that was 2018 or something. It's a bit hard to remember now.

CLAIRE: Got it.

PINO: Did you move to work on Postgres or were these already remote development jobs?

ANDRES: I moved to the US when working for Citus, but I didn't do it full time.

So I guess that qualifies in a way, but I was also doing about 50 percent development beforehand at 2ndQuadrant. And that was remote. So I think I mostly did it remotely.

CLAIRE: And then even when you switched to EnterpriseDB, whenever that was, 2018 or something like that, you just stayed in the U.S. You did not physically move locations. Correct.

ANDRES: Yes.

CLAIRE: And you just worked remotely? Even pre-COVID?

ANDRES: Yes, I've worked much more of my career remotely than on location.

CLAIRE: Okay, so Heikki, going back to Melanie's question, what made you decide to stick with Postgres and seek a job where you could work? Full time.

I know you said you've landed, you saw this post from Simon Riggs and you got a job at EnterpriseDB. So was that just opportunistic or was that deliberate on your part? You wanted to work on Postgres full time.

HEIKKI: I think it's opportunistic. Although, I mean, I guess there was a reason I was looking at the Postgres jobs mailing list. So there's that. So there must've been something in the back of my mind thinking about that opportunity back then. I don't really remember anymore.

But what made me stick? I don't know. I guess nothing really ever drew me away from that path. So, that's what I've been, I've been on that path since.

CLAIRE: Okay. So, one of the questions we did not answer earlier was, Heikki can you give us a few examples of projects that you've worked on in the Postgres space beyond that first project with two-phase commit? Is there something you're known for?

HEIKKI: At some point people called me the B-Tree guy, so various little optimizations. Nothing major, I think, but I fixed a lot of stuff.

Issues I worked a lot on write ahead logging. Also the transaction logging of Postgres. I did some big refactoring there to write a pg_rewind. That was one of the things I wrote at some point. So a lot of work around storage, and indexes, I guess those are the main parts.

CLAIRE: Got it. And then Andres, back to that question of example types of projects, because I think it's useful to learn from like, are you always working in the same part of Postgres? Or, or has, is there some breadth to the type of work you've done?

ANDRES: It definitely has changed over time. One of the features I spent, like was being paid to work on, was logical decoding.

Which is basically work that happens on the source site for logical replication for to get all the changes that happen and to extract them from the WAL in logical form that can be applied to another server took quite a while. And one of the things that was very interesting about that as a project was that it involved touching lots of different parts of Postgres because it is involves the WAL logging part, involves the how are our catalogs represented, it involves like transaction visibility, and lots of other parts.

And that was one of the big projects I worked on. Another big theme from around that earlier time was making the locking algorithms in Postgres scale better. At the time we didn't have any lock free algorithms because we didn't have any atomic operation support in Postgres.

So one of the other bigger projects I did at the time was to introduce an atomic operations abstraction and then use that to improve various parts of Postgres to make them scale better across larger systems, because the motivation for that initially was that during my consulting time, I saw a lot of systems that couldn't scale further, and were spending lots of time in small pieces of code. And it was at the time, much harder to see on the systems that developers had because those developers didn't have multi socket systems and so on. But it was already on the bigger end, very visible. And so I tried to resolve those and it was work for a couple of years.

CLAIRE: Okay.

PINO: Now that we've asked you about some of what you've programmed, I want to ask you how you program, and I'm thinking of this the way you'd ask, you know, a writer. How do you write? You know, what time of day? Do you have a cup of coffee with you? Some tea? How do you get into the zone? How long do you stay in the zone?

Can each of you answer that question? Maybe I'll start with Heikki.

HEIKKI: Emacs. I use Emacs. So I mean, that's a good question. Getting into the flow requires, first of all, ignoring all the distractions. So I tend to turn off Slack and notifications. That's one thing. I do a lot of experimentation.

So, I write a lot of patches that never go anywhere just to test and see what might make sense. And then once the prototype's done, I often start to just keep polishing that until it becomes something that can be reviewed by others.

PINO: Do you have a favorite place where you like to do your work?

An office, a coffee shop? And do you code at all hours of the day?

HEIKKI: I code all hours of the day. My typical routine is that I wake up in the morning and that's when I get my kind of first shift of coding. And then in the afternoon when the house is more busy with the family, then I often, you know, take a break for dinner and so forth.

And then I do more in the evening or after the kids are in bed. So that's my typical routine. Does that answer the question?

PINO: No, I think that's it. Just wanted to know sort of where and how and where.

HEIKKI: So I work from home. But at home, I tend to switch places. So I don't have a big screen.

For example, I work on my laptop all the time, which is 15 inch screen. But I like to move around the house. So sometimes I'm on the sofa, sometimes on the kitchen table, sometimes in the bedroom. So that I try to vary that throughout the day.

CLAIRE: Wait a minute. Why don't you have a big screen? Like, should we get you a big screen?

Would you get more done? I find myself so constrained when I have to work just on my laptop and the fonts are small and I can't have as many windows and it drives me bonkers.

HEIKKI: Maybe, you know, I've never tried. I do actually have a screen here, but I never use it. I think I'm a little afraid that if I get into the habit of using the big screen, I can't go back.

PINO: And it would limit your mobility, right? If you want to change...

HEIKKI: Yeah, definitely limit the mobility. Yes.

CLAIRE: Okay, Andres, what about you? I love Pino's question about how you code, like when, where, what's your routine? Is there a special drink? Well, espresso, I guess, knowing you.

ANDRES: Yeah, I also use Emacs, and have basically since I started programming which is probably more like I'm used to it.

It's too late to change. I work different hours of the day, but I think I get more work done in the evening when I feel like I don't need to check in with work email or Teams or any other instant messaging. Not being distracted for a while for a lot of work. The rest, like, the most crucial ingredient for me, but it really depends on the type of work, I find it often easier to do stuff like bug investigations, because, I don't know, the pressure to figure out the problem and it makes it easier to get into the zone. And the next step is often clearer. Whereas, solving or exploring a space where it's nothing to follow, I have to figure it out on my own, that is hard to get started with. So for writing patches for new features, the hardest bit is like getting into the zone and once I'm in the zone, I unfortunately can stay in the zone until I drop there's no real limit, but it's not necessarily healthy, but no food, no drink.

CLAIRE: You might not go to sleep till three in the morning or something?

ANDRES: That has happened on more than one occasion. Yes. I do like some coffee or for a while I was drinking maté while developing but I think that's more important before getting into the zone than once in the zone, once in the zone, it doesn't really matter because I'm in the zone.

CLAIRE: Now you both refer to yourselves as Postgres hackers. You know, in other communities, I think hacker has a bunch of different meanings than it does in the Postgres world. So can one of you please define, what does it mean to be a Postgres hacker? Is that the same as a Postgres developer?

ANDRES: Yes, it's the same.

I think historically hacker was not necessarily the person that was trying to break into a computer systems or something. There was an old jargon file definition of "somebody that creatively interacts with technology." And I think the reason that it sticks around in the Postgres world is that the main development list is called pgsql-hackers.

So, people that are on hackers call themselves hackers. I'm not sure that otherwise it would still be in common parlance given that it has changed the meaning a lot towards a computer security-related context.

Okay.

CLAIRE: One of the things that I love about the Postgres community is it's not just that the code is open source.

It's that everything that the community does is open and transparent. So anybody who's here or listening to this podcast can go and look at the pgsql-hackers mailing list and see the conversation between the contributors and the people submitting patches and the people reviewing them. Does that appeal to you as well when you were making the decision to work on Postgres?

Was it important to you that the whole decision making process and development process was open?

ANDRES: I think implicitly it did. That is a good reason to tell some part of the origin story. The first contribution I tried to make to an RDBMS was actually not to Postgres. I found some bugs in MySQL and I reported them and did not hear back for quite a while.

And the process around that was very opaque. I don't know actually whether that was... Just because I didn't follow the rules or whatever. But then a bit later I contributed to Postgres and it was much easier to see what was happening. And because of that, I found it much more appealing. It wasn't like a conscious analysis of the situation.

It was more like how I was experiencing it and how that worked out.

CLAIRE: Got it. What about you, Heikki? Is it appealing to you that the development processes were all open to the world?

HEIKKI: Yeah, yeah. The open nature of that is very appealing. I didn't quite know how the community operates back then.

And it has also evolved over the years a lot. I find it very fascinating these days, how well it really works, even though there is no formal structure behind any of these things. Who can call themselves, Postgres hacker or developer. There's no rules really. Anything goes.

And even the official project is very distributed and there's separate teams doing the infra stuff, there's the core team, then there's the committers and the packagers. And those are not the same people. It's overlapping sets of people who decided to work on parts of the whole ecosystem. Then there's the extensions. It's all very vague and not well defined, but somehow it seems to work. And I find that really fascinating.

CLAIRE: So we've been trying to ask you questions to dig into your origin stories and how you got your start and why you stayed, right? Why you're still working on Postgres.

There's another question in the chat, also from Melanie, about how each of you approaches mentoring other contributors. Who are starting to work on Postgres full time and how is it, what's easy and what's a challenge as you mentor tomorrow's Postgres, you know, committers, if you will.

HEIKKI: I mean, I'm all open for suggestions on how to do that better, actually. I don't know how to do that kind of trying, I guess, to be pleasant. And teach how things work. If someone submits a patch, explain why, you know, what to look after and so forth. I did participate as a mentor in the Google Summer of Code a few years, but I actually stopped doing that because I felt like I don't know how to do that properly.

And it's not really fair to the mentees if they get a bad mentor. So I actually stopped doing that. But so yeah, I'm all ears if people have suggestions on that.

PINO: Heikki, does that mean that mostly your mentoring is sort of asynchronous remote where, you know, you're interacting over the email thread about the patch or do you pick up the phone sometimes if things get too complicated for text?

HEIKKI: Yeah, I've always done it over email, but picking up the phone probably would be the right thing to do in some many cases. But I haven't done that.

CLAIRE: Andres, what about you? How do you approach mentoring contributors who are new to Postgres?

ANDRES: One thing that I've found is that it differs a lot between people what works for them. There's some people where what they're lacking most is a prompt response to technical questions. There are other people that are looking for a direction what to do and others are, I don't know, need to check in that what they're doing is not a bad idea.

So I found that when I do it, I have to vary it a lot. And one of the difficulties I have is that I didn't have a lot of mentoring experience towards me, so I don't really have anything good to model. It's more like experimenting and trying to figure it out. So I don't think I have really a formal approach that I follow over and over. It's more like I try to do it and I change it if it doesn't work and then change it more, think it's not part of the Postgres community that it's super strong.

I think there's been a lot of one kind of mentoring. I think there have been plenty of committers that have seen that somebody is growing and then spend more time on their patches, but that's a very distant form of mentoring, if you can call it that. And it doesn't really, it only works for people that are... mentees that are themselves very self directed, people that are mostly looking for the technical input rather than, like, something more than that. And I think as a community we could definitely do better around that.

PINO: I think it was Melanie that mentioned to me that you all were either planning or had done some hands on sessions at PGConf.

Did that happen?

ANDRES: I think we are. I mean, depending on what you mean with hands on session, because sorry, go ahead. It's always been mentoring, I would say at PGCon, but like just in a very informal one on one,, or one to two kind of ways by just discussing the products people are working on and talking. But I think we've been talking about doing something more formal and more organized than that, but I don't think it has happened at a real scale yet.

But I think Melanie is actually working on organizing some more of that.

PINO: Yeah. So Melanie just commented those are next year in the chat.

CLAIRE: Exactly. And what does exist today and so far is that I think Melanie and somebody else, and I think one of these was featured in today's Postgres Weekly newsletter that just got published, but they have created sessions where they walk you through how to submit your first patch.

So those types of tutorials or classes exist today, but yeah, I think Melanie, as she just said in the chat, is working on a more formal workshop that will grow Postgres contributors, people who have contributed already and want to take on more and do more. And there's a lot of thought going into thinking about, well, what do they need?

What I think is so fascinating about what you just said, Andres, is that you basically recognize that there is no one-size-fits-all when it comes to mentoring, right? People are different and they each have different gaps or different needs. And so part of like, there are people who can only mentor one way.

And so that means they can only serve, like, a subset of the people who are looking for mentorship. But it sounds like you're trying to meet the needs of what the different folks are looking for. And I think that's huge and that's valuable but it's not always easy. And sometimes...

ANDRES: That also does not mean that I can actually do that successfully for everyone's needs.

Even if I try to vary it, I definitely have strength and I have weaknesses like everybody else.

CLAIRE: Yeah, I hear you. You're really good, but you're probably not superhuman is what you're saying. Is that right?

ANDRES: That is definitely true.

HEIKKI: And mentoring is a good assumption that any of the people who are currently developing Postgres would be very good mentors. Like, just because you're good at something doesn't mean you're good at teaching. It's a separate skill set. So if there are people in the community who are good at mentoring that I think that's super valuable.

ANDRES: I suspect, though, that the bottleneck is more people not doing it and not being used to it, then the actual absolute lack of skill or something, as many others, like you could grow it to a certain degree, at least.

HEIKKI: And I'm sure it's a skill you could learn or like any other skill, but I don't think a lot of people just innately have it.

PINO: I wonder if all Jedi in Star Wars aspire to have a Padawan at some point, or perhaps there are some Jedi that never do. That's something I'll have to go look up.

CLAIRE: You lost me there. What's a Padawan? I'm not a Star Wars person.

PINO: Oh, I see. Well, you know the Jedi.

CLAIRE: Yeah, yeah.

PINO: They can use The Force, they have lightsabers well, often we encounter Jedi that have Padawans.

Padawans are trainees, and so a Jedi will have usually only one. I'm not sure if there are any exceptions, but has a Padawan trainee that is sort of their mentee. And so I was trying to be a bit facetious, but even there, it's a good point. Sort of anyone who has a very specialized skill set there's a separate question of, can they teach it as well?

But perhaps in mastering the skill set, teaching possibly is one of the maybe one of the steps that everyone could aspire to.

CLAIRE: I agree with Heikki that... The ability to teach is a separate skill set, but also when I was a kid growing up, I didn't like to do the cooking in the family, and my sister was very good at baking, and so I made a point of broiling a cake once, which apparently you're not supposed to do and doesn't make for a very tasty cake. But I broiled it and was therefore, I was able to say, yeah, I just can't cook. I don't have those skills. And I was able to get out of that work. And I'm not accusing anybody who's not mentoring of being... is devious the right word? As I was as a teenager trying to avoid all the cooking responsibilities.

But I do think that you have to, if you're going to become a good mentor, you do have to be comfortable being uncomfortable in the beginning. And you do have to be willing to try and learn, even though you might not get it quite right in the beginning. I do think that potential is in a lot of people.

It just takes time to develop it. So I guess I'm agreeing with what Andres said, too.

ANDRES: I would actually say that, I mean, the teaching part of mentoring is often, for some people it is the biggest part, but for a lot of people it's only a relatively small part. And it's more like the being in a sounding board, giving a bit of directional advice and taking some fears away.

That is, I think the most important bit, but like for others, it's also teaching like more technical things, but it really differs.

CLAIRE: So there's teaching and there's coaching and there's listening. There's all those things. Okay, so, one of the questions that came up as we prepared for today in the introductions, in the very beginning, we mentioned that Andres, you are on the Postgres core team.

And somebody said, well, what does it mean to be on the Postgres core team? What is the Postgres core team? What's your short elevator answer to that question?

ANDRES: It is a good question. I think we could, as a community, definitely outline that better. I think in the end, it's basically the place where things that could not be resolved in another way go.

That is one aspect. And another aspect is that there are some issues that for various reasons cannot be discussed publicly. One obvious thing is legal issues. You can't discuss something in a public way and then still have confidentiality, obviously, and sometimes in a legal context that is required.

So that is another aspect that the core team deals with. But I think in general, the core team does a lot less than what a lot of people expect the core team to do. And I think the core team also, and this is largely before my time, used to have more things that the core team did than it does now.

It used to be, for example, that the core team nominated who would become a committer. But these days that's done by the existing committers. And so most of the core, several of the core team members are committers. So in that role, the core team contributes to that discussion, but not in their role as a core team member.

CLAIRE: Okay. So it's not like another project steering committee where a lot of people associate the steering committee with steering the direction of the project, right? Roadmaps and such. It's not like that, but there is some amount of a super high level oversight of the overall project. Did I get that right?

ANDRES: I think it's like, if normally the technical direction is done on the public list by everyone. But sometimes, such conflicts become very personal or cannot be resolved and have continued on and on. And in those cases, sometimes the core team has intervened or tried to resolve it.

And sometimes that involves getting away from a public consensus to just making a call at some point because the community just could not reach a consensus. But I think those cases are very far in between, like, I think there have been at most a handful of those. And sometimes it's the core team gets involved when there are personality clashes and stuff like that, but that's also not a technical discussion, really, it's more like escalating things that could not be handled in another way. So, yeah. There's no directional steering of the roadmap for any of that.

CLAIRE: Oh, I was just gonna say that's a bit of a segue from the whole topic of how you became a developer, how you got involved in Postgres?

We talked a little bit about how you each got your start and you have in common that you are self taught programmers and what I'm curious about is our other contributors to Postgres. Do they have similar stories to yours? Or are there different different origin stories too? Like how similar have your paths been to all the other contributors that you know?

HEIKKI: I guess you would have to ask the other contributors that one thing that I find fascinating or surprising about my path is that I've never really used Postgres. I started as a developer without the journey of using Postgres professionally myself. I think it's pretty common that people start by using and running into a problem and then fixing the problem and getting hooked.

But there are a lot of different paths.

CLAIRE: I know that was Thomas Munro's story and he called it like "needing to scratch an itch." That's how he got involved. Andres?

ANDRES: Yeah, I think there's a lot of different paths. There's some people that come from the university background that have done hacking on Postgres as part of some database course.

There's people that worked on other databases and moved to Postgres because they were sick of all their work being not available to themselves after they left that company, it really varies, but it's definitely other self taught people. I have not recognized one common pattern.

I do agree that one of the interesting different types of Postgres contributors; how to differentiate types of Postgres contributors, is whether they use Postgres or not. I found that there's a surprising number of Postgres developers that have never, or very little, used Postgres themselves.

I think one of the reasons that I was able to succeed was that I was actually consulting around Postgres. So I was seeing a lot of problems that people had with Postgres when used in anger, and that allowed me to see people the problems that other Postgres developers didn't see, which then basically provided a niche for me to exist or to work on where I had a small advantage over others because I was still seeing all those day to day problems by people.

CLAIRE: Did you say when used in anger?

ANDRES: Yes, like when, if you use Postgres in a toy context, it will teach you something about using Postgres, but you will not see the same type of problems as when it breaks down in production and why it broke down in production or why it's when it's too slow.

So under high load, you're not typically going to see those in a real world context, if you just use it in some small application for home monitoring or whatever it is for a small website, because you're just not going to put enough load on it for that to matter.

CLAIRE: Okay. I actually think that empathy for the end user is hugely important and it's a superpower for whoever manages to develop it because there were so many individuals and it doesn't have to be as a developer, you could be in other disciplines in the technology space, whether you're a manager or you work on documentation or you work in marketing, but when you optimize for yourself, or you don't fully understand that end user, you're going to make different decisions than if you're really laser focused on that real world customer and what they need and, and their pain.

HEIKKI: There's a funny story back in Postgres 8.4 days, I was working on a patch to rewrite the free space map. And one of the advantages of that was that you didn't need to configure it anymore. There used to be a few configuration settings that you needed to set, you know, high enough or correctly.

Before that time, and I didn't realize for me, that was a segue to some other paths I was working on. But after that was released, people thanked me so much for getting rid of those configuration variables because they were so annoying and I didn't really realize that that was a problem that people had because I hadn't run into that myself.

ANDRES: After that, I was discouraged from those variables being wrong in production.

HEIKKI: That was accidentally, apparently a very good change. After that, I've, you know, spoken with a lot with users nowadays, but back then, not so much.

PINO: And I guess that means that sometimes when you do get questions on Slack or email, maybe not in the hackers lists, but the other lists people are coming with frustrations.

And so all we've been focusing on the path we had, or you all had as developers and Postgres hackers, that empathy that Claire talked about, those are also moments where you're getting the most, the best problem and the best feedback that you can react to do you find that you can mine for that kind of information in the lists?

HEIKKI: Yeah, I mean, there are some recurring themes that people run into I guess, but it's hard. It's hard to, I don't have like statistics on what is the most pressing problem that people have. I have some guesses. People complain about the vacuum all the time and there's a few others that people run into.

But mostly those are also the common problems that if you Google them, you will find answers because someone else has already run into them and there's a blog post or two.

PINO: Was it a big jump though, to realize that those GUCs weren't needed in the case of the free space map? I mean, sometimes we get used to things working a certain way we take for granted: "well, I've just got to do that. I didn't do it right. That's why it broke." And then someone comes along and shows that it's not needed at all.

HEIKKI: I mean, it was a surprise to me because I didn't realize that this was the real problem that people are having. I just figured it's nice to get rid of a few settings. But yeah, I guess users appreciated that.

ANDRES: I found that it's not easy to use the list or other channels as a real development steering
mechanism, because often the problems that people talk about or ask about are very far downstream of the source of a problem. And one of the things that I found very useful in doing consulting is that it will allow to get to the source of problem and then address that instead of people talking about the consequences about of a problem that are not easily diagnosable, and it's so hard to diagnose complex problems remotely via email or via other such channels.

CLAIRE: Okay, so I think it's time for us to wrap. We have one last question that we want to ask each of you. Pino?

PINO: Yes, we wanted to ask you if you could give advice to your younger self, to say a past Andres, or a past Heikki, about how to forge your path as a developer and in Postgres, what would that advice be? Andres, do you want to go first?

ANDRES: It really depends on which Andres, like which version of Andres. One of the first in the Postgres community was that it took me a while to learn was that I had to adapt my language to not use my... technical German is much more blunt than english and I had a lot of conflicts in the early years that turned out to be just me not understanding that I can't translate everything literally and have to adapt more on that and the number of conflicts I had just reduced drastically after that and it made it much easier to collaborate and I think it's interesting to have to learn that you have to adapt to the audience much more than I was aware at the time.

PINO: Andres, is that something common still today or something personal to you in technical language in German?

I mean, people tend to speak bluntly to each other and it's much more direct as a language, I think.

ANDRES: Huh. So if you translate it literally in technical, not technical context, it comes over as yeah, insultingly blunt. Whereas it's just like, "Oh yeah, you just stated a fact." So you formulate it as a quick fact and that doesn't necessarily work well.

PINO: So that's a problem. Right. Okay. So that's a problem that's not going to go away. New developers, new German speakers will have that problem. And folks that interact with them might find it useful to learn that. Because they'll adjust their expectations as well.

ANDRES: Yeah. It's generally like the learning that you have to take into the background of people into account is something that I think was important for me and is important, I think, for a lot of people.

CLAIRE: I think we could extrapolate from that lesson, which sounds like it was about the German language being translated to English and different cultural expectations for how you communicate, but in general, when people are communicating only an email and it's not face to face and it's not verbal or, you know, on the phone or whatever.

I think there's a lot of cultures where the written voice... and I've seen misunderstandings over every year of my career, not, luckily, I'm not always in the center, you know, sometimes it's my teammates, but it happens over and over again. And that's why that there's that common advice that sometimes you have to pick up the phone, right? Because in writing, you're just gonna keep missing each other.

I don't know if y'all would agree, but that's. Yeah. That's my learning.

ANDRES: I found that conflicts are harder to do via phone because, not always, but often because there's no cool off period between reacting and hearing and reacting. Whereas with email, I don't know, lots of people, I definitely write a response and delete it and write it a new, and that's harder to do on phone.

Yeah, technical differences are sometimes easier to respond, like resolve if you can hear where people are certain where they're not certain, but some types of interpersonal stuff is harder for me. Okay.

CLAIRE: Heikki, back to Pino's question about giving advice to your younger self about how to forge a path as a developer and in Postgres.

HEIKKI: Gosh, I don't have any big regrets in my career and path. So I think the advice would be like, "it's going to be all right."

CLAIRE: What do you mean by that? It's going to be all right.

HEIKKI: Well, you know, when you're young, you don't know what's, what's going to come in the future. So I think it would be comforting for everyone to know that, you know, good things will come you know, things will be all right.

ANDRES: Don't stress. My second answer was that don't stress too much about what's happening in the moment the longterm is something very different than the short term.

CLAIRE: Okay. Well, on that note, that it's going to be all right: I want to thank both of you, Andres Freund and Heikki Linnakangas, for joining us today on Path To Citus Con, our still somewhat new in our first year podcast. For developers who love Postgres really appreciate your time. This will be published in the next 48 hours.

So it's going to be available on all the podcast platforms for those of you who listen live and want to share the link with other people. Our next episode will be recorded on Wednesday, November 1st, also at 10 am PDT. And our guests for that are going to be Dimitri Fontaine and Vik Fearing with a topic of "How to solve every data problem in SQL."

Those of you who want to mark your calendar now can go to aka.ms/PathToCitusCon-Ep09-cal. And we'll drop that link in the text chat as well before we leave.

PINO: We just want to ask you a favor. And especially if you've enjoyed the podcast, please, please, please rate and review us on your favorite podcast platform.

It helps other folks find this new show. And last but not at all least, a huge thank you to everyone who joined the recording live and participated in the live text chat on Discord.

ANDRES: Thanks for having us. This was fun.

CLAIRE: Thank you, Andres. Thank you, Heikki.

PINO: Thanks, Andres. Thanks, Heikki.