Talking Postgres with Claire Giordano

CLAIRE: 00:00:05
Welcome to Talking Postgres, a monthly podcast for developers who love this database. I'm your host, Claire Giordano, and in this podcast, we explore the human side of Postgres, databases, and open source, which means why do people who work with Postgres, like Adam, do what they do, and how did they get there? I want to say thank you to the team at Microsoft for sponsoring today's conversation. Now, today's guest is Adam Prout. He is a distinguished engineer at Microsoft. He works on Postgres on Azure, and specifically, he's one of the founding architects of Azure HorizonDB, which is currently in preview, which is Microsoft Speak for Beta. Previously, Adam was co-founder and CTO of SingleStore, which those of you who've been around for a while remember was originally called MemSQL. Adam lives in Canada, and early in his career, he also worked on SQL Server. Basically, his entire career has been spent in relational databases. So welcome, Adam.

ADAM: 00:01:09
Thanks for having me on, Claire.

CLAIRE: 00:01:11
All right, so today's topic is from MemSQL to HorizonDB, an engineer's journey. So this isn't going to be an advertisement for HorizonDB nor for MemSQL, but it's really to talk about your whole journey across all these relational databases. So I thought we'd start with your origin story, even before you maybe got into databases, but just as a developer, how did you get started?

ADAM: 00:01:39
Sure, yeah. I think I probably had a more traditional path than maybe some of your other guests. I listened to some of your other shows. But my dad was a programmer in the 60s, 1960s, so in the punch card era. And he started his own software company in the 1980s called PFW. And so I always had my dad look up to and kind of teach me some stuff about programming. So we always had PCs in our house in the very early days of the PC. I think we had a 386 or 286 or maybe both. And it had basic on it. So you remember basic came with this little monkey guy that threw the banana and you tried to hit the other guy. And it came with the code for that. So you could kind of mess around with it in basic. And so that, yes, yes.

CLAIRE: 00:02:26
Did you print your name in big letters? The big A's. Yeah, I mean...

ADAM: 00:02:33
Yeah, I don't know if I printed my name, but yes, that was the easiest thing to program in basic was just to have kind of the blocks form some pattern on the screen. You know, I don't remember if it was my name, but yes, that's the type of programs I was making. That's about all I could do. But yeah, it taught you about, you know, loops and, you know, logic and if statements and whatnot. And in high school, I switched to C++ because this was the 90s and object-oriented programming was all the rage and C++ was the relatively new cool thing to learn. And the first kind of production app I built was a scheduling app for the office of my high school. I don't even remember what it was scheduling. It was something about teachers that would be absent for the day and other teachers would have to fill in if they couldn't get a supply teacher or whatnot. But the vice principal would have this app to figure out who was free. And I wrote it for him as a project for my computer class. And that was my first production code. That was in C++ just because I wanted to use C++. I don't think it actually needed C++. It could have been written in something else. But and from there, I kind of knew like in high school, I wanted to study computer science in university. And that's what I went and studied at University of Waterloo. And that was fun. I did grad school. I did a master's degree there, a little bit of grad school. I wanted to see if, you know, if academia was a thing for me. It turns out it's not that I had a bad time in grad school, but it was I don't know what the right. It's like I prefer to ship production software. It's a little too experimental. It was hard to tell what you're doing mattered because you're just kind of playing with, you know, toy systems, not working on real production systems. So that's when I jumped to Microsoft. And so working on SQL Server. So that's kind of my little arc, but I say kind of traditional, you know, follow, follow in your dad's footsteps. It's kind of a traditional approach to becoming a software engineer.

CLAIRE: 00:04:42
Sometimes people follow in their mom's footsteps. It can happen, I'm just saying. I followed in my dad's footsteps too, so don't get me wrong. But, yeah, all right, let's stay focused here. So did you do anything extensively with databases while you were at University of Waterloo or was it your first job at Microsoft in SQL Server that really kind of brought you into this space and hooked you? It must have hooked you.

ADAM: 00:05:13
Yeah. I mean, at Waterloo, I kind of knew I wanted to work on systems programming. So either compilers or operating systems or databases. And at that point, I didn't have a strong opinion on which. It's like, okay, let's just... So that's where I interviewed at all the companies you would think of if you wanted to work on those things. I even interviewed at Sun Microsystems. Sun was still alive back then. And I ended up at Microsoft mostly because that team impressed me the most. I was not really optimizing for like money or prestige or anything. It was my first job. I wanted to join a team that I felt like I could learn the most the fastest about systems programming. And the SQL Server team was the one that impressed me the most where I felt like I could learn the most the quickest. So that's where I went. That was my first real, like real building of databases experience. Right? The app that I built for the school office, that was using ODBC. ODBC was new back then. Now it's like some ancient technology. Everyone ignores it. And it was talking to like a Microsoft Access database or some tiny little database. So I had to toy a little bit with building apps around databases, but yeah, I didn't really get into it until I joined Microsoft to work on SQL Server. I guess it was the early 2000s.

CLAIRE: 00:06:33
Obviously, there's something about it, though, that has hooked you, besides the fact that systems. You didn't leave databases to go off and switch to compilers or operating systems or anything like that. You've stayed in this space. So do you know what the appeal is?

ADAM: 00:06:49
I didn't maybe recognize it at the time, but most engineers that start working on databases get kind of hooked because there's so many. If you take a look at the spectrum of computer science, there's bits and pieces of that inside of database. So there's language runtimes and compilers inside of databases, right? The SQL language has the same steps you would have if you were building a compiler for C++ or Rust. It's a little more primitive, but the steps are there. And it has all kinds of algorithms for in-memory data structures and on-disk data structures. And depending on the database you're working on, distributed systems, you know, consensus algorithms, Paxos, Raft. So there's such a breadth of things. So you're pretty well not working on the same little narrow domain, you know, year in and year out. You're kind of jumping around, you know, whatever issue happens to pop up. Maybe today you're working on some network protocol. Tomorrow you're working on some on-disk data structure. Today, you're working on something in the query optimizer, so you get quite a bit of exposure to different things, so that's why I say the database, once I started working on it, is like, wow, why would I want to work on a compiler? That's such a more narrow domain than databases, right? I can get exposure to compilers and operating systems. SQL Server very famously has its own operating system layer mostly because I think Windows was fairly primitive when it was built. It's called the SOS, SQL OS. So it has its own little mini operating system that it comes with. It's like all these things are there. I just, you know, you don't need to leave the database space to get exposure to them.

CLAIRE: 00:08:35
So you were at Microsoft for a number of years. By the way, I love what you just said, and it's probably going to turn into a great quote because there are so many interesting bits and challenging problems in databases and in Postgres specifically, which is, of course, the database that I pay the most attention to. But I guess I want to jump ahead. I can't remember how many years you worked on SQL Server at Microsoft, but I do know that at some point you left and you co-founded MemSQL and currently called SingleStore, but then called MemSQL. And I've never asked you, is there an interesting founding story there? Were you eating pizza in Paris and there was a conversation and all of a sudden it happened? What's the story? Is it interesting?

ADAM: 00:09:26
I don't know if it's that interesting. I remember how I said I joined Microsoft because I felt like it had the highest quality people that I could learn from. So a bunch of those folks, some of them had already left SQL Server, but it was a bunch of SQL Server folks. Some of them were at Facebook because Facebook was pre-IPO and it was the cool place to go back then. And we just got together and we were looking at kind of market signals and saying, and we were seeing other databases get funded, like seed funded. I was like, hmm, interesting. So the market is funding database startups right now. There was a couple small acquisitions that happened before that, which is usually what causes more funding. It's like, hmm, we can get funded to do a database startup. And we just started brainstorming where we thought there was an opportunity. And in that time, MySQL was much more dominant than Postgres. This is like early 2000s. So the LAMP stack was the big thing. Linux, Apache, MySQL, PHP. No one talks about LAMP now. No one's talked about LAMP for, I don't know, a long time. So it's gone. But at the time, doing MySQL, it was everywhere. [It was everywhere.] So we said, hey, we'll do a MySQL compatible, but distributed, because MySQL obviously doesn't scale out. And this was the NoSQL era. MongoDB was out there saying web scale, SQL databases don't scale, blah, blah, blah. All these things that we know are false now because there's so many examples of successful distributed databases, SQL databases. But back then, there wasn't one. So we're like, hey, we can combine these things together and it looks like we could get funded. And that's just kind of how it started. We wanted to try our own thing, I guess. I mean, that's one of the pluses and minuses of big company versus small company. But we were like, we were experienced engineers, I would say. It's like we're new. We're like five to 10 years experience. We kind of knew how to build stuff. But at Microsoft, that's still kind of junior. Like Microsoft is maybe unique a bit. And it's like there's so many people with multiple decades of experience. And they're kind of the decision makers. And you have to kind of convince them. And Microsoft, fairly conservative place at starting new products. So to do something like we wanted to do, there is just no way we'd ever be able to do it inside of Microsoft. So that's why we said, okay, let's just do a startup and maybe it'll fail. Maybe it'll last a couple of years, but at least we'll get some experience in doing something completely different because we were all at big companies at the time, doing something different than big company stuff. So it turns out it's still going. It's 15 years later. None of the initial founders are still there, but it's a sustainable business and it'll probably keep going for a while. Then, you know, they're still trying to grow it bigger, but you know, that's one of the reasons I ended up leaving is I couldn't find a way to grow it any bigger. And so it was time to do something different, different challenge.

CLAIRE: 00:12:24
So maybe that's part of the answer to the next question I have, which is why did you come back to Microsoft from SingleStore. I mean, you just touched on why you were ready to leave SingleStore, but why did you come back to Microsoft? You could have gone anywhere in the world.

ADAM: 00:12:41
Yeah, I still want to work on databases for the reason. I just think it made sense. Even though the AI thing had started at that point, this was like three years ago. So chat, oh yeah, almost like two and a half, something like this.

CLAIRE: 00:12:51
Has it been three years that you've been back?

ADAM: 00:12:57
But I left, I took a couple months off. I wanted to take a little break, career break, which I think was very nice. So I took three or four months off between. So it's been almost three years since I left though.

CLAIRE: 00:13:09
I'm always recommending to people when they're between jobs that they take at least two weeks off, which some people don't do. Some people leave a job on a Friday and go to the new company on the Monday, and I'm like, no, no, this is your one chance to not have to read email, not have to get behind, not have to worry about the fact that you might be missing something. Just go sailing in Greece, do something.

ADAM: 00:13:34
Yeah, exactly. Just have a blank slate in your mind. Yeah, it was nice. I just read a whole bunch of papers. I always have a queue of papers. And when I'm working, I can maybe read one a month or something. So that queue just gets bigger. It never gets, I'll never empty it out. But I did a bunch of work to empty it out in between jobs. But I was swinging back. So I think that the reason I came back to Microsoft was I wanted to stay in databases. And so I was looking around. I want to go back to a big company. So I'd been in a startup for 13 years. I wanted to go back to a big company, mostly because at the startup, I saw the journey from nothing to hundreds of millions of revenue and thousands of customers. And at a startup, that's a difficult journey. You don't have the brand name of a big company to piggyback on. But I never got to see the journey above that. Can we take a service from hundreds of millions to many billions or millions of customers? And that's something that's much easier to do at a company. Some startups do achieve that, but it's exceedingly rare. You know, Snowflake, Databricks obviously have achieved that. But at a big company, I mean, most of Microsoft database products have that scale. And so I knew Shireesh, I think you had on your show before, who runs the databases team. He's the CVP and runs the relational databases team because he had worked at SingleStore for a bit. So we had worked together and he was pitching this new thing that they want to do, which was HorizonDB, a new Postgres service. They were going to invest big in it. We're going to build a team around this thing and build it from the ground up. And the existing Postgres service that Microsoft already has called Flexible Server was growing very fast. There was a lot of validation that we could build a very big Postgres business. And so I wanted to see that journey. So it was kind of lining up. I had a boss. I didn't know Affan, but a management chain that I trusted, which I think is important at a big company. You kind of want to trust your upper management. And saying they're going to invest in an area I'm interested in, a new database service. And it was Postgres, because HorizonDB is a Postgres service, which is a service I knew a little bit about Postgres before this point, but I haven't dived deep into the code or the community. So it was a chance to learn that. So it was basically all these things were coming together. It was just like kind of the perfect opportunity to do. I probably couldn't have found something better. I did look around and see what I could, but there was nothing even close. So that's how I came back to Microsoft.

CLAIRE: 00:16:15
Awesome. Yeah, I will make sure to drop a link to the episode with Shireesh Thota into the show notes. It was fun to hear his journey. A lot of time spent working on databases for him as well before he took on his current leadership role. Okay, so when you came back to Microsoft, I imagine it was quite different from the company you left. Were there any surprises? Is there anything that struck you? And I know you, just to give context to anyone who's listening, you're part of the Azure Databases organization and specifically the Postgres one. You work predominantly remote and you're based in Canada.

ADAM: 00:16:58
Yeah. I mean, when I left, like SQL Server, I was working on SQL Server, but the database business was primarily an on-prem business, like packaged software that you sold and people installed it.

CLAIRE: 00:16:59
And I would say you probably have teammates all over the world in different places. So surprises?

ADAM: 00:17:18
They had a cloud offering in the works, but it was a very small, it was kind of like a skunkworks effort. They were just trying to get it off the ground when I left. But now flip to today and almost all of the effort is around the services. Obviously, Postgres, we don't sell it on-prem at Microsoft. You can't buy support from us or something. We only sell the services. But even SQL Server has four very big successful services built around SQL Server, and they still have the on-prem. It's not there. But the emphasis on running the services in live site, none of that was there. Azure barely existed when I left. So that's quite a change to how things work. I mean, we weren't on call when I left. There was no on-call for the engineers and now there's on-call. So that was the biggest change. But I think that's true. If you wanted to work on databases anywhere, I think that's the same. Any of the big companies now, this is the way they tend to operate. So that's quite, quite different.

CLAIRE: 00:18:29
I know from a conversation that we had, I think the first time I met you, which was at an offsite up in Redmond in like your first six months. But what one of the things you told me is that you were surprised to learn about Azure Database for PostgreSQL and just how big that business is or, I don't know. I mean, maybe it just wasn't on your radar when you were at SingleStore. But I don't know.

ADAM: 00:18:59
Yeah. Yeah. Yeah. I remember that conversation. I wasn't much into the Postgres community. [I was.] I was in the, MemSQL is MySQL compatible. So I was kind of more on that side of the fence. And in the end we were just selling to big businesses. Nobody cared that we were MySQL compatible by the time we renamed a SingleStore, but we were kind of our own thing. But yeah, people did not associate Postgres with Microsoft in, at least in Silicon Valley, that's I was before I came. I moved back to Toronto a little bit before I joined Microsoft. So in Silicon Valley, everyone would use Amazon and maybe a few people would use Google. And so I was surprised the size, the scale and maturity of the Flexible Server, the existing Postgres business, the size of the service and the maturity of it, the feature set of it. And I also didn't know that Microsoft supported open source Postgres to the extent it does, which like a lot of the folks you've had on the podcast, like these, these guys are a whole team.

CLAIRE: 00:20:01
Oh, you mean that we have a whole team of open source contributors and committers?

ADAM: 00:20:04
Yeah. That is very unique thing. Like, I mean, that it's like a group of people that are paid just to do thing they would probably do anyways. I mean, they have to pay their bills, so they need someone to pay them, but they're working on open source. Obviously, we consult with them and on HorizonDB, we consult with them on various things on how we want to do it and whatnot, but their day job, where they're typing code is open source. They're not typing code on our closed source stuff, but that's very unique. I don't know. I've never seen, to this scale, I don't know how big the the community. A good number of engineers are employed this way. I think that's a very unique thing. I had no idea the extent to which Microsoft had committed to the open source community for Postgres and yourself too. That's mainly your job, your day job. You're going to how many conferences a year? I don't know.

CLAIRE: 00:21:09
As many as I can. Yeah, it's interesting because a number of the people that were acquired as part of the Citus Data acquisition, they have startup DNA in their blood. And many of them have gone off to work, you know, after being at Microsoft for a number of years, they went off to go found a startup or join a startup or get back into that world. But I'm still here. And one of the reasons I'm still here is just because it's, I love working with the Postgres community and on the upstream open source project. And so it's wonderful that Microsoft will pay me to do that. And I just think it's a privilege. Obviously, I do some things that aren't just for the upstream project. So my job, unlike the committers that you just referred to, where 95% of their time is focused on the open source project. I don't know. I've never figured out the percentages. Am I 60/40? 40/60? Whatever. But it's still a privilege and I love it. So I am still here. So the title of this episode is From MemSQL to HorizonDB, an engineer's journey. And I think I would like to ask you to share a definition of Azure HorizonDB for anybody listening who maybe either hasn't heard of it or isn't quite sure what it is yet.

ADAM: 00:22:37
Yeah, it's a more cloud native service built around Postgres. So if you want to think of Flexible Server, that's our existing Postgres business. We mostly take open source Postgres. I think there's something like 12 very small patches inside of Postgres, like little tweaks to the security model here and there so we can run it as a service. But by and large, it takes Postgres, it takes Azure. So Azure VMs, Azure managed disks, Azure Blobstore, builds a control plane. And basically everything is off the shelf if you you want to think of it that way. Where HorizonDB, we're looking at what-- if we open this up a little bit and say, we're OK with making some changes to Postgres, and we're OK with making some changes to Azure so that the two can fit together very nicely, so we can have a more cloud-native Postgres. So things like very specific storage systems for transaction logs that are low latency, do quorum commits, and are much faster than an Azure managed disk that you just get off the shelf. the managed disks weren't designed to store transaction logs. This is one example. There's like five or six examples where we're willing to say, "Hey, we're going to make some changes to Azure. We're going to make some changes to Postgres. We're going to try to make those two puzzle pieces fit together very cleanly." Whereas Flexible Server is very, very compatible with Postgres. It's like almost just normal Postgres, open source Postgres, community Postgres. And so in that form factor, it was hard for us to do this because that's the way customers buy Flexible Server. They pay for an Azure VM. The VM type is exposed to the customer. In HorizonDB, it's not. You're paying for cores, but it's more hidden from you. We have more flexibility in that form factor to make changes to both Azure and Postgres. I know that's more than a sentence, but this makes some changes to both to give you capabilities that are just hard to give you. Because remember, Postgres is this amazing piece of software. It's designed to work everywhere. It's designed to work on-prem, on like, I lost track how many operating systems it supports. Definitely, it's not designed to run in any of the clouds. It runs well, but it's not like there's people back, whatever, 30, 40 years ago when Postgres started, it's like, "I'm in cloud. Let's design Postgres for cloud." So that's kind of the long and short of it, is just letting us make some changes on both sides so we have a system that really fits together and make Postgres run as best as we can in Azure.

CLAIRE: 00:25:07
You mentioned 30 years. So I just want to pause for a second and tell anybody who's listening that this year is the 30th anniversary of the start of the Postgres open source project.

ADAM: 00:25:14
Nice. Please.

CLAIRE: 00:25:18
So yeah, people are pretty excited about that milestone. And I know at PGConf.dev in a couple of weeks in May, which is the annual development conference for the Postgres open source project, there will definitely be some celebrating around that 30 year birthday, which is cool. Okay, so can I try to summarize? Azure HorizonDB, just to give context to everybody, tell me if I'm getting this right. It is, it's still in preview, which is beta, which means if you're going to look right now on the day we publish this episode for the documentation on like the Microsoft Learn portal, which is where all the docs are for the Azure services, you will not find it yet because it's in private preview, private beta, but it, it, it'll be coming out at some point. And then you'll be able to see the docs. So they're not there yet, but they will be soon, but it is a second Postgres managed service on Azure and it employs shared storage. And it has, you're trying to, what's that puzzle piece analogy you just used? That you're trying, in HorizonDB, you've made some changes to how Azure works and you're making some customizations to Postgres so that those puzzle pieces fit together really well. And it's hyper-optimized presumably for running in the cloud. Did I get it right?

ADAM: 00:26:38
Yep. We're running in Azure very specifically. We're using lots of things. that are only available in Azure that make the system easier to build. If we were going to build it on AWS, we wouldn't be able to build it this way. So it's very, because we don't, we're just, hey, we're Microsoft. We're not trying to sell this thing on someone else's cloud. So we can, that's part of what we can do here. [Of course.] We can say, hey, let's make the architecture simpler because we only need to run in Azure.

CLAIRE: 00:27:09
Okay, so it sounds like when you were recruited, there was already internal discussions about creating this new offering. But I presume you joined early enough that you can speak to why? Why we created it? Yeah? Why?

ADAM: 00:27:25
Yep. So part of it was market signals. So this split, having a very compatible Postgres and then kind of a more cloud native Postgres where it's not quite the same. It's going to perform a little differently. It's going to have maybe some of the extensions won't work. That split was already there. So Amazon has that split. Google has that split. And since we started the projects, I lost track of how many Postgres services are there. Snowflake has one. Databricks has one. So the demand for this more cloud native Postgres, it was clear. Everyone looks at DB-Engines, for example, this very famous ranking of databases and shows the Postgres popularity going up and MySQL and Oracle going down. And SQL Server, unfortunately, is also going down a little bit, unfortunately, for Microsoft. So, yeah, the market demand was definitely there for the system. And Azure just had this gap. We had Flexible Server, it's growing well, but we can really service the extreme top end of the market. So Postgres is kind of like this very cool database where developers love it. So your smaller apps, your Supabases, your your Herokus and all these guys. And then at the very top end, it's also a target for Oracle migrations. So your big iron tier zero, tier one apps that, you know, must be always on and have lots of scalability requirements like this OpenAI, which I think, I don't know if it's been talked about on your podcast, but OpenAI runs part of ChatGPT on Flexible Server, the existing Postgres service. And there's some blog posts out there. I would check them out. It's pretty neat to see how they do it at the scale they run at. I lost track of how many read replicas they have now. It's like a hundred or hundreds. But that is common, right? And so we knew we needed to build a little bit of a different system for those very enterprise-grade workloads that are coming to Postgres, they're coming from Oracle, or they're just being written on Postgres. If it's a new app today, Postgres is what people choose by and large. So the market demand was there and Microsoft had a gap here. Flexible Server doing very well. As I said, amazing. I just had no idea. I don't think think I'm allowed to say the scale of it, but someday you guys will get clearances to say how many cores or how many VMs or whatever, but it's big. It's a big thing, big numbers. So we had a lot of validation that if we build a more enterprise-focused server, not to say Flexible Server isn't, but again, we just can't make the changes to Postgres that we want to make in Flexible Server. So we had the kind of the market signals that there was demand. Microsoft is very conservative about this. Like Amazon launches new database services every month. I just lose track of them and they kill them off like a couple of years later, if they don't take off, Microsoft does not do that. Microsoft, when they launch something, they put a lot of effort behind it. And it's because they have very firm belief it's going to work. And then we'll just keep going. We won't stop. That's just the Microsoft way. We may be a little behind because we don't act first. But once we do act, we act in a big way. So I think you'll see that with HorizonDB. You're going to look at this, you know, some of the stuff in the preview and think, maybe that's kind of similar to my existing system. But there's lots of things coming. Stuff that I can't even talk about today yet beyond that's in the works. And, you know, you'll see it showing up in the next year. It's not that far away. So, yeah, that was the reasoning. And Shireesh, as I mentioned, was kind of the guy really behind this, which is a big deal and interesting thing for you. I'm sure you maybe asked him about this, but SQL Server is a ginormous business for Shireesh, like a huge. Again, can't tell you. It's just unbelievable how big it is. So the fact that he's willing to invest in Postgres just tells you the belief he has in it, right? Because he already has this massive database business in SQL Server, and he's also willing to invest in Postgres. He's not just saying, no, no, no, you guys should all use SQL Server. He's willing to say, wow, there's a big piece of the market that wants Postgres and we need to be there. In spite of us having this massive SQL Server business.

CLAIRE: 00:31:54
You know how a lot of times people view the world from their own little vantage point? You know, what does that mean for me? And so I'm human. I fall into that bucket as well. But one of the things I really like about Shireesh is that even though he has this big engineering and product job where, you know, he owns the whole Azure Database business at Microsoft, I also know within the Postgres space, he appreciates the community work that I do. He took the time to be on this podcast. If I ever get stuck or I ever get in trouble, not that I misbehave, but you know what I mean? I could turn to him for help. He supports our Postgres open source upstream project work and backs it. And that is just huge. And I really, really appreciate it. So that's the world according to Claire.

ADAM: 00:32:47
No, I'm with you on the engineering side. He's 100%. I mean, that's one of the reasons I joined, because I knew if he said we were going to do this, that he meant it. So he's definitely on the engineering side also getting us... Headcount these days is not easy to come by at any of the big tech companies, all the layoffs and stuff. So getting us headcount to work on this was something he had to do quite a bit of work. So...

CLAIRE: 00:33:12
And successfully.

ADAM: 00:33:14
Successfully, yep.

CLAIRE: 00:33:16
Okay, so let's talk about database architecture. I really liked one of the things I was listening to this talk you gave as I was preparing for today's episode. You gave a talk, one of the database tech talk series that Andy Pavlo organizes at Carnegie Mellon. Every year or every semester, I should say, there's always like a different theme and a different title to the tech talk series. And I think this particular semester, which is where you just recently gave your presentation, the title is PostgreSQL vs. The World, which I just love. But you gave a talk about HorizonDB there. And in the beginning of that, and I'll make sure to link to that presentation in the show notes, because I think you do a really good job explaining things. And in a way that I think makes sense to both experts and maybe even people who are students or who are new. I think you communicate well at both layers at the same time. Okay, enough of the compliments. But you said in the beginning, you said, okay, I'm only going to go knee deep into this topic. And so maybe I'm only looking for a knee deep answer to my question. But I want to know how to think about the HorizonDB architecture relative to other similar architectures. What's different about it? What's interesting about it? Versus like Aurora and other architectures like it.

ADAM: 00:34:45
Yeah, so Aurora was the first in this space. So this is kind of shared storage for OLTP and they did it for MySQL and then they later did it for Postgres maybe, though we're talking 2015, they did it for MySQL. I think Postgres was maybe a couple of years later, 2017. So this idea has been around for a while that you can kind of take the storage layer out of a database 'cause it can be done for, and Microsoft did it for SQL Server in 2019. So it's been done for three different databases now. You can take the storage layer out of the database and push it into a separate service. And our goal with HorizonDB was to take this to the extreme. So we just wanted Postgres, the Postgres compute nodes in HorizonDB, we wanted them to only run queries and transactions in your app logic. All the work to like replicate the data, make it durable, take backups, is all pushed into the storage layer. So you don't pay any compute or I/O or network cost in your kind of compute VMs because that's how they're sold. They're sold based on, and this is industry standard, they're sold based on cores and memory. And so Postgres is using those cores and memory to replicate data or write it to disk, it means you can't use those cores to run your app. And so in HorizonDB, our goal was to push as much as we could into the storage layer. So the you're paying for run your app. They do what you want them to do. The storage layer you pay per gigabyte. So it doesn't matter how much work that storage layer is doing, you don't get charged for it. So it's such a nice model for customers. And Aurora pushed some things to the storage layer, but they didn't push near as much as HorizonDB did. So a lot of that talk, I was discussing the various things. I think I even might have showed a table at one point saying, "Hey, look at all this stuff that your open source Postgres would do. And I think I listed the process names that you would see in Postgres doing that work. And I was in HorizonDB, those things just aren't there. You won't see those processes. They're not doing anything. It's hidden in the storage layer. The work still happens. You just don't see it. So that's the similarity. So the similarity, obviously shared storage, HorizonDB, as I said, we're not the first to do this. But one of the nice things we have had being kind of later is we and look at what everyone else did and see what worked and what didn't work. Because one of the nice things of the database, everyone writes papers, right? There's SIGMOD and there's VLDB, these conferences for database builders. And I think there's maybe 25 or 30 papers a year published in these things. And there's an industry track where it's just people from industry, like production systems. So I lost track of how many papers have been published on shared stories for OLTP, but it's dozens. And this is real gold stuff because these are, again, these are papers of production systems. These are not research systems. So we could look at these papers and say, okay, this worked, this didn't work. And so pushing as much as we could into the storage layer was one of the things that's fairly different with HorizonDB versus Aurora. But the similarities that the architecture looks like this paper from Microsoft, actually, it's Socrates, the new SQL Server for the cloud. I think it's a 2019 paper. That's the very high level architecture, the very big boxes. If you zoom way back out, that's where the boxes go. And I think that paper doesn't get enough credit. I think I mentioned that in the CMU talk, because if you look at that, go ahead.

CLAIRE: 00:38:28
I think you put a screenshot of it up on the screen [Yes, I think I did.], which I find is always super helpful to show, not just to tell, right? [That's right.] And that probably prompted a few people to go download that paper and take a look at it.

ADAM: 00:38:40
I hope so, because it's a very good paper. I'd like everyone to read it, because Aurora also wrote papers, and they're good papers too, but, if you look at the study of those, I know several dozen papers. As soon as that Socrates paper came out, everyone switched to that architecture thereafter, they were taking to their versions of it, their tweaks on it, but that is the most common architecture. And that's the one HorizonDB follows. And the big thing about that architecture is separate storage for logs and data pages. So there's a specialized storage system for transaction logs, like I was mentioning earlier, because they have very different write patterns and very different latency requirements. You don't care about the throughput that much. Hundreds of megabytes is not that much throughput for a storage system, but you do want the latency to be very, very low because every time you type commit, you're waiting for that data to become durable. And so your app is stuck waiting. So you really want to limit that.

CLAIRE: 00:39:35
So you're talking about different latency patterns for writes and specifically for logs?

ADAM: 00:39:41
Yeah, for transaction logs, right? So the transaction logs versus data pages, a lot of the writes are async. You're like relatively rarely wait. You can. It depends on what your app is doing. You're rarely waiting on those. So you have a lot more of them, right? That's your whole data set. You might have a terabyte of them. So where your transaction logs, you probably have the tail of the log that's hot that you're writing to hundreds of megabytes. So this paper is the one that said, hey, you should use separate storage systems for logs and transactions, optimize them for the patterns, the read and write patterns of those two different pieces of data. And then you should put your pages, you should tier them into blob store because some pages are hot and some pages are cold. And so they were the first to really do this OLTP over blob store in production. Like Aurora does not do that. Aurora only stores backups in blob store, at least as far as I know. I see Aurora could be changing internally and not publishing new papers. So they all store their backups in blob store, but the full data set is on like equivalent to an EBS type disk. Whereas in the Socrates design, they're still caching in front of it because the blob store is obviously too slow. But the durable layer is the blob store. So basically all our storage services in HorizonDB could die, and we will not lose any data because your data is in the blob store in the end. The durability layer is the blob store. And this is the two things they did in that Socrates paper that we also follow for HorizonDB, and lots of others have followed too. So that's kind of the big similarities.

CLAIRE: 00:41:22
I know in your CMUDB talk, you also talked about Rust specifically. Tell me more. Open-ended. It couldn't be more open-ended than that.

ADAM: 00:41:38
So, yeah. So, at Microsoft, there's a push to replace anything that you would historically build in C or C++, so high-performance systems, programs, you know, operating systems, databases. push those things to be built in Rust. And the reason for this is because it's mostly for security. Like Microsoft, Azure in particular, is probably one of the most attacked clouds in the world because of the number of government customers and very big customers, like nation state attacked and looking for vulnerabilities. And so they know where the vulnerabilities typically happen. And in C and C++, it's a very high-performance programming language, but it's also very error-prone. You have to be very, very careful. And there's tools to try to help you, but the compiler helps you almost not at all. So your program can compile with ridiculous bugs in it, buffer overflows and whatnot, and it'll happily compile and run. So Rust, it doesn't catch everything, but there's a large class of bugs. Your program just won't compile. And there's another class of bugs. It will crash at runtime in a very predictable way, not let an attacker get in there. So this kind of memory-safe programming in Rust is a very big thing at Microsoft. And so we were--it's not mandatory, but we were kind of strongly encouraged when we were building HorizonDB that you should use Rust, because this is going to be a new storage system. system. We would probably traditionally have built it in C++ if Rust wasn't around. And so that's kind of where we went down this path of using Rust, which is problematic mostly because even though Rust, I think it's like 10 years old now, it's not like brand new, but C++ is like 30 years old. So there's tons of experienced C++ programmers in the world. There are relatively few experienced Rust programmers. So it's a little harder to bootstrap.

CLAIRE: 00:43:39
Okay, but just for clarity, Azure HorizonDB is still predominantly, it's Postgres under the covers, and that is, of course, written predominantly in C, right?

ADAM: 00:43:47
Correct. C, yep.

CLAIRE: 00:43:51
So the Rust bits that you're talking about are the shared storage implementations. [Correct. Yep.] Okay, got it.

ADAM: 00:43:58
The new log service and the new page service, the stuff that you don't really see when you're using HorizonDB that are giving you the kind of the performance and scalability benefits, those are all Rust. And also some of the Postgres extensions, because we had to make some changes to Postgres to hook it up to these different storage systems. Because Postgres designed to talk to a disk and we're not a traditional disk underneath. So those extensions are pgrx, which is also Rust. So we're using it on both sides. Now some of the extensions are in C. We couldn't do everything in Rust, but...

CLAIRE: 00:44:29
We should include a link to pgrx. I will include one in the show notes as well. So it is a framework that people can use to build Postgres extensions in Rust. And it's, you know, obviously getting used more and more as people. There's over a thousand Postgres extensions in existence. And many of them are probably not built in Rust, but more and more in the last couple of years, people are looking at that. So how do you, here's my, my required, my requisite Rust joke. How do you know if somebody is a Rust developer?

ADAM: 00:45:05
I've heard this before, but I don't know exactly how it goes. Something like they will tell you about it or something like that.

CLAIRE: 00:45:11
They will tell you, they will tell you. So I don't know why I get such a kick out of that. [That's true.] I try it at cocktail parties, but you know, when your friend is a doctor, it doesn't necessarily land the way you want the joke to.

ADAM: 00:45:25
Well, yes, if you're ever on Hacker News, it's always XYZ system written in Rust, right? So it's very true. I mean, Rust is very cool. The other nice thing about Rust is like C++, it's not a garbage collected language, so you can still write high performance code in Rust. So you're not, it's like Java or C# or something where yes, it's harder to write bugs in it, but oh, by the way, you got a garbage collector and you got all these abstractions that make it harder to write high performance code. So it's a nice middle ground, I would say, between Java and C and C++.

CLAIRE: 00:46:12
I want to circle back at some point to talk about you and your journey, but I have a few more HorizonDB questions. I'm curious about how, because I work on the upstream project part of the time, the upstream Postgres open source project, I mean, I'm curious about how open source, whether it's open source Postgres or open source anything, helped with the development of HorizonDB, helped you with that whole architecture and dev process to bring it to market. Did it help? How did open source help?

ADAM: 00:46:47
Yes. How did open source help? Like I mentioned, the committer team was useful because we want to get their feedback on the stuff we're trying to do. I think there is a patch out there to try to make it easier to hook shared storage to Postgres. I don't think that patch has been in the right terminology for Postgres, but it needs to be approved and reviewed and whatnot. And we are looking for things like this where you know, where can we benefit the community as we're fixing things in Postgres. But just the fact...

CLAIRE: 00:47:27
Well, that was my next question. Is HorizonDB helping the open source project in any way?

ADAM: 00:47:33
Yeah, we're trying and we're trying to help with some future work. There's been some talks about memory elasticity in Postgres. So famously, if you want to change the size of shared buffers, you have to restart Postgres, which is something our customers don't like that much. And so there's a bunch of work in the open source to kind of push that forward. And we're kind of helping behind the scenes. We're taking those patches and we're running tests on them because that's something we're not in Postgres yet, but we're actively, now that we have HorizonDB, we really want this for our kind of enterprise customers. So there's things like this. I'm sure there'll be more examples where we're like, we might not be actively typing all of the code, but we're working with the committers and testing the patches and giving feedback before they're in open source Postgres. And I think that flow, that flow already is kind of there for Flex, like Flexible Server, you have lots of feedback on where we're seeing problems with Postgres, either crashes or customer confusion or customer not being able to do whatever they wanted to do in an easy way. And so we expect that same flow with HorizonDB too where we have, you know, make our telemetry available to the committer team if they want to go see something like, "Hey, I'm thinking about this thing. Can you tell me what's going on in your workloads?" That can also be very useful for those folks just to have production data that they can look at and say, "Oh, you know, whatever, this type of query pattern or this feature or whatever.

CLAIRE: 00:49:00
A massive amount of production data from the fleet.

ADAM: 00:49:05
I mean, we can't look at the customer's data, but it's more data about the workloads. That's kind of what we care about in this case, what the workload's up to.

CLAIRE: 00:49:10
Yeah, yeah, yeah. So I'm curious about your job, how you spend your time, what it's like to be an architect on a database service like this that is, like you said, it's ultimately going to be used by how many millions of customers around the world? going to help drive a lot of applications. But I don't have any idea what you spend your time on. I don't understand if AI has changed how you do what you do. And are you spending most of your time coaching or mentoring others? Do you actually do development anymore? Do you have agents do your development? I am just curious. Talk to me. What's a day in the life of Adam Prout?

ADAM: 00:50:01
Yeah, it depends. I think that's one of the...

CLAIRE: 00:50:03
Oh, he said it. It depends. Okay, there we go. Yet another episode.

ADAM: 00:50:05
Yeah, it really depends on the stage of the project, I think, more than anything. So when we were bootstrapping, it was, again, a lot of research, a lot of prototyping, and kind of getting some alignment around how, you know, the high-level picture of what we were going to build. And so this is writing some docs and having some discussions and doing some research. And then when we kind of decided the rough high-level picture of what we wanted to do, I was typing a lot of code because we didn't have, as I said, we were bootstrapping this team. We didn't have an existing team that we said, "Here, you're going to build this thing." We were hiring a new one. Now, we took some people from Flexible Server. So it's not like we started at zero developers. But we started nowhere near enough developers to build the service. So I had to type a lot of code. It was just fine. I enjoy typing code. But this was before agents. I mean, it depends who you talk to where the agents got really good. But I think most people that like the last six months, they've really turned the corner. I think if I had the agents that were around today, you know, two and a half years ago, probably would be more agent written code. But back then, it was autocomplete was the agents. It was mostly hand-typed code. Now, today, a lot of the code is typed by agents is the way I like to say it. I don't know that the agents are designing much of our code because we need to read it all. So today, I don't type as much code by far. The agents do the typing, but you still kind of tell it what to do. But it's so good now. Previously, it would even mess that up. But in the last six months, Opus or the latest OpenAI models, whatever they're up to now, 5.4 or 5.5, they're insanely good. So I don't know that a lot of our code is being typed by hand anymore, but it's all reviewed. We're definitely not to the point where we can't review the code. So that was where I was leading to is right now I spend, I don't know what number I should say, but a large percentage of my time reviewing code, like an immense amount of time reviewing code. And most of it written by agents, but obviously a human there telling the agent what to do and reviewing it too and sending a PR. It's not like it's agents without hands around it, but still we review every line of code in detail. So it's kind of faster to type the code now. As I said, the agents are very fast keyboards. That's the way I think of them. I don't think the Microsoft people like me to say that. But that's honest truth in my view is they're extremely fast keyboards. So once you know what you want to build, sometimes it's like boring code. You're like, oh, man, I don't want to type this. Agent will do this for you in two minutes. It's beautiful. But now we've got a lot more code to review as a result.

CLAIRE: 00:52:52
Do you use... When you think about creating the instructions, the prompt, whatever you want to call it, for the agents, to the extent that you're triggering that versus somebody on the team that you work with, do you use voice or do you type? Because some people are saying voice is a lot faster to just speak to it.

ADAM: 00:53:15
It is, yeah. Yeah, I think the weird thing is now the Microsoft offices are all this open concept stuff. So I asked people this too, and they're like, I'm afraid to do it in the office. But I'm working from home. So yeah, I talk to it a fair bit. Yeah, it's like it's much faster than typing to it. But yeah, I don't know how the folks at the Microsoft offices in Redmond in the open concept, if they're all talking to their computers or not.

CLAIRE: 00:53:42
Oh, because of the background noise and people like sharing a space [Yeah, exactly.]. Okay. Yeah, I don't know either because I work remote as well. So that's an interesting question. Okay. So you are reviewing more than typing these days.

ADAM: 00:54:00
Reviewing code and also looking at some of the future stuff that we, you know, we're starting to get into another round of design phase. You always want to be a little bit ahead of the team. It's not that I design in detail everything we want to do, but kind of pathfind for the team, see what might work, what might not work. Or should we kind of spin up a bigger effort with a bunch of developers in the future? So it's nice to be a little ahead of the team. So doing some pathfinding there and then also, like I said, reviewing a lot of the existing code. And maybe hopefully I can get out of reviewing as much code as I am right now. But it's just the phase, like I said, it's the phase of the project, right? We've grown a new team with a whole bunch of new engineers and relatively few engineers that have been around since the beginning. But eventually, those engineers that have joined in the last year will get very comfortable in the code, and they'll be the ones doing more of the code reviewing than I am. So that's why I said it depends. It's the same at the startup, too. In the very early stages of MemSQL versus the end, the types of things I was doing were completely different, almost unrelated.

CLAIRE: 00:55:08
Got it. You mentioned pathfinding. How does one, if someone's listening and they think, oh, I want to be an architect for a complex system in the future, which might look nothing like Azure HorizonDB, who knows what space they're going to be in. How do you get good at pathfinding? How do you build that skill? There's got to be a little sense of taste to decide I'm going to go down this rabbit hole and maybe it will be productive and maybe it'll be a dead end, but I'm going to try these things.

ADAM: 00:55:41
Yeah, it's a good question. I think you got to care a little more broadly than just your current task, I guess, is the main thing. Even when you're doing some particular piece of the system, you should be looking about what's going on more broadly, either what the competition is doing or what customers are trying to do and having issues or even what they're doing in a way that could be done in a simpler way. Maybe even the customer is not that angry about it, but hey, look, you could do, you know, maybe we'll get a competitive advantage against the competition if we come up with some different way of doing it. So having the kind of open mind or whatever background thread that's kind of watching for these things. You've got to get your current work done. You know, no one's going to be happy if you don't have, you know, you're not making progress on things you already know you need to build. But being able to look more broadly at the set of things going on, because most of the things I'm pathfinding now are just problems in the current system, or we perceive will be a problem based on some of the data we have where we need to do something better than we have today. Or it's something where we're looking at the competition or what the users are doing with Flex or starting to do with HorizonDB in the public preview where we say we need a new capability. And that necessitates some changes in the design of the system. Or we're not exactly sure what the capability should look like. We know a rough idea of the capability. We have to go find and make it specific. And usually that means trying things to try to see, is that was that hard to build? Was that easy to build? That's what I mean by pathfinding. A lot of the times I'm not doing an extremely detailed design, but you're trying to figure out what's easy, what's hard, what's harder than you might have thought and kind of getting those insights back to the team. So you can kind of do the ROI analysis. Maybe this feature is an amazing thing, but it's so expensive for us to build it. It doesn't make sense. Whereas if you could build it in a very cheap way, we would do it because, look, we get nice ROI on that investment. So trying to pathfind all of those things. Yeah, I just think you need to look more broadly. You need to care more broadly than just your tasks in front of you. I think that's the key thing. And yeah, there is some taste. I don't know how you tell. When I say care, what do you care about? Yeah, you need to have some way to triage. What matters? Because, you know, some things you just shouldn't care about. It doesn't matter. So you do need some sense of what the product is doing and where it's going and what you should care about and what you shouldn't care about. So there is some sense. And I don't know how to gain that sense other than just work in that area for a long time. You should develop it. That's what I like about staying in data. That's why I decided to stay in databases. If I shifted to something entirely different, that sense would be gone. I would have to rebuild it from scratch.

CLAIRE: 00:58:41
It sounds like you also pay attention to, like you're reading papers. It sounds like Microsoft benefits from the fact that you don't watch Netflix as your hobby, [Yep.] but you read papers. And so that gives you, puts your fingers on the pulse, if you will, on the other learnings and experiences of other databases. So that's kind of, it's not so much, I don't know if you'd call that a market signal or more of an engineering signal.

ADAM: 00:59:06
Yeah both, yeah sometimes you can get both.

CLAIRE: 00:59:09
Now, you mentioned all these papers that you read. Are you going to be writing a paper? Is that something I'm allowed to ask about? I'm not throwing your softball because I have no idea what the answer is.

ADAM: 00:59:19
Probably we will. Yeah, I mean, the SQL Server team writes a lot of papers. They write a couple papers a year in the premier database conferences. But since we're just so busy getting this thing working, so I don't know if we'll do it this year. But I think we will write a paper on HorizonDB. I think that some of the things we've done are worth a paper. I think we probably can get one published in SIGMOD. But yeah, we're not currently working on it, but it is something we've discussed. Hey, when should we... It's a bit of work to write those papers. Most of us are better at writing code than we are at writing prose. So by and large, it takes engineers a long time to write those papers.

CLAIRE: 00:59:59
I know you mentioned SIGMOD and VLDB as two of the premier conferences where database papers get published and shared. But Aaron Wislang has asked on the Discord chat if you have another source for papers that you want to recommend to listeners who care about databases. Are those the two primary?

ADAM: 01:00:18
So yeah, those are the two primary ones, but, I just, if I'm studying an area, I do the obvious thing. I find a few of the premier papers, then you look at their references, and you can usually start seeing common references between them. And I better go find that paper and read it. But that's if I'm studying one area, like some sub problem, I want to gain an understanding of what others are doing. Another great thing is social media. If you follow the right people or have the right friends list or whatever it's called on LinkedIn, lots of people post interesting papers and say, "Hey, look at this." Not even papers they wrote. Here's this interesting paper. Sometimes it's like a five or six-year-old paper. You just missed it. And there's some interesting insight in there. So following the right people, I don't know who the right people are on X these days or whatever. It's just like there's still a database community around. If you follow them, you can find interesting papers that way. But that's kind of random. You say, "Oh, that's an interesting paper on some topic that I was interested in, or previously interested. But that's why I'm like, that's why I said I'd build a queue, because usually I find these papers out on social media somewhere, but I can't stop and read it on the spot. So I just, put it in my queue. And then, at some point I'll get to it.

CLAIRE: 01:01:35
Where's your queue?

ADAM: 01:01:35
It's like a Word doc. Yeah, it's like, it's not, it's not an advanced system.

CLAIRE: 01:01:36
What software manages your... it's a Word doc? Yeah, I've started to use the Notes app on my phone to create certain cues, especially if I'm at a conference and I'm hearing ideas from people of things I should look into or learn about or whatever. All I've got with me is my phone. And I know if I don't write it down in the moment, the spark of an idea is there. It's gone. I'm not going to remember it when I get to my hotel later that night. So for me, it's the Notes app. But yeah.

ADAM: 01:02:11
Yeah, it's Word for me. But yeah, someone, and I also like to take notes on the papers too. So I also have another Word doc that's like, I think it's up to 200 or 300 pages of notes on previous papers I've read. Because sometimes you want to go back and you're like, what was the key insight in that paper? I know I read it. I know that I have a rough intuition. And so having those raw notes is also very useful. There's like someone should build a little app here. Maybe it's too niche. Maybe I can just have an agent vibe code one up for me.

CLAIRE: 01:02:43
Yeah, I think you can solve that one. I was the kid in college who, you know, you could buy or rent books at university. And I don't know, I always wanted to scribble in the margins and highlight. And like, that's part of how I would learn, like the process of writing the notes. And so I would buy my textbooks. And then if you looked at them, it was like a cornucopia of color because I had my yellow highlighter and my green highlighter and a little note scribbled in the margin that only I could read. And anyway, it worked for me. [Yeah, similar.] Did you do that too or not?

ADAM: 01:03:19
Well, I like to print the papers and scribble on them, but then I tend to lose them after I print them. That's why I also keep the electronic copy, which is harder to lose.

CLAIRE: 01:03:33
Yes. Well, impossible. I've lost things electronically, but yeah, a little bit harder to lose. I should probably link to the Socrates paper in the show notes. So I'll try to dig that up and cover that. You started to talk a minute ago about building things, architecting things, or how you did your work at a startup versus a big company like Microsoft. And I'm wondering if there's an onion to be peeled there, like architecting a database in a startup versus Microsoft. Is there something insightful to share on that?

ADAM: 01:04:09
Yeah, I mean, startups versus big companies, I could probably talk on that for like an hour or something. I'll just put a few bullet points and see what is interesting. But yeah, startups, you have much more desperation. You maybe raised a seed round of a few million. These days, they tend to get a little bigger. Maybe it's 5 million or 10 million, but it still doesn't go that far when you want to hire an engineering team and have an office and whatnot. So you have a few years to ship something and get traction, which when building databases, databases are one of the slower things to build. It's not like a lot of other apps where you can kind of toss something over the fence database. We can't lose data. We have to have correctness. And so it's hard to go fast. So that is the biggest thing that you tend to have smaller, higher performing teams and startups because you have to. The startup just doesn't exist. You will just fail very quickly if you don't. And so you get these teams that work very hard and are very cohesive. They really believe in the mission you're on or they wouldn't join the startup. They would just leave. Where the big company, it's not that they don't have good engineers. It's like, there's lots of this. It's like a job. They do a good job. They put in their time and they just don't have quite the same passion. When I say startup, I mean early stage. Seed or Series A or Series B, like Databricks is not a startup in my definition. That's just a big private company. It's whatever they're at, 5,000 people. So that is one of the big differences. Smaller team, you have to move fast. And the teams tend to be much more cohesive than a big company. And you also ship faster and with a little lower quality. Like Microsoft, yeah, at the startup, because you have to.

CLAIRE: 01:05:57
Wait, wait, wait. You ship faster and it lower quality. In which case?

ADAM: 01:06:05
Again, you have to cut some corners somewhere. And you try to do it in spots that won't burn you or won't burn your customers, but you just don't have a choice. Whereas Microsoft won't do that. Like, big companies won't do that. We know when we put something out on Azure, it will get a lot of use very quickly. Startups, you're not sure. Maybe your thing will be viral and take off. Probably not. Probably you'll be fighting it out for many years. So you have some time. And so they're more willing to put something out there, see how it's used, know it maybe has some missing features or is really an extreme MVP. At big companies, they tend not to ship extreme MVPs. It's just like the quality bar for the initial product at a big company is much, much higher. So that makes it a little harder. And that's why sometimes people could look and say, "Oh, big companies are slower." And it's not always the case. They just have to ship more stuff. With HorizonDB, for example, we had to get customer managed keys and we had to get private endpoints and all these security features working immediately. Whereas at SingleStore in MemSQL, that's something we added in year three, four. We went many years without those features. Where Microsoft is just a requirement. They know their enterprise customers are going to want it. They know if you don't have it, you're going to have a bunch of angry customers immediately on the first day you ship it. So just don't ship without those features. It's pretty well a rule. Same with zone redundancy. Everything must be zone redundant in Azure, basically. Startups is not always the case. It's going to be zone redundant. If a full availability zone goes down, startup probably does not have the full zone redundancy built in, Azure definitely will. And that takes longer to build. It's not like that's a free thing. So that's one of the huge differences. Now, mind you, the split side of that is, so we have to build more of these features in Azure, but it has such the craziest thing to me coming here versus a startup. The startup we fought for 10 years to gain customers, you know, kind of tooth and nail. We're out there fighting for every customer. They're doing a POC. They're comparing us to 10 other systems. And you have to really convince them to use your system. Or Azure just has such incumbency. It's crazy. The amount of customers that will use it almost immediately, it's quite powerful. Your feedback loops can be very quick and tight because you're going to see a lot of feedback very quickly. where the startup, most startups, again, are not going to go hockey stick immediately. Even the ones like Databricks that did eventually become huge, like they had to fight for many, many years to become what they are today. So that's such a powerful thing. That's something a startup would just die for to have that kind of, hey, if I just ship something, I'll get whatever, a million Azure user eyeballs on it, or it's just something we can do with HorizonDB, like startups, there is no such way to do it. You have to fight it out. You're a brand new kind of brand that nobody knows what it is. Like MemSQL, nobody knew what we were. And even at the end, people didn't know what we were. So that was after 10 years.

CLAIRE: 01:09:27
I'm going to pivot for a second. Let's say there's somebody listening who has decided they're going to work on databases, but is picking between a few different jobs or a few different projects if it's open source. And Postgres is one of them. Are there things about Postgres that you really like? Are there things about Postgres that would cause you to recommend it as a career path to a contributor or a developer? [Oh, yeah.] Gosh, I hope your answer is positive.

ADAM: 01:09:56
Postgres. Yeah, I mean, it's maybe leading me on a bit there. But yeah, Postgres is extremely valuable. Postgres internals knowledge, the hyperscalers will pay a lot of money for that versus MySQL, right? Being a good database engine developer is a valuable thing in and of itself. It's not like that's an enormous community of engineers. But you combine that with Postgres and that's, yeah, you will have a lot of demand for your skills. If you have, you know, solid Postgres internals knowledge and you're a good systems developer, you, you know, you have years of experience in the industry. And Postgres, I think is, it's an interesting system. I mean, it's C. So you've got, you've got to like C and I've done most of my development in C++. So SQL Server is all C++ and MemSQL SingleStore was all C++ with a tiny bit of C. And so that's maybe a little bit, I mean, I feel like I'm in the 1970s or something when I'm looking at that code sometimes. But it's the same thing. It's just the dialect of the programming language feels a little old. But it's still the same stuff.

CLAIRE: 01:11:17
Don't knock the 1970s. There's nothing other than the pants and the fashion. There was nothing wrong with the 70s.

ADAM: 01:11:25
It's still the same types of algorithms and useful systems. Knowledge is all there in Postgres. It's just it's C instead of C++, which is like for database engine development, I don't know how many others are written in C these days. That would be maybe only one slight downside.

CLAIRE: 01:11:46
You said as you were talking about this you said that it's being a good database engine developer is valuable. And if you combine that with Postgres and you're a good systems developer, dot, dot, dot, what does it mean to be a good systems developer? If you were trying to hire someone and assess if they're a good systems developer, what does that mean?

ADAM: 01:12:09
Yeah, it tends to mean you're very paranoid. Anything can go wrong and you need to care about all of those things all at once. Usually the very, very best systems programmers are always thinking about the error cases, the edge cases. They're trying to solve things systematically. There can be no room for bugs. In Postgres, the worst kind of bugs, you either have wrong results to a customer. So the customer app returns wrong results to their customer. Who knows what kind of problem that can cause. If it's a banking application, big, big problems. Or it loses data, which is an even worse problem. That's kind of the ultimate bad case. And to avoid these, you have to really care about the quality of your system. And it's not to say like developer off the street doesn't care about the quality. It's hard to describe it. If you send me a code review, I will put comments on your code review and you'll know what I mean. Do you like when you're thinking about your code, are you looking at all the contracts? Are you asserting all the contracts? Are you like making it as simple as you can possibly make it? Are you thinking about how you can test it? Are you layering it so we can test it in layers? Are you writing interesting randomized test cases to try to exercise all the weird code paths that could happen in your system? That's what I mean by a good system programmer is going to do all those things. They're not going to think about it. It's just the way they do it. Whereas if you take someone that's only built, say, control planes or web apps, they're not going to think that way because maybe it's not a serious problem if the web app crashes. Maybe the customer is a little ticked. They refresh the browser. As long as it's kind of rare, life goes on. Whereas your database goes down, we can tell you very clearly you're going to have lots of angry customers. So it's just like in the end, it's the focus on quality and just extreme precision. You're always trying to think where could the bugs live? How can I make the system more stable? And also simplicity. Is this the simplest way of building this thing? Just because it works, is it the simplest way? Can I simplify it? Will the simpler version have fewer bugs, have fewer error cases? So that's kind of what I mean. People that have worked on Linux forever or Postgres forever, that's the way their brain will think. Just out of experience.

CLAIRE: 01:14:32
Okay, that was my favorite part of this entire conversation, which is not to say the rest of it was boring or uninteresting, but I just, everything you're saying, I so resonate with. Oh, I love it.

ADAM: 01:14:45
Yeah, the folks you have on this show, this is the way their brain thinks when they're typing code. And it's not the way everyone thinks. And you don't have to. That's the thing. You don't necessarily have to think that way if you're building in different layers of the stack. You're probably wasting your time thinking that way. But for databases, it's required.

CLAIRE: 01:15:05
It also resonates for me having spent the formative part of my career in the kernel group at Sun Microsystems. The philosophy that you're espousing was the philosophy we had in the kernel group. And of course it is. You think about the fact that someone might be running on top of my OS in an air traffic control tower. So damn, it better be right. And it takes all that. So let me ask you this. You said, if you send me a code review, I will put comments on the code review. Do you ever hold back like, oh, this person, they can't handle more than four or five things. It might be discouraging to them. So I'm not going to mention the other issues. Does that ever happen? Or are you just complete and thorough in your review?

ADAM: 01:15:47
Yeah, I think you have to put the personal aside in code reviews. I think that's... If it's your code being reviewed or you're reviewing the other person's code, you're kind of reviewing the code. So that's the way I look. I don't mind who sent it. It's not like you're being rude or mean or anything, but if something, you know, there's a simpler way of doing it, you just say, hey, did you think about it this way? Because oftentimes the person did and then they'll have some feedback. Oh, yeah, it didn't work out. Or they didn't. And then it's, you know, let me explore that idea. I think it's better just to say to people, you're reviewing the code. And I think broadly speaking, these days, most people don't get too angry in code reviews. I mean, outside of stupid style things, on the structure of the code, usually the arguments are on fact. Is this simpler code? Usually it's obvious. You know, it's either half the lines of code or the data structures are far simpler. It's a factual thing. But for style things, this is where it gets a little touchy. And that's where I like Rust because Rust, the style is built in. There's a formatter and there's this program called Clippy that checks even extra things. And we just say run those. That's our style. Let's not argue in code review about style, tabs versus spaces or whatever the style might be. Or your lines are too long, your functions are too long. We're like, no, we're not going to argue about this stuff. We're going to run the Rust tools. If the Rust tools say it's fine, it's fine. Let's just argue about the code that's in a meaningful way.

CLAIRE: 01:17:28
It's interesting how style plays a role in almost any language. Like I'm just thinking about the English language and the Oxford comma or the em dash. And does the em dash have the spaces between it and the words? And of course the answer to that is no, it does not. The em dash should touch the words on both sides, but you know, yeah, it's everywhere, I guess.

ADAM: 01:17:47
Yes. Yeah, those things can get religious. And that's why it's not argue about style. Let's just argue about actual problems in the code.

CLAIRE: 01:17:59
Got it. Okay. So the topic for today, let's see if you think we covered it well enough or not. It was from MemSQL to HorizonDB, an engineer's journey. So you are the engineer in question. Is there anything else about your journey that you wish I had asked you about that I didn't? Did I forget anything?

ADAM: 01:18:18
No, I think we did a good job. Yeah, I don't think we talked. The one thing we didn't talk about, I don't know if this is a controversial topic, I don't have much to say about it anyways, but I have written code. I haven't contributed to open source Postgres, but obviously we have a private fork. I've written code in that. I've written code in the MySQL ecosystem because MemSQL again, we were MySQL compatible and then the SQL Server ecosystem. So I've written code in the three of the top five most popular databases, if you believe dbengines.com, which I mentioned earlier. And they're actually similar in the big ways, but the way those systems implement things are quite different. So I don't know if that's a topic of interest, but that's the only other thing I was thinking about.

CLAIRE: 01:19:04
I, you know, it's so funny. I do have a list. I call it my safety net of questions. I don't always ask them all. And the one at the bottom that I didn't ask was, what are the differences between systems? SQL Server versus MySQL versus Postgres. At a high level, is there anything interesting you want to share with people? So tell me. You said they're similar in big ways, but...

ADAM: 01:19:27
Yeah, let's just talk about it really quickly. I think the interesting thing for me about those three systems is you can tell the culture of the engineers that originally started those systems, or at least that have impacted the culture that currently exists around the engineers that build them. So, MySQL is kind of the Wild West. Its code is a little rough in the front end, and it just tries to make things work. And it's willing to like bend the rules to make them work. So it's kind of like it was designed for people who want to type code and get angry if it gives an error. Even if MySQL did a weird thing, it'll do something weird as opposed to give you an error. And for like, again, systems programmers, this is terrible because you can accidentally insert something into your table that you really didn't expect. Because MySQL is like, hmm, you tried to insert a string into a double column. Let me just try to do that conversion. And the start of it is a bunch of numbers. And oh, that's probably what you meant. But you have ABCD at the end. I'll just ignore those and insert the number. It does things like this, which that's kind of dangerous because it lets customers have little bugs in their code. And now MySQL folks have tried to fix this over the years. They added strict mode. And I've lost track of what's in MySQL 9 now. But the early MySQL, it's kind of the Wild West. Let's just try to make this system work. Let's not give errors to the user because that makes them angry, apparently. Postgres is the exact opposite of this. We're going to be very ANSI standard. We're going to stick to what's in the standard. We're not going to invent weird behaviors. We're going to give you errors. And that maybe holds Postgres back. Because some of the stuff in MySQL is kind of nice. It's not in the standard, some of the non-weird stuff. But Postgres will tend to try to stick to the standard, try not to add too weird features that aren't in the standard, be much more conservative around what they're going to add. And so you get a little cleaner surface area, much cleaner surface area than MySQL. So that's the culture. You can tell the people making the decisions, right? It's not even the structure of the code. It's like the people making the decisions. How are we going to build this thing? Completely different. And SQL Server is different again. SQL Server was a reaction to Oracle at the time. And Oracle has a million knobs, an unbelievable number of knobs. And if you know how to tune them, you can make Oracle go very fast for a given workload. And SQL Server was like, no, no, we don't want to have a million knobs. We don't want to have to hire an expensive consultant to come in and tune my system. And so SQL Server is extremely adaptive. It's the most adaptive of the three, as far as like you can throw a random workload at it and it'll try to shrink its buffer pool. And it'll try to cause some big query to wait if it thinks that query is going to use too much memory and it would need to fail it. These are things my SQL and Postgres doesn't have. They're just like, "Ah, whatever. Let's fail the query. Let's not try to shrink our buffer pool. Let's have a bunch of knobs. Maybe you can tune how many queries you're running or tune how many amount of memory per query." Where SQL Server will try to make it very dynamic, very adaptive. It's quite nice. I think Postgres should pick some of these things up. I honestly, you know, I've been talking to a few people at some point, it's just because it's kind of just goodness. The code is a little harder to write. It's harder, it's easier to put a knob in the code and say, "Hey, a customer tune this," as opposed to have a knob that you can override and the default is like auto or something and the system tries to adapt. And that, and you have to be careful because you know, it can adapt in very negative ways. So SQL Server had to build this stuff up over many years, just seeing where it succeeded and failed, but honestly, they're not doing something fancy. It's not like this is AI or something. They're like really super adaptive. It was like fairly simple algorithms. They just took the time to build them. So anyways, that's the high level of the three, I would say. It's more the culture than the code. The code, I don't know, you can probably figure out all three of them. You'll see similar components. But the decision-making culture, if you try to add a knob to SQL Server, they're going to tell you no. They're going to argue with you, right? Why is this knob here? Why do you need it? Where I don't think you would get such pushback in the other communities quite as aggressively, for example.

CLAIRE: 01:23:42
I think the decision-making culture being visible in the software and the feature set is a really interesting concept. And it's, yeah, it's probably true.

ADAM: 01:23:54
Yeah, that's exactly it. Yeah, it's how the systems are evolving, the community of folks kind of making those decisions.

CLAIRE: 01:24:01
Now, many of the people that listen to this podcast are already Postgres users or Postgres contributors are somehow connected to the project or to the technology at least. But where was I going with that? If you're not familiar, the mailing lists are all public for Postgres. So you could actually have a front row seat to watching the decisions get made on the pgsql-hackers mailing list, for example. But it's not for the faint of heart. The discussions are And I actually think it's because a lot of the decision making on Postgres is, first of all, it's global. It's asynchronous. There are committer meetings, developer meetings, just twice a year that are in person for a subset of the committers, whoever's there at FOSDEM or whoever's at PGConf.dev in Vancouver. But a lot of the rest of it is all done via email. And so you actually have to be a good communicator in writing in English, I think, to be an effective contributor. I think it's fair to say that. I can't imagine how you would do it otherwise. There is a Discord group now for the Postgres hackers that does exist that I can add a link to in the show notes as well. So that's nice, particularly for new people. It may be a little less intimidating to ask a question on Discord than it is in the mailing list.

ADAM: 01:25:28
Yeah, definitely. Yeah, that's a very good point, Claire. The other two, both SQL Server and MySQL, all the decision-making is private. MySQL is open source, but it's not really open. It's open source in license only. All the decision making is inside of Oracle. So you can't really see what they're deciding to do. I can you can just see the results of it or at least with Postgres, you can watch it. Like you said, you can see exactly why some feature was designed that way if you want. All the details are there.

CLAIRE: 01:25:57
And the history is there too. So it's not just the current conversation, the last year or whatever. You can go back 10 years, 15 years. And, yeah, it's a great place to learn if you talk to today's committers and contributors. Many of them got started by watching the mailing list, reading the mailing list, getting involved in patch review, reviewing other people's patches long before they ever proposed their first patch and submitted it. So it does, that transparent openness can help people learn. But you got to be able to digest that information and read it and process it. [Yeah, makes sense.] So, all right. Well, thank you so much, Adam, for coming on the show. I'm really excited that you made the time. Congratulations on what you've done so far with Azure HorizonDB. I'm looking forward to when I can read the docs pages when it's in preview and they'll be public.

ADAM: 01:26:53
Soon, soon. Yep.

CLAIRE: 01:26:57
So, and you haven't done, PR is not part of your job. This is not PR, of course, but you're not on a ton of podcasts. So I feel special for you joining us here.

ADAM: 01:27:08
Thanks. No, I'm not a big podcaster. You're right. I'm usually busy typing code.

CLAIRE: 01:27:11
As evidenced by the fact that you didn't actually have a good microphone, which we found out when we did the tech check. So now you do, which is great. It means somebody else can invite you on their podcast. There you go.

ADAM: 01:27:26
I'm ready.

CLAIRE: 01:27:26
Open invitation. Adam can thank me later. So thank you, Adam, for joining us. And if you like today's episode and you want to hear more of these Talking Postgres episodes, you should subscribe on Apple, Spotify, YouTube, or wherever you get your podcasts. And please tell your friends because word of mouth is gold in the podcast world. You can always get to past episodes and get links to subscribe on the different platforms at TalkingPostgres.com. And we include transcripts on all the episode pages on TalkingPostgres.com too, for those of you who prefer to read rather than listen. A big thank you to everybody who joined today's live recording and participated in the text chat on Discord.