Intro - 00:00:08: Startups Podcast by Excedr. Join us as we speak with first-time founders, serial entrepreneurs, and experienced investors about the challenges and triumphs of running a biotech startup from pre-seed to IPO with your host, Jon Chee. Our guest today is Scott Chetham, co-founder and CEO of Faro Health, a company transforming clinical trial design and protocol development through artificial intelligence and software automation.
Scott's journey began in Queensland, Australia, growing up on the Gold Coast with a passion for biology and physics and human performance. That early curiosity led him through applied biomechanics and a PhD in biomedical engineering, where he was immersed in the first massive wave of medical device innovation, spending years in cardiac electrophysiology labs, observing early 3D cardiac mapping catheters and pioneering diagnostic tools. Over the next four parts, Scott shares how his early immersion in medical devices, clinical electrophysiology, and applied engineering shaped his vision for modernizing life sciences, taking him from diagnostic commercialization to leading health tech at Verily and Google, and ultimately founding Faro Health.
Today we dive into Scott's formative years in Australia, his academic journey bridging engineering and medicine, the electric early days of cardiac mapping innovation, and why foundational QA and rigorous evaluation matter more than ever when deploying modern AI. Without further ado, let's jump into part one of our conversation with Scott Chetham.
Jon Chee - 00:02:07: Scott, so good to see you again. Thanks for coming on the podcast.
Scott Chetham - 00:02:09: Yeah, thanks for having me.
Jon Chee - 00:02:11: I know it's been a while in the making. I'm glad we finally have the time to kind of sit down here and we always like to start at the very beginning to really learn about what were some formative moments growing up that really shaped who you are and like where you are today. So take us all the way back. What was young adolescent Scott like? What was your upbringing like?
Scott Chetham - 00:02:27: You probably have to go back a little bit. Before that it was a different time also because I was born in Australia, and I, from what then was a small town of probably 200,000 people called the Gold Coast. It's beautiful. It's about 100 miles of perfect white sandy beach with a rainforest behind it. It's sweet. Whenever anyone, I take my wife or other people home, everyone goes, why did you leave? And it's a great place. It's obviously like tourist central. Yeah, even back then that's what drove the economy, you know.
My first job was at SeaWorld. I fried chips over summer starting at 15, which was a short-order cook for a while. So I can't cook for two or four people, but I can cook different kinds of vegetables and anything for 20 people in no time. Still, I learned how to be a production cook and that's a hard, frankly, it's a hard job because you're at the mercy of the chefs and you're 15. So I did learn early on that if you have to cut up three or four bags of onions, the only thing that will save your eyes is going and standing in an industrial refrigerator makes your eyes stop watering. But that was in the holidays in between things. And it was a very different time back then. You know, you get kicked out of the house and get sent on your merry way and come back by dinner time. And it was safe to do that. Obviously, today if I did this with my kids, I would be arrested. So you can't.
Different upbringing, but from like a parenting perspective, my dad was a cabinet maker and taught manual arts. He went from a trade into that. My mum was probably a little bit of the person behind my education because when I was probably about four or five, and I would have been four back then, so pre-kindergarten equivalent here, no one in her family had actually been to university. And this is suddenly when the law changed and briefly in Australia it was free. So she enrolled at the University of Queensland. So it was an hour away north in the capital city of Brisbane.
And I remember even at four, she did a commerce degree and I still remember sitting around a lake. She would take me up there and go to lectures and I had to be really good and quiet. And it was the first time I saw actually a computer, computer, and with the huge, massive floppy disks, not 5.7, like these massive, looked like 11-inch things, and at the first applications of spreadsheets, and if I was good we'd get to go and have lunch with the ducks by the lake. And so that was how I kind of got introduced to university. And she put it... I don't know how she did it, and still today I have kids, she got a degree with two children. Had to drive an hour for there while looking after, you know, being a full-time parent. Managed to do that and then got her first job for KPMG. So she was the kind of, with her sister, were the very first people on any side of our family to ever go to university.
Jon Chee - 00:05:05: That's awesome.
Scott Chetham - 00:05:06: And then she worked her way up to being partner. So one of the first female partners. And so it was kind of interesting to see that journey. And I remember her studying and working at night all the time because she became a chartered accountant and she had to go through kind of that pathway meanwhile with kids. And then my dad was working weekends to pay for some of this and everything. And she did that so she could send both my sister and I to private school. And so that was kind of my introduction to higher education. Ironically, I ended up going to the same university.
Jon Chee - 00:05:37: Was it free too? No. Oh no, no.
Scott Chetham - 00:05:40: No, but it's, compared to the US, it's substantially free. So I mean I had a scholarship for my PhD and everything, so I didn't have to pay anything, which was nice and a stipend, that system still exists, but no. So that I ended up going to the same place. Well, for my undergraduate. And I changed a little bit. As I said, that was kind of what led me to pursue more the academic side of things.
Also, I think the work ethic is... I mean I just remember my parents working all the time, which is good and bad. But I also, the Australian lifestyle, you get four weeks at minimum vacation a year.
Jon Chee - 00:06:12: That's awesome.
Scott Chetham - 00:06:14: And people take it. Yeah, I think I remember a lot of that, like camping around as kids and doing things like that. But it was also just seeing that because both my parents both came from, I would say the nicest thing I could say, not affluent households. People have to understand like in the 70s, Australia was very poor compared to today. Very high what I call quality of life in that you could always afford to eat and there's a safety backbone. But you didn't have the kind of acquisition-maybe-based culture you have here or possession-based culture. But it was a little bit simpler, but it was great. I mean, as I said, it was great to grow up that way. But then see that changed in itself just as I've gotten older and society there has changed and evolved. But that's kind of what got me going down the academic pathway.
Jon Chee - 00:06:57: When you were going through high school and starting to think about university, was biology, chemistry, kind of like life science, always something that piqued your interest or what were you gravitating to while you were in school?
Scott Chetham - 00:07:10: My main interest has always been kind of biology, functional medicine. And not functional medicine necessarily the term here, more what I call functional anatomy, more like orthopedics, biology. But then I also always loved computers, so I used to pull them apart and do that sort of thing as a young kid as well. So I was always in those two camps and always knew that's what I wanted to do. I always knew I wanted to be in healthcare. By the time I was like 15, something around that, probably a little bit before. I always knew I wanted to be in healthcare.
Jon Chee - 00:07:40: Well, so your high school had like programs that would support that interest?
Scott Chetham - 00:07:46: No.
Jon Chee - 00:07:47: Okay.
Scott Chetham - 00:07:48: I have mixed feelings about high school. I went to a very good high school, but it was a boarding school, a little bit military, Anglican church-run. So quite strict and regimented. Six days a week, one day of sport, two afternoons a week. So you were there from like generally 8 till 5:30. And then, you know, I played rugby. So you get bussed around competition games and things like that. But I wouldn't say at that point it was academically orientated at all. And I think there was a little bit of a... not a great fit on that side.
But having said that, it was all very much leadership focused. And it's boys only too, so focused on leadership skills. And its whole philosophy was about being well-rounded. Back then, a couple of times a year they used to just dump us in the bush for a week. Oh no, I had to get back. Yeah, I mean it's that kind of like, become self-sufficient, learn to lead groups and things like that versus necessarily academic. So I was excited to go to university because I was getting a bit bored by the last year of high school. I kind of wanted to do, and plus I wanted to study stuff they just didn't have. And so I was glad when it kind of ended. I enjoyed it. But on the last day of high school, I remember my mom was like, some of the, like we've been to school together for a long time, some of the other boys were crying or upset and hugging and I'm like, I just can't wait to leave.
Jon Chee - 00:09:07: I'm ready for the next thing. Like I'm ready for a change of scenery.
Scott Chetham - 00:09:11: Yeah.
Jon Chee - 00:09:11: One of my close friends lived in Australia in the Gold Coast. He met his now-wife there and he ended up playing for the NRL and he played football here and he was on that reality show, the rugby reality show.
Scott Chetham - 00:09:23: Oh, I didn't even know they did that. Okay.
Jon Chee - 00:09:25: Whenever we're like, we're catching up, there's funny clips of him on YouTube but there's basically a television series for NRL rookies and getting them into the league.
Scott Chetham - 00:09:35: I can just imagine. To be honest, I know that culture really well. So yeah, I was part of it.
Jon Chee - 00:09:41: He's hung up his cleats.
Scott Chetham - 00:09:42: I am so glad. That's why I feel bad for kids and other things, like the fact that they can't be anonymous. That's a whole 'nother discussion. But there's a nice thing about not having perpetual evidence of everything that you ever did forever hanging around. I do feel for people now that stuff doesn't go away.
Jon Chee - 00:10:01: Yeah, it's rough. I was thinking about that too. Even though I'm kind of in between everything being online versus just being purely analog. Growing up, I was very much, my parents are just like, just don't get arrested. Like just don't get arrested. Hit the books, you can just go out and mess around. It was something kind of just being able to not feel like everything is just entering into the permanent corpus of the Internet. That was liberating and now it's rough because everything's on the Internet and everything's recorded and obviously there's ramifications to that. I think the advent of having these devices in our pocket is kind of brutal.
Scott Chetham - 00:10:42: It is. And the context is often wrong. I mean what's fascinating is, particularly with AI, you can do a search on yourself and it's terrible. Or half the information is wrong. And what is really fascinating, and this is way off topic, but we experienced it because we just closed a Series B. There are several venture capital firms who use a well-known research firm and one of them shared the research report on us and it was all wrong. And this is a major company and the thing is, all of their stuff and who they interviewed, it was all summarized incorrectly. It was totally misleading to the point that it was almost damaging.
Jon Chee - 00:11:14: Oh yikes.
Scott Chetham - 00:11:15: And it was clearly some of it wasn't even us. And so it was really fascinating. Thankfully most people saw through it. But a lot of these services rely on truthful interviews. And what was fascinating is you could tell that some of these people have never used our product or anything ever from us or haven't worked with us, but have given interviews as if they did and they were believed because they were credible. And then AI goes and summarizes it because they don't go and verify it. And so that's a lot of the danger right now too, is that any misleading information just gets folded in and folded in and folded in and you have to track it back.
Jon Chee - 00:11:49: And it's also too, I was thinking about it, even just on a non-business side, your Internet identity is forever in a Google search, right or wrong. And it's like, how do you scrub that? If this is incorrect information, we've seen this misinformation spreading like wildfire. And you're just like, this can't be good.
Scott Chetham - 00:12:12: Yeah, well, no, that's what I'm bringing up. It was just fascinating. And thankfully, I would say good investors actually talk to your customers for first-account information. But what I've discovered is these research firms using AI, it is dangerous stuff. And I would caution anyone who's using any of that, and these are big companies, so I'm not going to say the name of them. But the couple, the leading ones, and their stuff was all embarrassingly bad because we're not even in that area. It was stuff where it's like, obviously this is not us. Because everything they say that we don't have a product, that's not even... we don't work in commercial. Like, we've never worked in commercial. I don't understand any of this. It's clearly not us. And it was a few like that. And then you had the good ones where I was like, oh, they actually did talk to a customer. So it was just fascinating to see because thankfully that, I really liked this VC firm. They actually shared it with me and said, can you talk this through? I was a little, first I was like, what is this? What am I reading?
Jon Chee - 00:13:06: Yeah, it makes you really wonder how many decisions... Because that sounds very atypical. Usually they're just like, we'll say no, thank you, and then you go in different directions. But how many of those bunk reports?
Scott Chetham - 00:13:19: Actually, we had three firms. Actually, I was so impressed. Three times people went, can you walk this through with me?
Jon Chee - 00:13:25: Oh, really? Okay.
Scott Chetham - 00:13:26: Yeah. I think the good firms are starting, my suspicion is I think they're starting to realize you can't trust it. And I say, I'm sure there's others who probably did do that.
Jon Chee - 00:13:35: Yeah, it's interesting too. It's kind of like to get back to an analog upbringing, despite all this quote-unquote progress on the digital front, what it's coming back to is you got to do the analog work of picking up the phone, talking to people, meeting people. It's just so much digital noise now that the only way you can cut through it is by being uniquely analog and human. It's ironic. Everyone's like, yo, everything digital, your life will be fully taken care of. You're just like, oh, this is a bunch of gobbly word salad gook. And then you just go and hash things out with someone in person, over a coffee or whatever you got. That was way better. And I know we do it whenever we do our diligence. We're always just picking up the phone, back channeling, hey, what's your experience here? And it's like, this is so much better.
Scott Chetham - 00:14:26: Yeah, no, I mean, and that's the same for sales in this field. All of our sales is bespoke. We have change management services, which is essential in AI. Without that, because people are the hard part. And the generative AI, it's amazing and what it can do, but it's amazing how badly it can do it if you don't have the right guardrails. And so I think people want everything to be automated. And yes, you can save months and months and we've done that now and shown it again and again. But there's real art to it and there's real art, when does a human have to come in and the right guardrails. And so I just don't see them going away anytime soon.
And what's really fascinating is, this one's off topic but then we can pivot back, is we ran an experiment where you take, and I'm sure every big AI company has done this, you take a piece of generated content and then you generate a new piece of content off it. Then you generate a piece of content off the last one and you go how far down the chain before it degrades to nothing. And it's not as long as you would think. It's pretty fast.
Jon Chee - 00:15:22: Like how many turns of it depends...
Scott Chetham - 00:15:24: On how complex it is. For something short, it could be 20 before you get massive obvious degradation. Particularly in my field where you have derived content like flows. It's a workflow. If this makes sense. It means you have to have a lot of what I would say strict guardrails and also extracted-level type things to ground onto a source of truth. Because you just can't keep regenerating. If you want to cheat, just get an image, take a photo and have it just keep changing something. And by the 10th turn, watch for the degradation in the image. That's a faster way to see it because it's pixel dense and the eye picks it up. But in complex documents, you get dangerous drift. I mean this is a lot of stuff we have to protect against and other bits and pieces, but it's the nature of it.
But gone off onto a tangent of AI and what got me into STEM was I just knew I always wanted to be in life sciences and so that was an obvious step. And I think what worked for me is doing a combined degree in a degree that doesn't exist anymore, which is like engineering or medical engineering or medicine and engineering. It kind of combined the two things I wanted to do together. It was a short-lived program. I think it doesn't live in its current form anymore.
Jon Chee - 00:16:32: My major doesn't exist either anymore. They kind of wound it down. They got rid of the building. They were just like, this is not seismically retrofitted, so we actually got to knock this down. And sorry, Jon, you were in the basement of a building that could crumble. I was like, okay, well I'm not at school anymore, but thank you.
Scott Chetham - 00:16:50: Yeah, well, thankfully it wasn't that. It was more they were in the early days of trying to experiment on how do you get a combined skill set so that you can work across domains. Because there's a lot of things, and obviously it's more common now. But we're going back now nearly 30 years. This was at the beginning, it's like, how do you combine engineering and medicine and have a series of people who can have a conversation, who can cross domains. And this was kind of one of the first experiments. It's in a different form now and they streamlined it and it's now a combined 5-year, interesting degree with a different emphasis, different major, all sorts of things.
It was a good, I would say, entry point because you got to see all the medical specialties. You spent chunks of your summer as an intern in different hospitals. You got rotated through also while you're on there. So you had a lot of practical work and then there was a research component at the end. It's ironic that when I talk to people who went through the program and there was only about 30 of us, I think I'm the only one left who actually uses both parts of it. Everyone went in one direction. I think I'm it, but I also think technology and other things have changed. It's much more common now and you find Stanford and others run a more formal combined program and dual specialties and things like that.
But it was a good place to get started and it was also an interesting time, pre-AI or pre-AI in its current form, I would say, because I remember I did a honors project in neural networks for cardiac signal interpretation. And ironically the math and other stuff, it's not that different to the current form at all. But, you know, we just have much more computing power now to throw at it. That was the limitation is the training, the models and just that. And without the whole transformer model Google created, it's not changed that different. So yeah, it was fun, but at the very start of that, I would say that kind of revolution.
So I got to see many generations of technology. Even how I got into research is I was clinically placed at St. Andrews War Memorial Hospital.
Jon Chee - 00:18:55: This is during undergrad?
Scott Chetham - 00:18:56: No, this is when I finished. And then I was going into a PhD program and combined medicine, PhD. And I remember I was working in the cath lab there under the director of cardiology. So I would spend three days in research and two days to three days, because the shifts were very long, I was thinking of like as a cardiac scientist or surgical assistant. So in cardiac electrophysiology, which is basically rewiring the heart. So this was the early days of mapping out arrhythmias, like short circuits in the heart, putting catheters in, feeding them through the groin, and basically burning off aberrant tissue to get it to beat correctly, or putting in the pacemakers, implantable cardioverter-defibrillators. So this was the early days of that.
And how I got into research is St. Andrews is what you call a tertiary referral center in my home state of Queensland. So what that means is that most of the, all the difficult cases get sent there. And at the time, Queensland was about 3 million people. And so we were a captured population. You had to come to us. If you're in the private system, you couldn't go anywhere else. Nobody else offered these services. So we were the perfect site for clinical trial research. And this is actually how I ended up in this whole career. So it wasn't planned. I really wanted to be at St. Andrews because there it was one of the big premier cardiac centers. And I was lucky enough that they took me on.
Jon Chee - 00:20:21: How did you get that role? Was it just a cold call?
Scott Chetham - 00:20:23: No, I had an internship there for three months and I almost refused to leave. So I wore them... I just wore them down by just sheerly turning up all the time and asked, can I watch? Because you have students like, can I watch this procedure? And I just literally through sheer kept turning up, because it's meant to be random, but in the end none of this is. So by just what I would call sheer perseverance and inviting myself in, that they agreed, so they took me on. So what would happen is I would be researching three days a week, one to two full days and then a whole shift that could run to 2, 3am I would be the assistant helping through these arrhythmia surgeries, pacemaker implants and others.
But back to how I got into life sciences is we often did the first-in-man for Medtronic, so the second dual-chamber ICD in the world. We were meant to be the first. We had all the Medtronic staff fly in from the United States. We missed it by 30 minutes because the flight bringing in the stock was late and South Africa pipped us. But we did so early drug research. And that's how I kind of got into it because I started off while I was studying being a coordinator. So I was helping, because it pays quite well, filling in all the paperwork, which back then sadly was colored triplicate paper, you fill in and sign and filling in all these things and for patients and getting yelled at. There's a lot, and stock and inventory and other bits and pieces. So that's how I got and met a lot of the companies because we also did a lot of drug research. Then as I got through my PhD, we got into doing clinical trials for my work in the last year. So I ended up being the principal investigator for that and enrolling patients.
And my boss, who's this amazing, he's retired now, Wayne Stafford, who is the director of cardiology, who is really an amazing researcher himself. Under his supervision, I kind of did that. And it's funny because the reason I am in research is him and his practice partner, another guy called John Hayes, pulled me aside one day and they said, look, you can either stay going down this pathway and you can become a cardiologist and become a cardiac electrophysiologist. You've already done like 500 at this point. You got to the end of, you've already implanted now 500 pacemakers. Look at the logbooks and everything, you've already assisted in all of this. You will be doing the same job again and again and again. This job repeats every five to seven years and we think you'll get bored. And they went, you should consider a career in research.
And that was the direction I then took. And it was good advice because retrospectively, I'm sure they'll correct, still friends with them. Catch up when I go home to Australia. Big career mentors and really good physicians. But that was how I ended up on the life sciences research side and medical device and pharmaceutical research, just because of that being where I was at that point in time and then seeing this massive explosion of technology.
Because what happened is how our center got into closed chest cardiac electrophysiology. Only two, three years before I started, all this had to be done open chest. So it limited what you were prepared to do, because you'd have to open the chest up to find a short circuit and burn it. So you'd only do things that would lead to fatal or immediately fatal arrhythmias. Because a lot of what you see, young athletes and others, we hear sudden cardiac death, that's actually arrhythmia. It was a short circuit they most likely were born with. But nowadays, a lot of that is detectable and treatable by putting a puncture in the groin and feeding a small catheter, a small plastic tube with very fine wires all through this thing, inside the heart while it's beating, you map out the way the heart beats, the electrical rhythm, and you can find the short circuit, and you can just burn it in like two minutes. And they walk out. Cured.
I came in at that transition just after it went from open chest to closed chest. And while I was doing my PhD, we went through generations of technology rapidly, from looking at four or five big monitors of squiggly lines, because these little catheters have hundreds of electrodes on them. And you're trying to have this image in your head of the 3D inside the heart and how these little squiggly lines all line up to know, okay, that's the short circuit. That point, put the catheter there, and then the heart's beating, so it would kick them off. And you might have to spend two hours trying, you know, the surgeon navigating this thing from the groin to hit that part inside the wall and hold it for like two minutes to burn it.
Jon Chee - 00:24:39: Oh, my God.
Scott Chetham - 00:24:40: That's what I said. The hand skills to do this, I'd never had them, but I'm like, these guys, amazing too. That's where it started. When I was leaving, they had this 3D mapping technology where you could put in what we call a basket, a funny-looking basket with all these electrodes and you feed it into the center of the heart and it could do 3D renderings of what was happening and you would just go, oh, okay, there's the short circuit and put it over there. And you wouldn't have to do all this manual mapping and that research and everything happened while I was there. So it was an incredibly exciting time. It's still obviously improving what's in the...
Jon Chee - 00:25:12: Lake, now the 3D mapping, is that the current state of the art?
Scott Chetham - 00:25:15: It's now state of the art, yeah. It's now simpler and faster. And now they do a CT first and so they have fine anatomy and they project onto it and I'm less close to it. But the technology in the catheters is they don't use heat anymore. They can use focused ultrasound and other things to get more specifics and they can just treat more things. And so it's just dramatically changed. But nothing changed as fast as seeing that explosion. And it was, I said it was a really exciting time and there was some new drugs that came out as well for basically preventing if you had an ischemic stroke. So it was a fun time to be all involved in that.
I feel like then things slowed down and now we're in a second wave of that I think. And this is really off topic, you know, good friend of mine at Revolution Medicine for the pancreatic cancer drug that just got approval. It's fun to see colleagues now do really well. Several colleagues in these companies now. It's so exciting to talk to them because a lot of what I would say they're very promising earlier from like five or six years ago of where drug development has been, is starting to see the next wave of next-generation drugs that are much more specific, I wouldn't say accurate, but much more target based. And so it's an exciting time because I feel like we're coming back now into what I started in, because for a while it didn't look as great. We had a huge wave and then it kind of was like, yeah, yeah...
Jon Chee - 00:26:32: We're just getting beat up for a little bit.
Scott Chetham - 00:26:34: Yeah. And now you can see this next generation of drugs and we're in a great spot. We're lucky because we support a lot of large companies. We kind of have an idea of where things are heading and where they go. And I think the next five years is incredibly exciting. Like I think we're going to see a lot of stuff regardless of government that's going to be pretty amazing.
Jon Chee - 00:26:52: Yeah. I feel like for the past couple of years it wasn't fun and I can just feel the fun levels going up again and I'm like, okay, something's happening, something's in the air. And like if you look back on when you were getting your PhD and you're starting to see this rapid change, what do you think contributed to that, like rapid innovation when you were situated in the lab?
Scott Chetham - 00:27:10: Yeah, that's a great question. I mean like all of these things, I don't think it's one thing. I do think a lot of, I would say predecessor technologies had to stack up, but I mean I wouldn't discount, I think some of that first wave was actually the funding environment, particularly medical devices. I think we're in an exciting time and capital became freely available and I think that's what accelerated some of this. I think without that capital at that time, a lot of those medical device-based innovations wouldn't have happened.
I also think we were at the beginning of biotech as the start of biotech where traditional pharma, that was chemistry or chemists, then now we're having to learn biology. So we saw this initial wave and then I think we saw some really early successes. Then I think we kind of burned through the easy things and then I think then it required a lot of other things to unlock. My big takeaway through my career is I always gravitated towards very complicated things. And what it's taught me is that the second something's getting more complicated, you're going in the wrong direction, you should stop and step back. Hard things are hard. But the second you don't have a clear explanation for why something is working and how to unpack that if you can't teach it, then I start to get really worried that this will probably not work out if we can't explain it. And I think that is some of the dangers of LLMs. I mean we understand a lot of how they work. Obviously, the labs will tell you there's still a lot of work to go into, like why do they specifically behave the way they do? But it's a very complicated system. That is the definition of complexity magnified to get that effect. I worry that until we crack that, I wonder how far you can truly push LLMs.
Jon Chee - 00:28:50: Yeah, and we feel the same exact thing where we use a lot of technology at Excedr, AI included. But it's kind of this fine line of not overly relying on it. One of the things I'm kind of saying is, whatever this output is, one, you're responsible for it, and two, you gotta be able to explain it. At least that's for us, how we operate. It's kind of like you can't just take it at face value. You always gotta kick the tires and really unpack it before you decide to lean on that for whatever your next step in whatever you're working on. Because left to its own devices, we found the exact same problem. It's just like, why is this getting more convoluted? It should be like, it's kind of a meta thing and then...
Scott Chetham - 00:29:30: You change the model underneath and you get dramatically different behavior. Yeah. I mean, I would say the big learning curve for us is that two-thirds of our work is QA. Now clean your mark. It's not the development anymore. It's how do you ensure it's doing what you think it's doing and that it will keep doing it, it won't drift. And that's easily two-thirds of the work.
Jon Chee - 00:29:49: We're feeling the same thing. Yeah. It's the amount of QA that we're doing. It just like, really? It's like, is this up to the quality bar that we need it to be? And just knowing when it's off, which I think is probably just an overused term, but it's kind of like your taste or judgment or is this wisdom? This is kind of like, I just know after all this experience of grinding, I just know this thing is off. Like, it's triggering my spidey senses on this given topic, which I spent my whole career on. And you're like... and then you unpack it, you're like, ah, okay, I see. Let's redo this. But I think what's scary is when you don't do that, it's kind of like leverage, really. It has leverage to do amazing things, but also leverage works in the opposite direction.
Scott Chetham - 00:30:31: Yeah, exactly.
Jon Chee - 00:30:32: It just will amplify your losses as well, as much as it will amplify your gains.
Scott Chetham - 00:30:36: That's the trouble in a regulated space is it can produce what looks convincingly good. And that's the problem. So you spend more time yanking it apart. And then as it comes down to having a better understanding about what it's good and bad at and how to use its behavior correctly, I mean, we can see now, and I'm sure it's not just us where some of the limitations of the flagship models today are at. Because we shove against them all the time. But where I think I've been complaining, I think we're seeing degradation in fable already. And the way I think you see it in its language and how it's operating right now, you tune them in one direction to do something really well, which is what I think it's become good at. Really good at coding. But it's lost the ability to communicate effectively.
Jon Chee - 00:31:20: Oh, yeah. Oh, my God.
Scott Chetham - 00:31:21: Yeah. This is why I wonder if we're going to start to see much more specialization in the models rather than... or clever, I would not underestimate how smart the people at Anthropic are. Clever, I would say routing under the hood. So maybe you're going to a sub-model under a model and you don't know it. Different inference, behavior depending on the nature. So I'm not going to underestimate how they'll solve this. But I think we're seeing real butting up against the wall right now.
Jon Chee - 00:31:47: I feel that too. It has to do with code or it's agentic, like coding, it just boom. It's a breeze. But I'm like, why is this five paragraphs of explanation and it's unreadable?
Scott Chetham - 00:31:57: As I said, I fully expect them to solve this, but I don't know if they're going to admit how it'll be some clever piece of engineering that you don't get to see. I doubt it would be done by making a better model. I think that this is one of the problems. The more stuff you put in there, the more you'll get regression to the mean. And I think also the fact that the other risk is, and I know they're buying a lot of content, the more AI content you feed into AI, the more it'll degrade itself as well. And I think that's partly a little bit of what we're seeing.
Jon Chee - 00:32:23: Yeah, I'm on the same page. Are you thinking about doing a custom model for your company where you're tuning it?
Scott Chetham - 00:32:30: We have custom data models now and we do machine learning. Obviously we don't do like flagship. We use the flagship models. That's a very expensive exercise. I mean, we are getting much more selective. So we use all sorts of different models for the right problem. Small models, large models, different versions. We have to keep tracking their behavior and other bits and pieces. But I don't think we need to build a flagship model at this point. But we do have to have what I would call ontologies and data models that are proprietary to us that we use to constrain the systems. Again, I do think Anthropic will solve this and so AI. It'll just be interesting how I hope we get a viewpoint into it because they might not share it, but I don't think it'll be... it's an obvious like if we keep training these things are going to get better. I think we're beating that down now, but I fully expect to jump. You know there'll be other advancements. It's just hard to know from this side of things. Yeah, absolutely.
Jon Chee - 00:33:25: I'm jealous. It sounds like a really unique vantage point to this burgeoning industry and it's starting to hockey stick. I'm a guess and you can correct me if I'm wrong. You're probably like I want to go into industry, I'm seeing where things are going. Or did you have thoughts of staying in academia?
Outro - 00:33:43: That's all for part one of our series with Scott Chetham. Next time, Scott reflects on his transition from academic research into industry, early leadership lessons at Impedimed, and the operational realities of scaling life sciences teams while managing the challenges of founder execution. Join us on Thursday for part two of The Biotech Startups Podcast. The Biotech Startups Podcast is produced by Excedr. Don't want to miss an episode? Search for The Biotech Startups Podcast wherever you get your podcasts and click subscribe. Excedr provides research labs with equipment leases on founder-friendly terms to support paths to exceptional outcomes. To learn more, visit our website, www.excedr.com. On behalf of the team here at Excedr, thanks for listening. The Biotech Startups Podcast provides general insights into the life science sector through the experiences of its guests. The use of information on this podcast or materials linked from the podcast is at the user's own risk. The views expressed by the participants are their own and are not the views of Excedr or sponsors. No reference to any product, service, or company in the podcast is an endorsement by Excedr or its guests.
