
50 segments available
Welcome to the Engineering Room, wide-ranging conversations with thought-leaders from the software engineering industry. This is a new mini-series of additional content on the Continuous Delivery Channel. In this first episode, Dave Farley chats with Martin Fowler. Martin is a widely read author having written definitive works on several important topics, including Refactoring, NoSQL, UML, Extreme Programming, and several books on patterns. He also has a very widely read website that captures more of these thoughts, and more collections of patterns too at ➡️ https://martinfowler.com Dave and Martin discuss a wide range of ideas, from new work in patterns in distributed systems and Data Mesh, to the fundamental principles of software development that matter, whatever the technology or problem that you are solving. ------------------------------------------------------------------------------------- 👕 T-SHIRTS Do you like the t-shirts that I wear in my videos? YOU CAN SAVE £2 ON EACH ORDER from Qwertee via this link ➡️ https://bit.ly/3yUwvYC. This Discount is only available for Continuous Delivery viewers! 🚨 DON'T FORGET TO USE OUR DISCOUNT CODE AT CHECKOUT: ContinuousDelivery __________________________________________________________ 📚 BOOKS: 🚨 MY NEW BOOK! 👉 📖 "Modern Software Engineering" is now available on Amazon ➡️ https://amzn.to/3DwdwT3 In this book, Dave brings together his ideas and proven techniques to describe a durable, coherent and foundational approach to effective software development, for programmers, managers and technical leads, at all levels of experience. 📖 "Continuous Delivery Pipelines" by Dave Farley paperback ➡️ https://amzn.to/3gIULlA ebook version ➡️ https://leanpub.com/cd-pipelines 📖 The original, award-winning "Continuous Delivery" book by Dave Farley and Jez Humble ➡️ https://amzn.to/2WxRYmx NOTE: If you click on one of the Amazon Affiliate links and buy the book, Continuous Delivery Ltd. will get a small fee for the recommendation with NO increase in cost to you. ------------------------------------------------------------------------------------- Also from Dave: 🎓 CD TRAINING COURSES If you want to learn Continuous Delivery and DevOps skills, check out Dave Farley's courses ➡️ https://bit.ly/DFTraining 📧 JOIN CD MAIL LIST 📧 Keep up to date with the latest discussions, free "How To..." guides, events, online courses and exclusive offers. ➡️ https://bit.ly/MailListCD ------------------------------------------------------------------------------------- ⭐ SPONSORS: This series is brought to you by Equal Experts. Equal Experts is a product software development consultancy with a network of over 1,000 experienced technology consultants globally. They increase the pace of innovation by using modern software engineering practices that embrace Continuous Delivery, Security, and Operability from the outset ➡️ https://bit.ly/3ASy8n0 ------------------------------------------------------------------------------------- VIDEO CHAPTERS Intro to Martin Fowler - (00:00:00) Patterns in Distributed Systems - (00:04:12) “Data Mesh” Dealing with Data at Large Scale - (00:18:48) How to Replace Legacy Systems - (00:34:09) Addressing the Problems of the “Critical Aggregator Pattern” - (00:41:20) Collecting Patterns or Offering Advice? - (00:47:45) Self Testing Code - TDD As Fundamental Principle - (00:56:45) Accelerate Books & DORA Metrics - (01:07:02) Measurement - Sociology isn’t Physics - (01:10:14) Software as Tracer-Bullets - (01:12:25) Hardware & Safety Critical Systems - (01:14:38) Wrap Up - (01:18:20)
Dave Farley introduces the Engineering Room, a new mini-series featuring influential figures in software engineering. He expresses gratitude to viewers for their support and invites them to subscribe for more content. This segment sets the stage for the insightful discussions to follow.
"hi my name is dave farley of continuous delivery and welcome to the engineering room a series of conversations with influential people from our industry this episode is a little different to the usual..."
Dave Farley introduces Martin Fowler, a prominent figure in software development and one of the original creators of the Agile Manifesto. He highlights Martin's influential works and his extensive knowledge in software patterns, setting the context for their conversation.
"if you'd like to see more content like this please do subscribe and let us know your thoughts in the comments below today i'm talking with a friend of mine who has certainly significantly influenced m..."
Martin Fowler discusses his shift from writing to mentoring others in the software industry. He shares insights into his current projects, emphasizing the importance of helping others articulate their ideas and navigate the complexities of software development.
"to track what's on martin's horizon it's a way of crystallizing ideas uh that so that they resonate with people if you've ever refactored your code use dependency injection or created a dsl then you'r..."
Martin elaborates on a project focused on patterns in distributed systems, initiated by his colleague Unmesh Joshi. He explains the necessity for software engineers to understand the underlying principles of distributed systems like Kafka and Cassandra to effectively utilize and troubleshoot them.
"and others in their next steps you've certainly helped me and were kind enough to review the early chapters of my new book even though you were overloaded with work at the time um and you mentioned to..."
Dave and Martin discuss the complexities of distributed computing, likening it to quantum mechanics. They emphasize the importance of grasping fundamental principles to manage the inherent complexities that arise when software operates across multiple computers.
"and i mean it started out really because he felt that he that our folks at footworks needed a good grounding in what's going on inside distributed systems that we use all the time kafka cassandra all ..."
Martin shares insights on training new engineers, particularly those fresh out of college, to better understand distributed systems. He highlights the need for foundational knowledge to navigate the complexities of modern software development.
"performance or the way that things are operating yeah so you start go ahead yep so sorry i just i i didn't mean to interrupt you but i was i was just going to say i i think there are some um i i've ch..."
Martin discusses the development of patterns for distributed systems, detailing how they help explain various solutions to common problems. He emphasizes the importance of understanding trade-offs and context when applying these patterns.
"this so he started setting up some training work primarily based in in our india operation which is pretty sizable these days um and he contacted me and we developed the idea of trying to pull this fi..."
The conversation shifts to the complexities of distributed consensus protocols like Paxos and Raft. Martin explains the challenges of maintaining consistency and reliability in distributed systems, highlighting the importance of understanding these protocols.
"yeah so he's been publishing those on my website over the course of the last year year and a half perhaps but we've been putting them out we've just got another batch that's going into um some copy ed..."
Martin advises on the importance of avoiding distributed systems when possible, as they introduce significant complexity. He discusses the implications of concurrency and the challenges that arise when building on distributed architectures.
"very long time ago and to some degree i think that the technologies have made the problems more difficult because it used to be harder to do it it used to be harder to get you to begin to start to rem..."
Dave shares his experiences consulting for a client using a non-SQL data store lacking transactional integrity. He expresses concerns about the unpredictability of such systems and the importance of understanding concurrency management.
"so please because it's going to make things so much better but there are times you cannot or you know you're building on top of something that's got this distributed substrate on it and it's going to ..."
Martin discusses the complexities of managing system transactions, especially in distributed environments. He emphasizes the need for understanding the differences between business transactions and system transactions to avoid issues.
"consultancy for a client who i shot sharp name which is a very large development and they were using a non-sql data store that didn't have any transactional integrity and so i you know i kind of looke..."
The conversation delves into the role of leader nodes in distributed systems and the challenges that arise when a leader fails. Martin explains how consensus protocols help maintain system integrity in such scenarios.
"time in order for people to do their work but at the same time resolve them against the system transactions that need to be opened for a short time because you don't hold a system transaction open for..."
Martin reflects on the importance of understanding how distributed systems operate under the hood. He emphasizes the need for engineers to have a foundational sympathy for the technologies they work with to effectively troubleshoot and design systems.
"yeah i mean that's that's part of that thing and so it's just been very interesting to dig into that and i'm glad it's somebody else who's going through all this source code to figure out how these op..."
Dave and Martin discuss the challenges of creating effective examples for explaining complex concepts in software development. They highlight the balance between realism and clarity in educational materials.
"that the difference between it's no good just showing somebody an enterprise system because it's too messy and complicated to be able to see the wood for the trees and so being able to synthesize exam..."
Martin concludes by discussing the significance of context when applying patterns in software architecture. He emphasizes the need for engineers to understand the broader implications of their design choices.
"unmatches finding this as well is trying to get the right thing that will show what's going on and not be overwhelmingly complicated it's a tricky balance to to grab and it takes, a lot, of, time, to,..."
In this segment, Fowler discusses the complexities of consensus algorithms like Paxos and Raft, which are crucial for distributed systems. He shares insights into the challenges faced while working through these algorithms and the ongoing development of his book, which aims to clarify these concepts for readers.
"sympathy as to how it's operating under the hood to at least some degree it's it sounds interesting so so is it is it is that a book that's kind of near incompletion or is it is it just starting out y..."
Fowler delves into the concept of eventual consistency and its implications for system design. He discusses the balance between safety and liveness in distributed systems, emphasizing the need for systems to remain operational even when parts of them fail.
"water marks low water marks and things of that kind that there's a good chunk of material that that's there on the site at the moment cool that's i don't know what one of the other on where, you, can,..."
Fowler introduces the concept of data mesh, a decentralized approach to managing analytic data across large organizations. He contrasts this with traditional centralized data lakes, arguing that a decentralized model allows for better management of complexity and more accurate data representation.
"if you're going to get the kind of through button again it's the classic safety versus liveness trade-off yes i can make a perfectly safe system it just won't do anything i need it to be alive and so ..."
In this segment, Fowler elaborates on the idea of treating analytic data feeds as products managed by the domains that create them. He discusses the importance of documentation and product thinking in ensuring that data products are useful and accessible for analytics.
"practice cool and of course the in practice is kind of important because you do certainly run into some things that are talked about a lot in theory but not used in practice because there are practica..."
Fowler emphasizes the challenges of managing complexity in analytic data within large organizations. He discusses the need for a decentralized approach to analytics that allows teams to work with their own data products while maintaining clarity and understanding of the data's origins.
"known and similarly these distributed patterns they're not new they're in the open source products that we all use but they're not widely enough understood and so therefore need to be disseminated mor..."
Fowler critiques the traditional centralized approach to data management, arguing that it often leads to confusion and inefficiency. He advocates for a decentralized model that empowers individual teams to manage their data products, fostering better collaboration and understanding.
"all into one big data storage data lake storage approach and use centralized tools to understand that data and then discriminate it out to its users yeah and what jamaica said was it and the rest of m..."
Fowler discusses the difficulties organizations face in achieving a unified definition of data, particularly in large enterprises. He highlights the natural differences in how various departments view data and the importance of acknowledging and managing these complexities.
"to pull together a new analytics um a new analysis of what's going on in the organization my job is instead of going to this centralized warehouse or data lake that has everything i ever want except i..."
Fowler draws parallels between agile methodologies and the management of analytic data complexity. He emphasizes the need for flexibility and adaptability in data management, advocating for a mindset that accepts uncertainty and focuses on managing complexity effectively.
"find you can't they want to have this single view of what a customer is a single definition and you're smiling because you've seen it too right yeah different parts of the organization they will look ..."
Fowler concludes by discussing the pitfalls of centralized data approaches in large organizations. He stresses the importance of developing decentralized strategies that can better accommodate the complexities of analytic data, ensuring that organizations can effectively leverage their data assets.
"people are doing it many people are going with a centralized approach so we're having to figure out what the tools are figure out the governance structures figure out how to make this work and that's ..."
Martin Fowler discusses the shift from a centralized data lake approach to a decentralized model where individual business units publish their data as products. This segment emphasizes the importance of understanding data origins and organization, highlighting the challenges of managing data in large enterprises.
"how where do i go about to do that so in a centralized approach there would somewhere be a centralized data lake that would have all of this information of customer and operation stuff yeah you know a..."
Fowler elaborates on the complexities of managing analytic data in large organizations, stressing that understanding data is not just about its current state but also about historical context. He introduces the concept of temporal data, which reflects changes in both data and understanding over time.
"yeah but the point is the management you're not getting this centralized management approach yes consolidation so and how they publish that whether in terms of feeds of events or tables of some kind o..."
In this segment, Fowler shares insights on the upcoming O'Reilly book about data mesh, which aims to redefine how organizations manage analytic data. He discusses the importance of applying domain-driven design principles to data management, emphasizing the need for a strategic approach in large enterprises.
"you're talking about large enterprise things trying to do anything involving centralizing data yeah too often being a fool's errand and that's doubly true of analytic data because you've got a lot of ..."
Fowler addresses the challenges of managing complexity in data structures within organizations. He critiques the idea of a single data structure to rule them all, advocating for a more nuanced approach that considers multiple bounded contexts, as inspired by Eric Evans's work in domain-driven design.
"actually going to be a an o'reilly book hopefully out early next year um and there are i think some pre the the o'reilly preview system yeah beginning to work on it as well jemack has written a couple..."
Fowler discusses the complexities of integration in microservices architectures, highlighting the importance of understanding the distinct contexts of service interactions. He emphasizes that successful integration is a critical competitive advantage for organizations, often overlooked in architectural discussions.
"hold every possible piece of data um or it gets horribly abstract and just becomes you know record related to record and you know nowhere that way um so you're always trapped when you deal with these ..."
In this segment, Fowler introduces the 'Critical Aggregator' pattern, which highlights the challenges of legacy systems that aggregate critical information for decision-making. He discusses the risks of tightly coupled systems and the need for better interfaces to facilitate data flow and system replacement.
"i had um i had i had an interesting conversation with eric a few weeks ago we were talking we were talking about something else we were talking about micro services um architectures and he made an opt..."
Fowler presents the 'Divert the Flow' pattern, advocating for the replacement of critical aggregators in legacy systems as a first step in modernization. He explains how rebuilding these aggregators with improved interfaces can ease the transition and reduce the risks associated with legacy system dependencies.
"competitive advantage is if you're able to do integration better than your competitors um it can really give you a noticeable edge and so i i was very uh taken with uh what he's saying in that article..."
Fowler discusses an upcoming book project focused on patterns for effectively replacing legacy systems. He emphasizes the need for gradual processes and transitional software to manage the complexities of legacy displacement, aiming to provide a comprehensive understanding of this critical aspect of software development.
"major part of our work and yet there's not been very much written about it and not much understanding of how do we think about legacy replacements in an effective way and so what we're trying to do wi..."
In this segment, Fowler critiques the common practice of aiming for feature parity when replacing legacy systems. He warns against the pitfalls of creating legacy mimics that complicate integration and stresses the importance of understanding the context in which feature parity can be beneficial.
"you know you don't displace a legacy system all in one go you do it over time it may involve creating transitional pieces of software that you know you're going to throw away yeah in a year or two's t..."
Martin Fowler discusses the 'Divert the Flow' pattern, which suggests that when dealing with legacy systems, the first step should often be to replace the critical aggregator. This approach allows for better interfaces and safer replacements of other system components, ultimately leading to improved decision-making based on aggregated data.
"that's currently under in the review process there's a pattern that's called um divert the flow um and this is a very interesting pattern that the core heart of it is actually in another pattern there..."
Fowler emphasizes the importance of understanding the trade-offs involved in legacy system replacements. He highlights that many legacy systems were built for good reasons, and recognizing the historical context can lead to better decision-making in modern software development.
"that we that they're referring to is divert the flow and the the the divert the flow is a is a metaphor of saying but if you if you want to replace um a dam the first thing you want to do is divert th..."
In this segment, Fowler introduces the concept of 'compassionate coding,' advocating for empathy towards developers who made past decisions. He argues that understanding the context of legacy systems can help in navigating the complexities of modern software development.
"involved because the trade-offs are often not straightforward because people do come into this in the sense of oh let's just go for feature parity and move it to the cloud yeah and we go no that's pro..."
Fowler contrasts feature branching with continuous integration, explaining how the context of a development team affects the effectiveness of these strategies. He discusses the importance of integration frequency and how shifting mental models can lead to healthier codebases.
"getting the chance to say well what do you mean by this how is this working out really interesting conversations i i i think that's one of my skills is that i i i need to dumb things down to understan..."
Fowler concludes by discussing the significance of applying software patterns in the right context. He stresses that what works in open-source environments may not be suitable for private teams, and understanding these nuances is crucial for effective software architecture.
"switch my mental node from saying i don't integrate when my features complete i integrate when i've got a stable um build that i can integrate with and i want to get that stable build all the time yea..."
In this segment, Fowler introduces the concept of self-testing code as a fundamental principle in software development. He distinguishes it from test-driven development (TDD), emphasizing that the goal is to create code that can verify its own correctness. This principle is applicable across various programming languages and environments, highlighting its significance in achieving reliable software.
"deeper insights too in terms of what are the durable ideas and it seems to me that often we we as software developers get a bit obsessed by things that are really a bit more off or more ephemeral than..."
Fowler explains the 'humble object' principle, which advocates for isolating complex behaviors into simpler, testable components. This approach is particularly useful in challenging environments like UI technology and distributed systems. By creating simple gateway objects, developers can maintain testability and ensure that their systems remain robust and reliable.
"tomorrow's legacy system today um well i mean that's yeah that's what my whole writing is about right it's trying to identify those um i mean some of them are quite broad and some of them quite narrow..."
Fowler discusses the critical role of good interfaces in software design, linking it to the practice of TDD. He argues that effective interfaces lead to clearer, more maintainable code. This segment highlights how TDD encourages developers to think critically about interfaces, ultimately leading to better software architecture and design.
"um if you understand the importance of sales testing code whenever you go to a new environment the first thing you want to do is figure out how can i get my self-testing set up going how do i get that..."
In this segment, Fowler reflects on the value of test-driven development (TDD) as a practice that enhances software quality. He shares personal experiences demonstrating how TDD influences design and encourages developers to create better interfaces. Fowler argues that TDD is essential for effective software development, advocating for its widespread adoption in teams.
"yeah absolutely i mean i mean the the the idea of i tend to use the term tdd but self-testing code is i think much it has deeper implications than people who don't practice it realize very often i thi..."
Fowler expresses his support for DORA metrics, which correlate software development practices with business outcomes. He appreciates the focus on measuring efficiency and quality through stability and throughput. This segment discusses the importance of aligning technical practices with business performance, emphasizing the value of data-driven insights in software development.
"for a minute um i i i i i went i went through an exercise i wanted to demonstrate some some unpleasant code i wanted to demonstrate some some problems in in the code and so i started writing this exam..."
Fowler addresses the difficulties in measuring software productivity and output. He critiques traditional metrics like function points, advocating for a focus on delivering value to users instead. This segment highlights the challenges faced by the software industry in quantifying success and the need for more meaningful measures that reflect true productivity.
"looking at the screen saying is that okay and also i have the perfect regression test suite because i just build the entire website and diff it against one that i know is good which is a very crude re..."
Fowler discusses the evolution of agile practices and their adoption in the software industry. He reflects on the progress made in integrating technical practices like TDD and continuous integration into agile methodologies. This segment emphasizes the importance of maintaining a balance between agile principles and technical excellence to achieve effective software development.
"um but um but as i said i mean these things do take a long time to to work through particularly when we're in a profession that can't really measure our output and productivity effectively and when yo..."
Martin Fowler discusses the correlation between business performance and software practices, referencing the DORA report and its significance. He emphasizes the importance of understanding how technical techniques can lead to improved business outcomes, and shares his journey of exploring these concepts with Nicole Forsgren, leading to the influential book 'Accelerate'. This segment highlights the value of data-driven insights in software development.
"the business level with some of the software practices that they do then we feel we've got something that has a real value yeah and initially when jazz talked to me about it i mean when my very first ..."
In this segment, Fowler contrasts the precision of physics with the complexities of sociology in software engineering. He explains that while software development often attracts technically-minded individuals, the discipline requires an understanding of sociological principles. Fowler discusses the challenges of applying rigorous scientific methods to software practices and the importance of feedback loops in improving outcomes.
"them yeah and profoundly so i i i think one of the things that trips us up as an industry very often is that i think our discipline is one that probably appeals to people with a technical mindset i i ..."
Fowler uses the analogy of balancing a broom to illustrate the significance of feedback in software development. He argues that rapid feedback allows teams to adapt and optimize their processes effectively. This segment also references the concept of 'tracer bullets' from the Pragmatic Programmer, emphasizing the value of iterative testing and learning in software engineering.
"absolutely um that's always been something that we've been uh big fans of course you know do small things set up feedback loops um operate from there i i came up with a nice analogy recently i i well ..."
Fowler discusses the challenges of applying continuous delivery principles in safety-critical software systems, such as those used in space probes. He emphasizes the need for software to be always releasable, even if not immediately deployed. This segment highlights the importance of simulation and testing environments in ensuring software reliability and the role of feedback in refining development practices.
"under those circumstances and you have something different it's it's that's one of, the, uh, that's, one, of the, areas, where i i think i i i slightly disagree in that i think that working so that yo..."
In the concluding segment, Fowler reflects on the principles of software engineering discussed throughout the conversation. He expresses gratitude for the insights shared and encourages viewers to engage with the content. This segment serves as a wrap-up of the key themes explored in the episode, reinforcing the importance of continuous improvement and feedback in software development.
"yeah i i saw i saw a great example recently in a tesla have apparently um upped the charging rate for their model 3 cars and they did that in i think it was three hours so so the charging the maximum ..."