Design Lineage, Data Lineage, Semantic Lineage

[MUSIC PLAYING] [WHISTLING] Hey, Danny. How you doing. I'm sorry for calling unannounced again. Again, I'm doing some research, and I'm hoping you might be able to help me out. I'm doing this particular topic. I'm hoping you might be able to point me in the right direction. I'm going to call it out to you. And you-- I'm just going to read it. It's a good thing I'm a good dancer. Carry on. Design lineage, data lineage, and semantic lineage. And because there was lineage in the title and semantic lineage I was like, Danny would probably dig me out of a hole here. Any thoughts on that? Oh, lots of parts. All very, very important topics, things that are super important, not just for-- I should stop you, by the way. It's about database migrations, so we can narrow it right down. Yeah, yeah. OK well, but it's not just an important topic for database migrations. It's a very important topic for database migrations, but the real important topic is your organization's trust and faith in the data that you have. And the last thing you want from a database migration is to take data that was trusted by your business and make some mistakes, miss something, and have it turn into, let's say, high performing data that's not trusted by your business. Because again, it doesn't really matter how well it's performing. If they're not using it, then you're probably wasting time and money. So a couple of things there. Design lineage. Design lineage generally talks about the business requirement moves to actual deployed database. And it's very important to understand that because if what you're deploying is drifting away from what you understand of what the business requires, then again, you're building problems in and you're creating things to fix. You're creating problems for a system that probably doesn't need it. So understanding that sort of-- and I'll use fairly common data modeling terms here-- logical to physical transition. Logical being what is it that this data should be doing for the business, and how should we be managing it to accomplish that. And making sure that you're not losing that essence just because it's easier to do something this way in Oracle or easier to do something that way in SQL Server. And then again, take it another step removed where you've deployed it in Oracle, but you're now deploying it in Postgres. Losing that essence is a death knell for trust and data and a successful migration because now you're going to have to go back and do a lot of rework. Rework is always much more expensive. So you want to definitely make sure that you can stay true back to the original design. If that design needs to change as part of the migration or something's happened in the business, then you want to build on that. I thought that actually meant about design, documenting design decisions. So you're actually-- Well. Now you're taking in a different direction. I mean you should be modeling. Well, but your direction is not incorrect as well, because yes, over time a database will change based on changing requirements, new requirements, all sorts of different things that are happening in the dynamic business world. So you absolutely-- part of a design lineage would be managing the life cycle of changes to that database over time. Because again, it may not be fully formed or it may have changed. And again, data modeling is going to be very, very helpful for that. If you've been using your data modeling tool correctly, every new requirement, every new change to the database through its life cycle should be captured and deployed through the model. Which means now I can capture in the different versions of document with those versions what changes were made, when they were made, by who, and why. And that's the key piece of the data modeling process. So having that codified, captured so that if Charlie, Sue or Billy Bob have moved on to their next opportunity, that tribal knowledge isn't lost. So yes, design lineage goes both from a business to technical implementation, so logical to physical. It also talks about how that design has changed over time, what the impact of that was and why. And all of that is very, very well handled, documented, captured and automated, made much easier to use, analyze and understand in a data modeling tool. You, are you talking about the happy path. Let me qualify this by, are you talking about in an ideal world, people do this. It's fully documented within the data model. But my question to you is, what if an organization, not saying any organization does this, what if an organization was super lazy and just did the bare minimum to deploy, to release an application. The bare minimum. And they don't exactly have this, but now they're migrating. Like, do they have any recourse? Do they have to do this? Do they have to document everything now or is there any way around it? Well, yeah, in this case, unless you've got some other change management system or some other requirements management system that you're using. And generally, if they're not using a data modeling tool, they're probably not using good tools in those other areas. They all work very well together. So yeah, you pretty much lost your design lineage at that point, if you haven't been doing this type of work. And great example, massive, massive financial institution that everybody would know refuses to allow data to be deployed anywhere in their organization without a data model. And it's for that very reason many, many, many, many, many years and many, many, many wasted dollars just trying to keep things on track, because they weren't doing the hard work up front, and they were paying three times, 10 times, a hundred times as much downstream to do the rework.