Your DevOps project is going to fail
…and it could so easily have been saved.
DevOps engineer – job adverts for these positions abound, demanding years of experience in ten or fifteen skills and languages. In the real world the best DevOps engineers are talented coders with a passion for technology, enough skills to integrate with a delivery pipeline and install a specified software stack.
So, what’s the problem?
Well, not all the software in that stack is created equally. If you want to deploy a new version of an app it’s simple – stop the app, overwrite it with the new version and off you go! Need to deploy a new version of a middleware component? No worries! This is agile, we’ll take each node out of the cluster in turn, deploy the new software, add the config and back into the cluster it goes! Database change? Simple! Stop the database, overwrite it… err, oh, no more data. Whoops!
This is hyperbole of course. Everyone knows you don’t replace the database for every schema update, you just run an ALTER command on the table – databases have become much more capable of handling schema changes online. Oh – but is that ALTER a reorg recommended operation? Nobody knows? How many reorg recommended operations have run against that table? Not counting huh. Yeah, it’s strange how the database throws errors occasionally after a deployment isn’t it…?
Never mind, let’s have a look at some other tables. No not that one with the comically long 150-character name, we don’t want to have to be typing all that in – not that one in camelCase and special characters that will need escaping and wrapping in quotes every time they are referenced in code either. Could have done with some naming standards here… those choices are going to make looking after this system a pain for years.
OK let’s look at this one. Oh, there are only three columns: ID, key and value. That should have been normalised, yes, it’s more effort to define the schema properly but on the other hand the database would only have to read one row to return the data instead of the fifty it has read and sort now. At least extra hardware is cheap, right? And licenses for all that hardware is an operational cost, not development. No worries.
Right, here’s a table with more columns. A lot more. I guess someone thought it was easier to keep adding a column rather than a new table. Bit of basic design work required there, at least we have indexes – one for each column is quite a lot though. And each one contains most of the columns from the table, that’s quite some overhead. You can expect any data changes in there to take a long time – and as for maintaining all those indexes, well, I hope there are some long housekeeping outage windows available.
While we’re looking, I’m not sure BIGINT was the right datatype for the order quantity – people aren’t likely to order 9223372036854775807 widgets in one go so you could have saved a few KB. Yes just a few kilobytes, times hundreds of millions of rows, times all the backups and logs, plus the read and write overhead and memory and CPU to sort. Looks like we’re going to need a bigger VM for production, lucky that’s easy to do in the cloud. Shame about the cost.
I could go on, but, I know, I know – these aren’t the interesting things about development. Creating and maintaining a naming standard in your database isn’t sexy, choosing a data type is a tiny step to delivering the new features to the users and deciding which table a column goes on isn’t the sort of problem that the creative minds of developers are interested in solving. But – I bet you have standards for naming functions in code. I bet you keep the code as simple, streamlined and efficient as possible. I bet you ensure the code is documented. A lot of organisations have a standard style for indenting code even if it’s not strictly required. Why? Professional pride in some cases no doubt, given for a small system you might not notice. For an enterprise class system you will though. Try developing new features when the existing millions of lines of code is a mishmash of styles and techniques with no documentation and seemingly random naming, try quickly identifying and fixing an error or performance issue.
Increased development costs, development time and bugs. Increased support costs, resource requirements, issues causing user-infuriating performance and downtime. Technical debt piling up as fast as you can develop. These are some symptoms of a project that is not working properly… and it could be the code, or it could be something that sits behind all the code – the database. Which one has all the standards, quality reviews and is built by experts in that particular technology?
So, what’s the point to this sad tale?
DevOps engineers will implement a pipeline and deliver the functionality asked of them. The good ones will write clean, standards compliant, secure, performant and easy to maintain code – because that is their area of expertise and experience. Why does the database, a crucial component of so many systems miss out on some expert care?
Have a DBA help set up your environment. Define some standards the developers can follow. Suggest design solutions that utilise the features of the database you are already paying for. Review the changes to catch issues earlier and ensure the solution is going to perform when the system really gets busy. Once you’re set up and running, with the correct processes in place for the developers to follow, they don’t even need to be full time to save your project. The modern-day database administrators have moved on from the always says no, taking weeks to put anything live, grumpy dinosaur attitudes of the past to embrace DevOps, automated build and test pipelines and the opportunities new development methodologies present. The good ones will ensure you have a clean, standards compliant, secure, performant and easy to maintain database. Sound familiar?
Contact our team of experts for more information on DevOps.
About: James Cockayne – Principal Consultant
Favourite topics – Database security, performance & availability. And Guinness.
Mostly Guinness.
In over 15 years of experience as a DBA James has worked across government, retail, insurance and banking industries designing, building and maintaining databases that sit behind some of the UKs busiest online systems and the warehouses that support them.
Now working for UK based Triton Consulting, James focuses on extending the use of DevOps and cloud tooling and techniques with DB2 both on mainframe and LUW platforms.
View James’ blogs and tech tips.