06 Dec '21

DevOps and DataOps – the products of Generation Z

Think back to the early years of the 21st Century, to a time when we still carried cheque books, sent faxes and played on the Nintendo Wii.  Steve Jobs launched the first iPhone and Instagram, WhatsApp and Snapchat were yet to exist in our vocabularies.  

This first decade of a new millennium also saw the arrival of DevOps, the system which integrates developers and operations teams, speeding up and improving collaboration and productivity.

DevOps solves a problem

The problem was that there existed separately within businesses a development team, who would build the software programs, and an operations team, who would implement these programs. When something went wrong, as it almost invariably did, each team would blame the other.

The solution was to bring the Development and Operations teams together from the outset, and usually in an Agile way, by breaking tasks down into small pieces and delivering something of value every few weeks.  

An incredible amount of data is created each day; globally the figure exceeds a trillion MB. DataOps combines the principles of DevOps and Agile when working with data.

The benefits of DevOps have long been obvious.  The continual testing, monitoring and reviewing means that you can move forward to production with confidence.  Yet, applying these same rules to data has taken rather more time and invoking DataOps still first requires a change of mindset.

It should be obvious that the path to success when delivering a solution that involves vast quantities of data is to start working with small bite-sized chunks. Build a view of a small and manageable quantity of data and begin your data governance and data analysis with this before gradually adding more data to the mix.  

Why does DataOps matter?

DataOps matters because it would be completely unrealistic to try to work with all of your data all at once.  Take a small chunk of data and with it build, test, evaluate. Then repeat the same steps with the next chunk.  Work like this, in an Agile way, and you will see results every few weeks.  

Data Migration is a good example of this approach put into practice.  Planning to migrate all of your data at once at a single point in the future is doomed to fail.  But break that data down into small pieces, and apply the Agile principles, and you will have a far greater chance of success.  

Even if you were to start work today on a Data Migration to be completed in 18 months, what you deliver then will look very different to what you begin with now.  Working in an Agile way to constantly build and deliver the solution in small portions will allow you to respond to the inevitable changes that will happen.  

You may not always have the luxury of being able to work on your data migration as a smooth and gradual process and there may be instances where you are left with no choice other than to implement a hard switchover, but even in this situation you can go through staging and proving it out by running the two systems in parallel for a time.  

Why can’t the same principles also be applied to Business Intelligence?  

They can.

For example, imagine that you want to observe customer behaviours in response to your marketing.  This could be a challenging process, or you could start with a simple plan and expand it to become ever more wide-ranging and complex.   

In a world that focuses on data, your challenge is to understand how your customers use your product and what that product is. That is your vision, and how you achieve that vision may not be a linear route.  

Who should lead a digital transformation?

At DTSQUARED we recommend first setting up a Lean Agile Centre of Excellence (LACE).  This team within your organisation will be responsible for driving the adoption of the new mindset.  

This team should include management as well as those who handle your data on a day-to-day basis and should be representative of all departments required to drive your business forward. Think about the House of Lean:

The ‘roof’ is the promise to deliver value in the shortest possible lead time.  It is supported by four pillars, the first of which reminds us to work together rather than against each other and to maintain respect for people and culture.  Flow is the pillar that encourages us to keep things moving into production quickly.  Innovation encourages us to question constantly our methods and consider if there is a better way to achieve what we want.  Relentless improvement means that we never stop trying to achieve more and do better. Our entire house rests on the foundations of strong leadership.   

With DataOps, and using an Agile approach, you can see change quickly, within 3 to 6 months.  But to embed these changes take longer – typically from 18 months to two years.  

Times are changing – and you should too

The digital age means new rules, new ways of doing business, and new methods of communication. The impact of this digital technology is vast – and the rate of change is fast.  

Because of this speed, business agility will be key to your success.  You can no longer afford to decide on a strategy without being prepared to review it constantly (possibly as frequently as each quarter).  To complete, you must have the capability to adapt and the tools to do this. At DTSQUARED, our team has developed an exclusive tool called Gondola that facilitates the DataOps process by allowing continuous delivery of changes in an enterprise data platform, specifically for our partner platform Snowflake. Keep an eye out for future blogs that will explore this in more detail.

DevOps, and the younger sibling DataOps, are two ‘Generation Z’ers who have a major role to play in your future business success.

Keen to know more about how to implement DevOps into your business?  Get in touch with us at DTSQUARED today to discuss your data challenges and the solutions we can implement to help you get the most out of your data. 

Click Here to get in touch.

Get in touch with our data experts

Get in touch for a free session with our data experts