Flash for Big Data – be fast AND smart

Last week I had the opportunity to listen to a very interesting return of experience on the use of shared Flash technology for Big Data, and more specifically Hadoop workloads.

It is fascinating (yet not unexpected) to see how Flash has evolved and matured from application acceleration, to general transactional workloads and now to Big Data. The latest does break two pre-conceived ideas about Hadoop workloads (they are better suited to Direct Attached Storage and they run well on dense spinning drives).

The experience today is that not only low latency evidently has a positive impact on those workloads, but the shared model lifts many design complications inherent to the hyper converged approach. Not only storage protection is offloaded to the backend (freeing up CPU cycles from the compute layer), but it can also be grown asymmetrically – which is very consistent to the way data lakes get formed.

Does this mean that we should start deploying All Flash Data lakes ?

Probably not (at least not for now). Although we are seeing a number of options popping up, the reality is that all data does not have the same temperature at all times. Not only that, but we are also seeing more and more different flavours of Flash (TLC, 3D NAND, XPoint), all coming quite rapidly to the market and all yielding increasing performance and cost benefit.

Enter Software Defined Flash and Unified File Object

IBM announced last March a partnership with HortonWorks to support Spectrum Scale to run Hadoop workloads. While not the first time Spectrum Scale is used for Big Data obviously, it is a great demonstration and testimony of what SDS and a Unified File Object can deliver together.

There are 3 key elements I believe are crucial:

  • Flash, any flavour at any time – the Software Defined Approach from Spectrum Scale allows us to accommodate and leverage ongoing and constantly evolving Flash technologies, as well as combining the together.
  • Hot, lukewarm and cold data – it probably does not make sense to store dormant data on Flash, yet it can be complicated to move it out to other media, object or cloud. Spectrum Scale is designed to automatically handle those different media, and make it totally seamless to the application and end-user.
  • Screen Shot 2017-05-02 at 18.15.56In-place analytics – most importantly, because Spectrum Scale is a Unified platform it frees us up from the process of 1/ collecting raw information 2/ transferrring it into the Hadoop environment 3/ exporting it out to publishing. All is created once and no data needs to be transferred or moved.

The key takeaway of all this is that going fast is good, going fast AND smart is what we should aim for.

If you wish to further drill into the topic, I recommend the links below:

Cheers,

Eric.

Advertisements

Craftsmen in the eye of the storm

In 2007, I was offered the opportunity to take a role as an assignee with the IBM Academy of Technology in Somers, NY, six years after I joined IBM fresh out of university. The Academy (or AoT) is a self-governing body within IBM, that focuses on technologies which underpin IBM’s future. As such it regroups IBM’s top technical leaders.

More than any assignment I have taken throughout my 16 years with the company, this experience opened my eyes on how a large corporation like IBM operates, and the central role people – and their actions – play in its success.

As a field person on the front line working with clients, I am generally on the receiving end of strategies and directions. Until 2007, I had been quite detached from the making of these. Yet they are – for a big part – at the origin of change, success or failure.

This all came to light during this tenure, when I was privileged enough to shadow two senior leaders among the many present in Somers.

I had the opportunity to first-hand witness the making of strategies, products and technologies. I was truly fascinated by how the personal eminence, actions and decisions of these two individuals would affect me then, and which continue into today.

It felt like I was in the eye of the cyclone, where the outcome of an hour’s group discussion, would have a lasting and amplified effect on thousands of people and clients all over the globe.

Vision and execution for change and impact.

I took away from these close interactions the ability to envision a future , as well as putting people in motion to instigate and drive change to have impact. It was an experience that set the bar of what a leader and a manager should look like.

After this,  I was offered (on multiple occasions) the opportunity to take on a management role.

When offered such roles, I always connected the dots back to Somers:

  • What is my vision of success for this team?
  • Can I put this group of people – as a team – in motion and have a positive impact?

I actually find the term manager or management role quite misaligned to what the reality is – or should be. One can be a manager and work hard to do things right, to correctly execute the plan crafted in Somers. Or one can choose to not only do things right, but also do the right things – be a leader.

I cannot conceive my role as manager in any other way as a reunion of the two today.

Training and experience taught me how to put a plan in action correctly. But I felt totally intimidated by the energy, eminence and dedication required to envision a future and then to set wheels in motion accordingly – and stand for the outcomes

That is why I turned down management role offers a number of times in the past.

I took on the role of managing and leading the ANZ Systems Storage technical team in September 2016, because I finally had my reason: a vision in mind, a fantastic group of individuals, and a sense of urgency to make an impact.

I went through the same thinking as I was contemplating starting this blog: be more than a spectator of change and start infusing movement for impact – be in the eye of my own storm before time runs out.

Are you a craftsmen of your own storm? A manager or a leader in your own organisation or community? Please share your experience and your view on the journey, your expectations by commenting on this blog post below. It’s my first blog post, so I’d appreciate the support.

Cheers

Eric

Connect with me here or on Twitter or LinkedIn.

Create a free website or blog at WordPress.com.

Up ↑