Why Crowdsourcing and Open Innovation Can Rule Big Data

Big Data, this is a term that you will hear over and over again for many years to come. In a recent article, we discussed several strategic technologies that would dominate 2013 and our prediction was simple, if Big Data is big in 2012, it’s even bigger in 2013.

Let’s Start Small in Big Data

Possibly you’re questioning the term “Big Data” and wonder what it represents, so let’s start small. You can find definitions through a Google search of course, but I happen to appreciate this Quora submission by David Vellante that states:

“…data that is too large to process and manage using conventional database management technologies. Big data has numerous attributes in addition to its large size, including it is typically unstructured and often dispersed.”

With the emergence of web 3.0 – chip enabled every “thing” – the data we humans are set to collect will dwarf the amazing amount of data we already collect. There are industries that are interested in this data and I think they innately make sense.

Healthcare companies want more real-time data on your wellness.

Apparel companies yearn for more real-time data on your performance.

Retailers want to study emotional responses that trigger consumption.

The list goes on and the hyper-connected future will deliver to these companies streams of real-time data like never before.

Making Sense of Big Data

This is the “Golden Goose” of Big Data. Collecting the data will become easier and easier but understanding 2 important things – How to effectively filter it all & What to do with the data once collected – aka how to monetize the data in some way – will be where the winners in Big Data leave the losers in the electronic dust.

Algorithms will power the Big Data revolution. The beauty of algorithms is, they are never about a final solution. Instead they are focused on the innovation of better, faster, more productive, more efficient and the era of Big Data will be all about “better”. Filtering information better, pairing what seemed to be completely disperse data sets in a unique way that breeds a better understanding of behavior. The study of Big Data is the pursuit of getting better at understanding. Advanced algorithms will power this pursuit.

Open Innovation: Algorithms From All Corners Bring Unique Perspectives

Big Data needs Open Innovation to accelerate the value to the masses. There is no substitute for the outputs – in both quantity and quality of extreme value outcomes – created via Open Innovation competitions. 3 Key factors make this true. (Please Note – open innovation platforms vary greatly in how they share information)

1. Shared Knowledge through Transparency Improves the Individual’s Skill-Set – As Open Innovation community members compete, they gain access to best solutions and techniques. They learn from others and get better.

2. Near-Field Expertise Spurs Innovation in Brand New Fields – Quite often in innovation, it’s not an expert in their exact field of study creating a Eureka! type breakthrough. It’s someone with deep knowledge in a near-field repurposing their knowledge to solve something new OR an expert in a given field repurposing a near-field discovery and bringing it into their realm. Either way, in an Open Innovation community such as TopCoder, this is innately fostered.

3. Different Cultures Bring Different Perspectives – Algorithms solve complex problems by making things “better”. Because algorithms are essentially the idea on how to improve something made actionable through complex mathematics, you can not have enough unique attempts. The more the merrier. Couple volume with brand new ways to evaluate a certain challenge – brought on by a diversity of cultures and backgrounds – and Open Innovation challenges deliver the most potent formula to achieve repeatable success in algorithmic innovation. Remembering the Einstein quote; “If I had an hour to save the world I would spend 59 minutes defining the problem and one minute finding solutions.” A diverse community means scores of members analyzing and defining the problem in unique ways. That breeds unique solutions and in the world of algorithmic innovation, it can be invaluable.

The term Big Data will be supplanted in the technology lexicon for a long time to come. Most companies will not understand how to use this data most effectively to drive value for their business nor be able to source the incredible algorithmic talent to create the solutions. With that said, global platforms and communities that utilize competition and Open Innovation are extremely well positioned to innovate within Big Data. Companies successfully orchestrating these platforms will be best positioned to outperform their competition in Big Data.

Big Data will be a gold mine for some and an impossible mine field to navigate for others. How are you preparing for the era of Big Data?

Don’t miss an article (3,700+) - Subscribe to our RSS feed and join our Innovation Excellence group!

Clinton Bonner (@clintonbon) is a Marketing Manager at TopCoder – the world’s largest competitive community of software developers, algorithmists and digital creatives. Fun Job!!!