Blogs

Enabling the Internet-of-Everything with intelligent data distribution

When I speak with customers, a common goal is to achieve reliable, efficient, and scalable real-time distribution of data among people, devices, sensors, and machines – what I like to call the Internet of “Everything” (IoE).

The primary challenge of IoE is the “I.” The Internet is a congested and often unreliable transport mechanism for time and business-critical data delivery.

Yet we have 1 trillion devices exchanging 2.5 billion gigabytes per day (Source IBM). This deluge of data exchanged must be managed in the most efficient manner possible, in order to optimise delivery with respect to reliability and speed – a challenge that is not easily solved, says Andrea Skov, CMO at Push Technology.

The problem is that many data communication technologies are messaging systems that blindly send large amounts of data back and forth — an inefficient and expensive approach to data transmission. The specific demands of IoE preclude effective use of generalist data transmission technology solutions that work adequately for less demanding operational environments such as chat or social media.

In addition, a key problem posed by the IoE is that network power remains very centralised. Even in the era of the cloud, applications are communicating with relatively few massive datacentres that may not be located close to you. That is fine when you’re not accessing a lot of data and when latency isn’t a problem.

But it doesn’t work, for example, when you’re monitoring thousands of stock prices across the markets. In that example, if your request for price updates or order to buy or sell a stock is delayed, then millions of dollars can be lost and the opportunity is non-recoverable.

From my perspective, there are three core issues which must be addressed to solve the data distribution dilemma. The first is latency. Information becomes irrelevant if the data does not get to the user fast enough. Ten years ago companies were happy with 12 second delays. Today, time delays kill revenue opportunities. Seconds are no good and data has to be sent and received in milliseconds.

The second is efficiency i.e. “one pipe in and one out.” The magic calculation is how many gigabits of data per second can get through the pipe. If messages are smaller, you can either transmit more messages or hold more connections.

The third is scale – remember, one trillion devices are connected and in use today. The challenge is being able to scale over a busy and sometimes unreliable network – that’s the “I” again. Many companies try to shoehorn inefficient Enterprise messaging technology into their software, or use open source and try to build their own.

Unfortunately, organisations are trying to solve scalability issues using traditional techniques and solutions which weren’t built for the IoE world. These technologies don’t scale.

The IoE market is maturing and with that comes a realisation that network-efficient, high volume data streaming and messaging is critical for corporate applications and analytics. To put it bluntly, companies using IoE devices need solutions which increase reliability, and reduce bandwidth and infrastructure requirements.

This equates to intelligent data distribution and management, in conjunction with an architecture that has been designed to put the data as close to the end user as possible – whether that is a machine, device or person.

Also, to lighten the load on the network, companies need to understand their data. By understanding the data, they can then apply intelligence and only distribute what’s relevant or what has changed. This means only smaller pieces of data need to be sent across congested networks.

Faced with this technology challenge, Push Technology developed Diffusion™, a patented, bi-directional, delta-data streaming platform – available on-premise, in-the-cloud, or in hybrid configurations to suit companies’ business operations and regulatory requirements.

Diffusion is “data aware which means it intelligently and automatically handles data transmission and removes out-of-date and/or redundant data delivering up to 90% data optimisation across the Internet. This translates to a substantial reduction in bandwidth and infrastructure requirements, and minimal latency for data transport.

Push Technology cut its teeth in sectors where real-time data delivery is critical to execute time-sensitive and business-critical financial transactions. eGaming and financial services were early adopters of the company’s technology.

Today, Diffusion processes £5.2 Billion(€5.88 billion) in bets annually for leading gaming industry brands, a huge volume of trading data for financial institutions and is effectively servicing applications across many markets including healthcare, retail, transportation, and energy.

The author of this blog is Andrea Skov, CMO at Push Technology

Comment on this article below or via Twitter @IoTGN