CASE STUDY :NETFLIX AMAZON KINESIS

We all know that Netflix is one of the world’s leading internet television network, with more than 100 million members in more than 190 countries enjoying 125 million hours of TV shows and movies each day.Due to such high traffic results in an extremely complex and dynamic networking environment where applications are constantly communicating inside AWS and across the Internet.Netflix needed a solution for ingesting, augmenting, and analyzing the multiple terabytes of data its network generates daily in the form of virtual private cloud (VPC) flow logs.Thus,Netflix set out to establish a new data source that could give it more insight into communication among applications and regions by combining VPC flow logs with application metadata.

From the outset, AWS enabled Netflix to experiment with different approaches to analyzing its network data.They experimented with multiple designs and used many AWS products .The solution Netflix ultimately deployed — known internally as Dredge — centralizes flow logs using Amazon Kinesis Data Streams.The application reads the data from Amazon Kinesis Data Streams in real time and enriches IP addresses with application metadata to provide a full picture of the networking environment.

Dredge joins the flow logs with application metadata as it streams and indexes it without using a database, which eliminates a lot of the complexity.The enriched data lands in an open-source analytics application called Druid. Netflix uses the OLAP querying functionality of Druid to quickly slice data into regions, availability zones, and time windows to visualize it and gain insight into how the network is behaving and performing.It would have been daunting to publish, stream, and consume that much information from an external system such as Kafka.It took just a few API calls to centralize multiple terabytes of flow logs into Amazon Kinesis Data Streams. Now we can focus on getting insights from the data rather than simply getting access to it.The scalability of Amazon Kinesis Data Streams was a good fit for the Dredge application because of the cyclical and elastic nature of network usage at Netflix.

Netflix’s Amazon Kinesis Data Streams-based solution has proven to be highly scalable, each day processing billions of traffic flows. Typically, about 1,000 Amazon Kinesis shards work in parallel to process the data stream.Amazon Kinesis Data Streams processes multiple terabytes of log data each day, yet events show up in our analytics in seconds.We can discover and respond to issues in real time, ensuring high availability and a great customer experience.With the help of Amazon Kinesis which enables them to identify ways to increase efficiency, reduce costs, and improve resiliency for the best customer experience,Netflix is now able to identify new ways to optimize its applications, whether that means moving an application from one region to another or changing to a more appropriate network protocol for a specific type of traffic.“Netflix is heavily invested in AWS in part because it abstracts the underlying network, so we don’t have to deal with switches and routers