Netflix is hoping to open source a system it built to
orchestrate the machine learning workflows that improve recommendations
to users on what to watch next.
The streaming giant first revealed the Meson orchestration and scheduling framework in a blog post, but used last week’s Spark Summit in San Francisco to flesh out the project.
“Netflix has one primary goal - we want to delight every single member every single time you interact with Netflix, and the delight really means we want to be able to help you find content that you will love and enjoy every time you interact with the service,” senior software engineer Kedar Sadekar told the summit.
“Happy customers will play more [content] and watch more. The driving force behind our efforts is really the recommendations or the machine learning pipelines that drive the recommendations."
The position of each movie or TV show tile on a viewer’s home screen is determined by these algorithms, he said.
But content recommendation is a complex problem - "people change, tastes change, people grow up, you discover newer content or you have to search for something you’re looking for,” Sadekar said.
“What we’re doing is, if you needed to scroll left or right or search for something, we are trying to see how we can improve it ourselves next time so your experience is much better. And the algorithmic engineering teams do that via machine learning pipelines.”
Structurally, a pipeline starts with an “expression of intent” by an algorithmic engineer. A “logical sequence of steps” is then laid out to realise that intent, and each step might use a different technology engine.
Sadekar and his colleagues in the personalisation infrastructure team needed to build a framework to orchestrate those steps so engineers could realise their intent and get it into production faster.
Whatever they built had to be heterogeneous, allowing Netflix’s engineers to continue using technologies of their choosing.
“To support innovation they like to stay with technologies of their choice, and [be able to move to] newer technologies, so whatever we built that would orchestrate these pipelines needed to work across a heterogeneous set of environments," Sadekar said.
“[For example, open source data processing engine] Spark happens to be something that’s used pretty heavily so the engineers wanted native Spark support, meaning we needed to be able to treat a Spark step [in a pipeline] or a Spark submit job as a first-class citizen."
It also had to be multi-tenanted: a nod to the large number of teams within Netflix that work with machine learning.
The end result is Meson, which has been in production at Netflix now for over a year.
The plan is to “soon” open source it, thereby becoming another tool born in web-scale companies that enterprises will be able to take advantage of.
Similar examples of this type of reuse include Kafka, which came from LinkedIn and is now used by the likes of CBA, and Polymer, a library developed by Google that has found its way into ING Direct Australia’s architecture.
Though created to manage machine learning pipelines, Netflix said Meson is a "general purpose workflow orchestration and scheduling framework", opening it to other types of use cases.
The streaming giant first revealed the Meson orchestration and scheduling framework in a blog post, but used last week’s Spark Summit in San Francisco to flesh out the project.
“Netflix has one primary goal - we want to delight every single member every single time you interact with Netflix, and the delight really means we want to be able to help you find content that you will love and enjoy every time you interact with the service,” senior software engineer Kedar Sadekar told the summit.
“Happy customers will play more [content] and watch more. The driving force behind our efforts is really the recommendations or the machine learning pipelines that drive the recommendations."
The position of each movie or TV show tile on a viewer’s home screen is determined by these algorithms, he said.
But content recommendation is a complex problem - "people change, tastes change, people grow up, you discover newer content or you have to search for something you’re looking for,” Sadekar said.
“What we’re doing is, if you needed to scroll left or right or search for something, we are trying to see how we can improve it ourselves next time so your experience is much better. And the algorithmic engineering teams do that via machine learning pipelines.”
Structurally, a pipeline starts with an “expression of intent” by an algorithmic engineer. A “logical sequence of steps” is then laid out to realise that intent, and each step might use a different technology engine.
Sadekar and his colleagues in the personalisation infrastructure team needed to build a framework to orchestrate those steps so engineers could realise their intent and get it into production faster.
Whatever they built had to be heterogeneous, allowing Netflix’s engineers to continue using technologies of their choosing.
“To support innovation they like to stay with technologies of their choice, and [be able to move to] newer technologies, so whatever we built that would orchestrate these pipelines needed to work across a heterogeneous set of environments," Sadekar said.
“[For example, open source data processing engine] Spark happens to be something that’s used pretty heavily so the engineers wanted native Spark support, meaning we needed to be able to treat a Spark step [in a pipeline] or a Spark submit job as a first-class citizen."
It also had to be multi-tenanted: a nod to the large number of teams within Netflix that work with machine learning.
The end result is Meson, which has been in production at Netflix now for over a year.
The plan is to “soon” open source it, thereby becoming another tool born in web-scale companies that enterprises will be able to take advantage of.
Similar examples of this type of reuse include Kafka, which came from LinkedIn and is now used by the likes of CBA, and Polymer, a library developed by Google that has found its way into ING Direct Australia’s architecture.
Though created to manage machine learning pipelines, Netflix said Meson is a "general purpose workflow orchestration and scheduling framework", opening it to other types of use cases.
Copyright © iTnews.com.au . All rights reserved.
0 comments:
Post a Comment