Project Coordinator (EU) :
Sorbonne UniversityCountry of the EU Coordinator :
FranceOrganisation Type :
AcademiaProject participants :
EU team:
The Principal Interface Timur Friedman is a faculty member at Sorbonne Université, and a member of the LIP6 and LINCS computer science laboratories. His research expertise is in the area of internet measurements.
The principal technical work on the project is carried out by Berat Senel, currently a research engineer at Sorbonne Université, where he is principal developer of EdgeNet.
US team:
US Principal Investigators are Justin Cappos, an associate professor in the Computer Science and Engineering department at New York University's Tandon School of Engineering, and Rick McGeer holds a Ph.D. from the University of California (UC) Berkeley
State of US partner :
New YorkWashington
Starting date :
CacheCash Experiment on EdgeNet
Experiment description
Delivery of large chunks of content, such as video, accounts for a substantial percentage of all internet traffic. Such content, when distributed by large corporations, benefits from CDNs, with nodes hosted by internet service providers (ISPs) providing local service to end users in those ISPs. CacheCash (https://ssl.engineering.nyu.edu/projects#cachecash), the CDN technology under testing by the project, has the potential to change the nature of CDNs by involving the end users themselves directly in serving content through machines that are under their control. Putting users in charge can lead to a wider range of content benefitting from CDNs. CacheCash provides a service in which interested users run caches, and they are incentivised to participate by receiving a crypto-currency (Cachecoin) in exchange for serving content to other users. Both cryptographic and economic analyses demonstrate that the incentives lead CacheCash users to honestly serve content. Analysis has also shown that CacheCash can scale to meet the workload of even the most popular services used today.
CacheCash, the CDN technology, will be tested on EdgeNet to evaluate its performance at a distributed edge network. The measurements would present performance results to see if any additional requirements to be handled before running CacheCash at scale by involving the end-users.
|
The experiment is based on two cutting-edge technologies:
|
The CacheCash experiment addresses the “Discovery and identification technologies” topic of the call. In testing a new blockchain-based CDN technology, it addresses the topic’s focus on “new methods of search, discovery, and access of large heterogeneous data sources” and “multi-media content”. CacheCash’s focus on end-users addresses the “personalised information retrieval” aspect of the topic, and the evaluation of the efficacy of CacheCash will be in terms of “improved quality of experience”.
The key enabling technology is the edge cloud technology provided by PlanetLab Europe through EdgeNet. Edge cloud nodes are an integral part of the 5G vision. EdgeNet is based on contemporary de facto industry standard technologies Docker and Kubernetes. The EdgeNet cluster consists of approximately 50 nodes located in North America, South America, and Europe. It has been successfully piloted in Fed4FIRE+ open call experiments and is now ready for larger scale experiments engaging real internet users.
There are several ways to look at the innovative aspects of this work. First, the innovative technology we deploy has never been tried at a scale even remotely approaching this. It is unknown how the cryptographic techniques for distributing content securely in a way that economically incentivises cheating will work given the latency, non-transitive connectivity, and scalability constraints of the internet. Further, our effort to use real world software distributed to real world users will provide a basis for understanding how such technology will work in practice and at scale.
To date, most efforts around building distributed data delivery have focused primarily on data storage or on trusted nodes that deliver content. The expected impact of the CacheCash experiment will be to push the boundaries more toward low latency and bandwidth, which are highly valued in the CDN space. The relaxation of trust assumptions in existing platforms also will open up the set of participants. Somewhat counterintuitively, if we are able to demonstrate that high performance can be provided by untrusted nodes, this may enable even CDNs with trusted nodes to have better security as they may alter their architecture to remove trust assumptions without harming performance. By having our experiment be used in a practical deployment by real internet users, we will be able to evaluate the strengths of such approaches, the weaknesses, and to identify research challenges that must be tackled to allow these technologies to become widespread.
Implementation plan :
The CasheCash team will use PlanetLab Europe (https://planet-lab.eu/), which is one of the EU Fed4FIRE+ platforms. PlanetLab Europe offers an edge cloud service that is particularly apt for 5G experimentation. Researchers can deploy their experiments on containers that run on servers distributed around the edge of the internet. These servers are completely open to the internet, meaning that researchers can deploy experimental services to real users, and those services will be subject to real internet conditions (latencies, routing changes, cross traffic, etc.) Currently, about 50 such servers in Europe and North America are available.
PlanetLab Europe is migrating these servers to the new EdgeNet (https://edge-net.org/) technology that Sorbonne University (EU/PI Friedman) is developing in a currently unfunded collaboration with NYU Tandon School (US/PI Cappos) and US Ignite (US/PI McGeer). With EdgeNet, we update PlanetLab Europe to the most recent and widely used containerisation technologies: Docker, and the emerging industry standard for container orchestration, Kubernetes.
Impacts :
Impact 1: Enhanced EU – US cooperation in Next Generation Internet, including policy cooperation.
Sorbonne Université’s collaboration with US Ignite and the NYU Tandon School is an outcome of the Global Experimentation for Future Internet (GEFI) workshop, 26-27 October 2017, in Rio de Janeiro, Brazil. It became clear that there was a shared interest in working towards a common successor platform to the existing LXC-based PlanetLab technology, to move to the contemporary de facto industry standards of Docker and Kubernetes. This collaboration had, until this experiment, not been formalized through joint funding, but it is a very real and enduring collaboration, featuring weekly meetings to advance our joint goals.
Impact 2: Reinforced collaboration and increased synergies between the Next Generation Internet and the Tomorrow's Internet programmes.
The US side has in the past benefitted from an NSF EAGER grant to support some of this work, and the EU side has received partial support from French national funding. At times, the development initiative has been taken in the United States and Canada, and at times in France. Our ability to continue together had, however, constantly been in question due to the uncertainties on both sides of the Atlantic regarding ongoing financing. This NGIatlantic.eu funding is the first official recognition of our collaboration, enabling the US partners to seek similar funding, and it sets the entire collaboration on a firmer footing for the future. This experiment also increased the level of collaboration between EU and US partners, allowing them to exchange information about the NGI Generation Internet and the Tomorrow's Internet programmes to seek further potential collaborations in the future.
Impact 3: Developing interoperable solutions and joint demonstrators, contributions to standards.
This successful CacheCash trial on EdgeNet means that CacheCash is capable of involving any Kubernetes-based cluster such as EdgeNet to run caches on its nodes, and EdgeNet can host any CDN solution. This mutual interest allows both software-based systems to be demonstrated to the trans-Atlantic community. Because both are open source solutions, they can either be adopted by high-tech startups or empower spin-off startups. Moreover, EdgeNet, as a testbed, can attract the EU-US ecosystem of top researchers to run their experiments.
Impact 4: An EU - US ecosystem of top researchers, hi-tech start-ups / SMEs and Internet-related communities collaborating on the evolution of the Internet
EdgeNet has now received funding from the US-based company VMware, which has now based much of its business strategy on Kubernetes, the same underlying technology that EdgeNet uses. Through our VMware connections, we have been introduced into the Kubernetes working groups that are relevant to our EdgeNet work: the Multitenancy working group and the IoT-Edge working group. We have made presentations to these working groups, and we are preparing code that we plan to present to the Multitenancy working group and that we hope will be taken up by the Kubernetes community.
Results :
We created six setups at different times to analyze the effect of cache location on content retrieval for clients distributed in diverse geolocations. Each setup contains caches on an EdgeNet node located at a different location. Throughout the measurement, for each setup, we made repeated CacheCash-specific curl requests from EdgeNet nodes. We performed these measurements to collect data regarding overall transaction duration for content retrieval, time-to-first-byte (TTFB), and remote procedure call (RPC) service latencies using the gRPC protocol.
TTFB in the 300-500 ms range is the standard expectation. A shorter TTFB than this range means good performance. Metrics confirm that TTFB in CacheCash is within the standard range despite encryption and decryption operations that are required as part of content delivery. Moreover, if a cache is close to a client, e.g., both are in the same city, the TTFB outcome indicates good performance.
On the other hand, operations regarding encryption, decryption, and puzzle-solving (the computation required for solving puzzles is a disincentive to cheat in CacheCash) add considerable overhead, causing the overall transaction duration to be nearly twice what it would otherwise be. However, this is not an unsolvable problem. The cryptography design can be altered to achieve a shorter transaction duration while still meeting security requirements.
In addition, we prepared a setup dedicated to throughput analysis. In this setup, we used the nodes in the LIP6 Lab, where we are based, to host essential services with caches, and we used a compute cluster at LIP6 to emulate CacheCash clients. The goal was to make as many concurrent requests as possible. Cache throughput reached 45 MiB/s with 150 concurrent clients. CacheCash’s publisher daemon is memory intensive, which limited us from increasing the number of clients that make requests in order to reveal the maximum throughput capability.
Measurement data is publicly available at the following URL:
https://drive.google.com/drive/folders/1AIkyOmovtFZn3gZHQWPz_Uf7SX8xeWST?usp=sharing.
Deploying CacheCash on EdgeNet uncovered implementation issues and required features for both the CacheCash system and the EdgeNet testbed. At first, caches in CacheCash were not reachable over the Internet, revealing modifications that were required to allow them to be reached via a node’s external IP address. We also found that secure communication between the CacheCash services was not possible to start with, and we implemented this feature. Regarding EdgeNet, there wasn't a component to automate secure communication between containers within the cluster. We therefore integrated a third-party solution in EdgeNet to enable secure communication. We also developed ready-to-use configuration files to allow CacheCash to be deployed on any Kubernetes cluster, and scripts to automate the collection of data from measurements.
Future Plan :
CacheCash is a blockchain-based content delivery network technology that aims to establish a broader network at the edge than conventional CDNs. The measurement data gathered throughout this experiment demonstrated that CacheCash is a promising framework for next generation CDN infrastructures. Despite the performance issues, which are understandable at this TRL for such a complex system, CacheCash meets general expectations for a conventional CDN. Furthermore, it has the potential to improve CDN performance in terms of TTFB and content retrieval time by incentivizing end-user with CacheCoin in exchange for serving content as a part of the network. To do so requires further optimizing the current design of the cryptography processes.
From EdgeNet's point of view, this experiment showed the testbed has the ability to welcome advanced experiments like CacheCash. A feature added to the testbed is a certificate manager to enable secure communication between containers. EdgeNet could improve its fair sharing of storage on nodes as it starts to welcome more experimenters.
As future work, the CacheCash team focuses on solving the implementation issues revealed by this experiment. Following that, they plan to enable an interface to change the cryptographic design easily, so that they can study performance versus security trade-offs. At this point, EdgeNet again will be the place to put CacheCash into the test. Once done, the blog posts will be published on relevant communication platforms, including the CacheCash website that is under construction. In addition to these, the preparation of a paper that presents the findings is underway. We also look forward to further collaboration to strengthen ties between the institutions, including researcher visits when the current pandemic is past.
Key results
-
CacheCash (https://ssl.engineering.nyu.edu/projects#cachecash)
-
Improvements to PlanetLab (https://planet-lab.eu/) with EdgeNet (https://www.edge-net.org/) capability.
-
Measurement data is publicly available at the following URL: results
-
Paper published at ACM EuroSys 2021.
-
Links forged with Linux foundation and Kubernetes community.