apache-spark

Good quality network connectivity is ever more important. For hybrid fiber coaxial (HFC) networks, searching for upstream \emph{high noise} in the past was cumbersome and time-consuming. Even with machine learning due to the heterogeneity of the network and its topological structure, the task remains challenging. We present the automation of a simple business rule (largest change of a specific value) and compare its performance with state-of-the-art machine-learning methods and conclude that the precision@1 can be improved by 2.3 times. As it is best when a fault does not occur in the first place, we secondly evaluate multiple approaches to forecast network faults, which would allow performing predictive maintenance on the network.

Comparing established and up-and-coming streaming approaches for an integrated real-time data model

Good quality network connectivity is ever more important. For hybrid fiber coaxial (HFC) networks, searching for upstream high noise in …

Getting started with simple dagster pipelines.

Using Apache Spark for sparse matrix multiplication

Combining the power of Scala and Python to make the calculation of percentiles in Spark easy and fast

Finally, nested types in Arrow.

Data preparation using spark without ACID tables

Execute the latest version of spark on HDP.

Use additional python packages with pyspark