Oh my God: how good is open source
tom
A few days ago, I had to write a Python script that would enable us to write some data to an Oracle database. At first, I had no idea how…
Curl and elasticSearch
One of the most useful utilities is “curl”. This wonderful tool can be used to transfer data from one platform to another. It is relatively easy to install in Windows,…
ElasticSearch
A new and popular nosql database is the Elastic Search database. This database is easy to install en easy to run. But is it easy to insert data and extract…
Scala merging files
In a previous post, I showed how two files can be merged in Scala. The idea was that RDDs were translated as data frames and a join was undertaken on…
Merging files in Scala
I understand that Scala may be used in an ETL context. In ETL, an important element is the merge of two files. We will get data from different sources and…
Getting a histogram from Big Data with Scala
Scala can be used as a tool to manipulate big data. If it is used in the spark context, we have a possibility to combine two strong tools: spark with…
Scala
Scala is a language that is used for general purposes. One may use it as a statistical tool, a tool to undertake pattern matching etc. Just like any other programming…
Another Pyspark scripts
In this note, I show yet another Pyspark with slightly different methods to filter. The idea is that file is read in a RDD. Subsequently, it is cleaned. That cleaning…
The 1000th wordcount example
I just discovered the 1000th wordcount example. It is based on Pyspark. The idea is actually quite simple. One creates a script. This script can be written in any editor.…
Joining files with Pyspark
Pyspark allows us to process files in a big data/ Hadoop environment. I showed in another post how Pyspark can be started and how it can be used. The concept…