Data, data en nog eens data

Author: tom

  • Show content of an AVRO file with Python

    —

    by

    tom
    in

    This note describes how we can show the content of an AVRO file with Python. We use python3 as tool here. We use this from an Anaconda framework. I checked whether this installation already contained an AVRO package, but this wasn’t the case. Therefore, AVRO was downloaded (as avro-python3-1.8.2.tar.gz) and next command was issued: C:\ProgramData\Anaconda3\python.exe…

  • What is inside an AVRO file?

    —

    by

    tom
    in

    In the Hadoop world, one may encounter AVRO files quite often. AVRO file can be used in more or less the same way as text files. They can be read, stored and written out. However, the AVRO files are binary files that make them difficult to read. However, help is there. I downloaded the avro-tools-1.8.2.jar…

  • mounting drives

    —

    by

    tom
    in

    Today, I played a bit with mounting external drives. Boy, that was more difficult than I thought. The situation was a follows. I had a linux desktop and I wanted to use network resources. I have a NAS, a samba drive and a few windows shares. A the end of the day, I only got…

  • Sqoop

    —

    by

    tom
    in

    Sqoop is used in an Hadoop environment to load data from a relational database upon an Hadoop platform. I have installed an Oracle 12.2 database. I tried to load data from that database to Hadoop. To do so, one needs to install a java jdbc driver to be able to access that database. For Oracle…

  • Installing Cloudera Manager

    —

    by

    tom
    in

    Cloudera Manager is a great tool to install a Big Data platform. I worked with this tool and let me share some findings. I realise that we have only a limited range of platforms that can be used to install the Cloudera Manager upon. I started with Ubuntu 16.04. In a first attempt, I used…

  • the Oracle Tuning Advisor

    —

    by

    tom
    in

    Oracle has now included an Oracle tuning advisor. Its purpose is to advise on a SQL statement. For me, it provides an answer on whether to add an index or not. I was taught that one should always start with tables that have no indices. Only when it is demonstrated that indices are used, one…

  • SCD-2 and the Oracle Merge statement

    —

    by

    tom
    in

    Yesterday I found a very nice script that allows processing records in a target table with a SCD-2 mechanism. The script is very elegant. It contains several logical steps that are knitted together in one SQL statement. Th mechanism of SCD2 is as follows. First, a comparison is made between a source table and a…

  • Explain plan in Oracle

    —

    by

    tom
    in

    I want to show the usage of an explain plan in Oracle. This will be shown in Oracle 12c. I will create a table and a sql statement. This sql statement will be analysed in a so-called explained plan that shows how Oracle will process the query. To do so, I first created a big…

  • Installing on new versions of an OS

    —

    by

    tom
    in

    I regularly install Oracle. In most cases, such installation is straightforward. Start the setup, provide a few data and click next, next, and next. Half an hour later, one has a fresh instance running. However, I recently got an error when installing Oracle 12c, version 2: installation was stopped as an error was encountered. This…

  • OBIEE First report

    —

    by

    tom
    in

    Oracle has a very nice reporting tool, called OBIEE. It is positioned on top of their database. This allows to exploit the data. To do so, a separate (Weblogic OBIEE) server is created that processes the data for reporting purposes. So on the server side, at least two server processes are running: the DBMS and…