Data, data en nog eens data

Author: tom

  • reading an HDFS file in Python

    —

    by

    tom
    in

    In this note, I show you how to get data from an HDFS platform into a Python programme. The idea is that we have data on HDFS and we would like to use these data in a Python programme. So, we must connect to HDFS from within a Python programme, read the data , transform…

  • Putting a file on HDFS

    —

    by

    tom
    in

    Putting a file on HDFS is relatively easy. There are a few steps to take. Let us assume the file is on a linux system. The first step is to copy the file to an area where it can be stored with the hdfs user as its owner. On my system, I have /tmp that…

  • Estimating with Python

    —

    by

    tom
    in

    It is relatively easy to do an estimate with a Python script. This is due to the fact that Python works with matrices and such matrices can be used as an input in a estimation procedure. I created an example where a dataset is retrieve from Oracle. Then the dataset is translated into a matrix.…

  • Read and write by Python

    —

    by

    tom
    in

    Python seems to be a very convenient way to transfer data to and fro Oracle. It has capabilities to set up a connection and it seems quite capable to transfer a matrix into a table and vice versa. Next code shows this. It first retrieves the content of a table. In a second step some…

  • Python in a map reduce environment

    —

    by

    tom
    in

    I have written a very small python programme that follows the mapper / reducer sequence. This works as a replacement of a more complicated set of Java programmes that might be created to generate a mapper / reducer sequence. The idea is relatively simple. We create a stream from an input file. That stream is…

  • Three little handy Oracle statements

    —

    by

    tom
    in

    I have three little Oracle statements that I us quite often. I realise I use these statements in most programmes I write. But at the same time, I often forget the exact syntax. Therefore this entry in this blog. It will act as a look-up whenever I once again forgot the precise syntax. The first…

  • Reading XML in Oracle -3

    —

    by

    tom
    in

    I now have a project where I need to write the content of an XML file into an Oracle table. The idea is actually quite simple. An XML file might be seen as a document that had an hierachical structure. It is composed of several seperate subdocuments, that could be seen as several trees.  …

  • Connect from 32 bit environment to 64 Oracle instance

    —

    by

    tom
    in

    One may retrieve the version of Oracle with a simple command: “select * from v$version;”. One may then see that the Oracle instance is a 64 bit version. In my case, the full version name is “Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 – 64bit Production”. Once this is known, one may ask whether it…

  • Create a report with BI Publisher

    —

    by

    tom
    in

    One could create reports within the OBIEE sphere with the analytics. Another possibility is to use the BI Publisher. This option has several advantages. One advantage is that one choose between a direct query on the database or use the route via the creating of a subject area. I can see advantages using a direct…

  • Bash script to upload file

    —

    by

    tom
    in

    In many environment, we have a situation whereby a file must be transmitted via a script from one server to another. Assume we have a linux environment where Bash is installed. In that case, a sftp connection can be used. The principle is relatively simple: a sftp session is started from one machine to another.…