Author: tom

Teradata is a DBMS that is more or less created to handle massive amounts of data. I now work with it and this DBMS is able to handle 500 million rows in a split second. Brilliant. Teradata has a tool BTEQ that can be used to run scripts. Let us show a very simple script:…

Recently, I came accross the SVN version control mechanism. The idea is that we have a so-called working directory that acts as a source for a final version of some documents. Once we have a final version, the final version can be committed to another platform. This other platform can be accessed by others. If…

Currently, I work with SAS and teradata. I notice, we have two possibilities to connect SAS to teradata. One possibility is to create a remote library. The definition is: libname Td_direc teradata tdpid =”127.0.0.1″ user=”tduser” password=”tduser” database=”financial”; Another possibility is to create an explicit connection: proc sql;connect to teradata as td(tdpid=”127.0.0.1″user=”tduser”password=”tduser”database=”financial”);create table work.dataX asselect *from…

Once an API is running, we may start a client. This client fires a call. This call is fired with “requests.get”. Between the brackets, we see a call that could be run from any webbrowser. The reponse can then be received with response.json() The json is very similar to a dataframe. The dataframe can then…

I found a brilliant script to handle API requests. I noticed three elements here: it contains a server process, it is able to receive a call and it is able to return results. The script: import flask from flask import request, jsonify import pyodbc import pandas as pd app = flask.Flask(name) app.config[“DEBUG”] = True def…

A small example on using an API is provided below. I have attempted to make the programme as small as possible. It shows how the response is derived. This response is then translated into a dataframe. this dataframe is then split into keys and values. import requests import pandas as pd response = requests.get(“http://api.open-notify.org/astros.json”) if…

This script combines many Python features, I used in the last few months. The idea is a file is read that contains parameters that are used is a API call. The result is then analysed and split in different data element. Finally, some data are stored in a dataframe that is subsequently read in a…

Whenever we have a data in a dataframe, we might have the urge to change the values. Below, three situations will be investigated. The first situation is the situation, whereby one field will be taken into account. The first approach is a one line creation of such transformation. df[“Birthdate”].astype(str).apply(lambda x : datetime.strptime(x, ‘%Y-%m-%d’) if x…

A dataframe in Python can be compared to a table in a database. I also get idea that database functionalities are translated into functions that can be found in dataframes. It is also true that a database table can be quickly read into a dataframe. See: con = pyodbc.connect(‘DSN=AzureSQL;UID=tomvanmaanen;PWD=*******’) cur = con.cursor() sql = ‘SELECT…

It can be handy to copy data from Salesforce to SQL Server if one would like to analyse the data. After all, the query language in Salesforce is not easy to use, whereas SQL Server allows easy access to data, good analytical possibilities and rapid response time. Currently, Python is my favorite tool. It allows…