среда, 22 января 2014 г.

How to read XML file into pandas dataframe using lxml

This is probably not the most effective way, but it's convenient and simple.

Let's pretend that we're analyzing the file with the content listed below:

<xml_root>

    <object>
        <id>1</id>
        <name>First</name>
    </object>

    <object>
        <id>2</id>
        <name>Second</name>
    </object>

    <object>
        <id>3</id>
        <name>Third</name>
    </object>

    <object>
        <id>4</id>
        <name>Fourth</name>
    </object>

</xml_root>

First, we need import lxml objectify

from lxml import objectify

Then, open the file:

path = 'file_path'
xml = objectify.parse(open(path))

Get the root node:

root = xml.getroot()

Now we can access child nodes, and with
root.getchildren()[0].getchildren()
we're able to get the actual content of the first child node as a simple Python list:

[1, 'First']

Now we obviously want to convert this data into data frame.

Les's import pandas:

import pandas as pd

Prepare a empty data frame that will hold our data:

df = pd.DataFrame(columns=('id', 'name'))

Now we go though our XML file appending data to this dataframe:

for i in range(0,4):
    obj = root.getchildren()[i].getchildren()
    row = dict(zip(['id', 'name'], [obj[0].text, obj[1].text]))
    row_s = pd.Series(row)
    row_s.name = i
    df = df.append(row_s)

(name of the Series object serves as an index element while appending the object to DataFrame)

And here is out fresh dataframe:

  id    name
0  1   First
1  2  Second
2  3   Third
3  4  Fourth


Full source code:

from lxml import objectify
import pandas as pd

path = 'file_path'
xml = objectify.parse(open(path))
root = xml.getroot()
root.getchildren()[0].getchildren()
df = pd.DataFrame(columns=('id', 'name'))

for i in range(0,4):
    obj = root.getchildren()[i].getchildren()
    row = dict(zip(['id', 'name'], [obj[0].text, obj[1].text]))
    row_s = pd.Series(row)
    row_s.name = i
    df = df.append(row_s)

понедельник, 9 декабря 2013 г.

Things to do after installing fresh Ubuntu on a laptop

(serves mostly as a reminder for myself, someday I'll probably turn in into a script; features updated regularly; applicable to 12.04 LTS)

1. Install power saving tweakers:

http://askubuntu.com/questions/300953/how-can-i-improve-battery-life-on-my-laptop

2. Enable Hibernation:

http://askubuntu.com/questions/94754/how-to-enable-hibernation

3. Disable notification baloons

http://askubuntu.com/questions/13464/how-can-i-customize-disable-notification-bubbles

5. Set up SSH config

4. Install password manager

вторник, 19 ноября 2013 г.

Enable SSH private/public key authorization & setting up SSH config file on Linux (Ubuntu)


If you access your SSH remote server with login/password pair and bored by entering password every time you log in (there is no way to save your password in /.ssh/config), consider switching to private/public key authorization instead and setting up the config file, so you'll be able to login like that:

ssh yourhost

Without specifying your host parameters & password every time.

The setup consists from two simple steps: generating a keypair and setting up a config file for SSH.


Generating a keypair


First, you have to generate a key pair using ssh-keygen tool. Like this:
ssh-keygen -t dsa

You'll see the following output:
Generating public/private dsa key pair.

Enter file in which to save the key (/home/yourname/.ssh/id_dsa): hit enter

Enter passphrase (empty for no passphrase): again, hit enter

Enter same passphrase again: hit enter one more time

Your identification has been saved in /home/yourname/.ssh/id_dsa.

Your public key has been saved in /home/yourname/.ssh/id_dsa.pub.


You can also use ssh-keygen -t rsa for better security.
Now you may notice that two new files apper in your ~/.ssh directory: id_dsa & id_dsa.pub

Now you have to copy id_dsa.pub to your remote host. We'll use scp for that

scp ~/.ssh/id_dsa.pub name@host:~/.ssh/

if you need to specify port, use -P option

scp -P <port number goes here> ~/.ssh/id_dsa.pub name@host:~/.ssh/

It will ask for your SSH password, type it and hit enter.

Okay, now your pub key is copied to the remote host.

Now, SSH to the host (it's the last time you going to do that, I promise!):

ssh name@host

Change the name of id_dsa.pub to authorized_keys

mv ~/.ssh/id_dsa.pub ~/.ssh/authorized_keys

Change permissions of the file & folder:

chmod 755 ~/.ssh && chmod 755 ~/.ssh/authorized_keys

Setting up the SSH config file

This step is much simpler.

Just create the ssh config directory  and then the config file with your host parameters:

mkdir ~/.ssh

cd ~/.ssh

touch config
nano config

Then type your remote host details:

Host host1
HostName host1.example.com
User yourname
IdentityFile "~/.ssh/id_dsa"

Save file by pressing Ctrl + X, then hitting Enter

host1 is now the alias for your remote host host1.example.com

To ssh to your host, simply type:

ssh host1

In the terminal, and you will be logged in. Much better now.

воскресенье, 17 ноября 2013 г.

Nate Robins OpenGL tutors on Ubuntu Linux

Nate Robins tutors are great helpers for everybody who wants (or obliged) to learn OpenGL.

The only problem: pre-compiled Linux binaries aren't presented on Nate's website and provided sources doesn't compile in Linux "out of the box".

As it took me some time to figure out how to compile them (I'm quite new to C++), so I'm going to share compilation details here.

All you have to do is to change LDFLAGS string in Makefile from:

LDFLAGS = -lGL -lglu -lglut

To:

LDFLAGS = -lGL -lGLU -lglut -lm

Then, just execute

make

in terminal (don't forget to cd in the directory containing source code), and you're done.

If any errors still appear in the output, try to install the dependencies:

sudo apt-get install freeglut3-dev

пятница, 23 ноября 2012 г.

Arduino FTDI reset

If your Arduino TX/RX LEDs are insanely blinking for couple of second after you've connected it to PC, and PC doesn't see the connected device, don't rush to throw the board.

You may accidentally make a connections between FT232RL pins (it's USB-UART converter on the back of the board), which may cause unpredictable behavior.
In order to fix that, follow these simple steps:

1. Find RL232RL chip on your Arduino
2. Connect Arduino to your computer
3. Find a GND pin on Arduino, and make a connection between GND and # RESET pin of the FT232RL (take a look at the pinout below). It will reset the chip.
4. Re-connect Arduino to PC. It should work. At least, it works for me.