Friday 29 March 2013

Using Python within C++

I tried to think of a Jake the Snake reference... but failed.

I now have a working (and easily available) package for programming my Lego Mindstorms NXT (look here for more info). The downside is that it's written in python and I'm more of a C++ kind of guy. I could root around to try to find something that's C compatible but as this was easily available in the Mint repos, I thought it might be a better idea to just call the python code within C++.

This is not as easy as you'd think, or at least, it isn't if you want to do it right. If you only care about running some commands through the Python interpreter and not paying any attention to the results, the you can easily use something like the following:

Py_Initialize();
PyRun_SimpleString("import os");
PyRun_SimpleString("print 'hello world'");
Py_Finalize();

This is essentially a python version of a system() call (kindof..).

Anyway, we're getting ahead of ourselves. First, you need to gain access to the python headers and libraries. Through Linux, this is fairly trivial as I would guess all distros would have the python2.7-dev (or whatever version you choose) available - note that it may not be installed by default though. For Qt, you can then add the following to your .pro file:

 LIBS += -Lpython2.7 -lpython2.7

and then you can include the python header:

 #include <python2.7/Python.h>

Putting the simple hello world code into a main function should build and run as expected. So that's the basics of accessing python within C++. How do you go about integrating this properly into your code though? For that, you need to go a bit more in depth into how python is written and how it handles memory and objects.

One of the main selling points of python is it's garbage collection and object reference handling - it deals with all references to any objects and deletes objects that don't have any. Within the framework of the python language, this is fine. However, when you're poking around under the hood, you have to be careful to do what python usually does for you. In other words, you have to be very careful about tracking object references that get passed back to you from function calls (and nearly all function calls pass back object references!).

To integrate with the PythonNXT python module, I've found the following functions the most useful. There are many others (obviously) that can be found here but this should give you the basic idea of what's going on and what you need to be careful about. The main things to remember is that everything returns/deals with a PyObject base type (or actually, a pointer to one) and you have to keep track of anything that gets returned to you from the API functions.

Py_Initialize

Initialises the python interpreter. Call this before doing anything else!

Py_Finalize

Shuts everything down to do with the interpreter. Don't call anything after this!

PyObject* PyImport_ImportModule(const char *name)

This will import the given module and return a new reference to the module object.

PyObject* PyObject_GetAttrString(PyObject *o, const char *attr_name)

This will return the named object associated with the given  object. For example, if you've just loaded a module and want to call a function, use this with the function name to return a new reference to that function. Or, you have a python object that you want to call a function on, use this to get a reference to that function and then use PyObject_CallObject to actually call the function.

PyObject* PyObject_CallObject(PyObject *callable_object, PyObject *args)

When passed a reference to a function (say from the above, PyObject_GetAttrString), this will call the given function with arguments given (specified as Python objects). This returns a new reference which will be the pythonified version of the actual function return value.

PyObject* Py_BuildValue(const char *format, ...)

This constructs a python object (tuple, list, single value, etc.) based on the given format and supplied arguments (e.g. Py_BuildValue("(i)", 1) will give a tuple of one integer value). Used for PyObject_CallObject above - note that it seems you should always create tuples for this, not just single values (e.g. "(i)" rather than "i")!

void Py_XDECREF(PyObject *o)

This will decrease the reference count of the given object and therefore delete it if the count goes to 0. Note that this version checks for NULL pointers being passed. This is the main thing to remember - you must call this on any returned objects when you're done with them unless the function returns a 'borrowed' (rather than 'new') reference. If you don't, you'll be leaking memory like sieve.


This covers the basics of using Python code through the C-API. In addition to these, the following (and related) are worth looking up to manipulate Lists and Tuples. They're fairly self-explanatory:

PyList_GetItem
PyTuple_GetItem


Finally, the basic types (int, double, etc.) have dedicated python objects associated so you can easily cast to these objects (e.g. PyIntObject) to access the actual data values from these objects.

In a future post, I'll go into a bit of detail how I've used this to create a basic interface to the PythonNXT module through C++/Qt.

Sunday 24 March 2013

Getting started with Git and Qt

I'll admit, they could have probably come up with a better name

So now I've started producing code that I would prefer not to lose, I've turned my mind to Version Control solutions and, ever one to go with a crowd, it seems that git is the way forward. I've used a few VCSs in my time including CVS (urgh), SVN (meh) and even Microsoft's Source Safe (Aaaarrrgghhh!), but now I've had a chance to play with git, it does seem to be significantly easier and more friendly. That being said though, I have to admit that one of the major selling points is the free github account with repo space. This allows me to not only store my code remotely, but also develop on whatever machine I happen to find myself in front of.

There are many good tutorials for git, so I'm just going to note here what I found useful to know and how to get it working with Qt. Fundamentally, git is a distributed VCS that allows copies of a code repository to be taken and edited in isolation from the main repo. Once the user is happy, they can push their changes to the main repo. Each remote copy can be committed to, rolled back, etc. by itself before this push takes place. The key commands are to achieve all of this are:

git init     # initialise a git repository
git clone    # clone an existing repo (e.g. from github)
git add      # add files to a repo
git commit   # commit changes to the local copy of the repo
git push     # push any committed changes to to the master repo
git tag      # Tag the current version of the repo

To get this working with Qt is very easy as Qt comes bunded with git support built in. For my projects, as I'd already started them, I first needed to create a git repo on github for them, clone the (empty) repo, copy the necessary files in to the appropriate directory and finally add, commit and push the changes. Note that you don't want to use git to manage the .pro.user Qt file as that is created on a per machine basis.

This gives you a working repo that you can now clone and start using. I found this easiest to do by deleting the previous repo from the local machine, firing up Qt and then selecting 'New Project' and then 'Project from Version Control' and 'Git Repository Clone'. Put in the Github details of the repo and Qt will do the rest. At this point you can do all editing, etc. within Qt and when you're happy, perform commits and pushes through this as well (Tools -> Git). The only thing I've currently found you can't do is tag through Qt - this must still be done through the command line.

Note that I'm sure there is a much easier way of setting up the project but this is how I got it to work and as it was trivial to do, couldn't really be bothered to look around for a more elegant solution. If you wish to look at my developing code base and laugh at the poorly written code, you can find it here:

https://github.com/doc-sparks?tab=repositories

Update: During my time with git, I found out that removing a tag can be a bit of a pain. I found the following to work quite well (creating a dummy one first as an example):

# start with a tag
git tag test

# push this to github
git push origin --tags

# delete this tag from local repo
git tag -d test

# finally, push this tag change to github (note that --tags won't work as the removed tag isn't included)
git push origin :refs/tags/test

Saturday 23 March 2013

A Spinning Cube with OpenGL and Qt

At one point, even Crysis looked like this

As I mentioned before, I've always liked 3D art and games. I guess it comes from seeing the transition from basic 2D platformers to the 3D awesomeness that was Doom first hand (and if you actually need to click on that link to know what Doom is, you should be ashamed). I still have (vague) ideas of doing my own at some point but that is obviously now a lot easier with things like Unity and the Unreal Engine. There is very little need to code your own engine these days unless you're a major game studio (in which case you're probably not reading this).


Having said all that, I think it's always good to go over some of the basics of these technologies so you have a vague idea how they work and are coded. Plus, there are many situations where the pre-packaged engines aren't useful for what you're trying to do (as in this case here - but more on that in another post some time down the line!). To that end, I decided to come up with the minimal startup to running an OpenGL program within the Qt framework. This would provide me with a good basis for going forward in anything 3D related in the future and also help me understand the basic requirements of an OpenGL program.

Note: There are many good tutorials on the web for this (I personally  use Neon Helium). I'm putting this here (as with all my posts) to record my own personal experience and to help me remember just what I need to know!


To start with,  create a New Project in Qt: New Project -> Qt Widget Project -> Qt GUI Application. This sets you up with a basic main window and main cpp file. Now, add in a new widget that will be your main OpenGL widget (Right Click project -> Add New... ->C++, C++ Class and make sure you set the base class as QGLWidget and Type as QObject).


This sets up the widget class for you to add to. Next thing is to make this appear (and take over) the main window. So add the following to the Main Window Constructor:


 
MainWindow::MainWindow(QWidget *parent) :
    QMainWindow(parent)
{
    // Show the interface fullscreen
    showFullScreen();

    // create and add the openGL Widget
    OGLWidget *w = new OGLWidget();
    setCentralWidget(w);
}

You may also want to override the protected keyPressEvent to allow you to quit out:

 
void MainWindow::keyPressEvent(QKeyEvent *e)
{
    if (e->key() == Qt::Key_Escape)
        close();
    else
        QWidget::keyPressEvent(e);
}

This has setup the basics for Qt, now comes the OpenGL bit. There are 3 methods you need to override in OGLWidget (or whatever your widget is called) to get your program to actually show anything (note the includes required - put these at the top of your implementation file):

initializeGL

 
#include "oglwidget.h"
#include <GL/glu.h>
#include <QDebug>
#include <QTimer>
#include <QMouseEvent>

void OGLWidget::initializeGL()
{
    // enable depth testing - required to stop back faces showing through (back face culling)
    glEnable(GL_DEPTH_TEST);

    // set up the timer for a 50Hz view and connect to the update routine
    refreshTimer_ = new QTimer(this);
    connect(refreshTimer_, SIGNAL(timeout()), this, SLOT(updateGL()));
    refreshTimer_->start(20);
}
  • Called (not surprisingly) just before the first call to resizeGL or paintGL
    • The only OpenGL thing done here is to set the depth Test through glEnable(GL_DEPTH_TEST); Not doing this can make back faces show through.
    • Other than that, I just setup a timer to repaint the screen

resizeGL

 
void OGLWidget::resizeGL(int width, int height)
{
    // Set the viewport given the resize event
    glViewport(0, 0, width, height);

    // Reset the Projection matrix
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();

    // Calculate The Aspect Ratio Of The Window and set the perspective
    gluPerspective(45.0f,(GLfloat)width/(GLfloat)height,0.1f,100.0f);

    // Reset the Model View matrix
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();
}
  • Called on the resize of the widget with the width and height of the widget passed through
    •  This is where the viewport is setup and the basic matrices are reset to the identity
    • glViewport - Set up where in the widget to show the 3d view
    • glMatrixMode - Set which matrix to use. Of interest here is the ModelView matrix (from local object coords to eye or camera view) and Projection matrix (how the eye coords are projected and clipped to the screen). Both are set to the identity here.
    • gluPerspective - A GLUT routine that sets a nice viewing frustrum with a z clipping plane

paintGL

 
void OGLWidget::paintGL()
{
    // cler the screen and depth buffer
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    // reset the view to the identity
    glLoadIdentity();

    // move into the screen
    glTranslatef(0.0f, 0.0f, -6.0f);

    // rotate the cube by the rotation value
    glRotatef(rotValue_, 0.0f, 1.0f, 0.0f);

    // construct the cube
    glBegin(GL_QUADS);

    glColor3f(   1.0,  1.0, 1.0 );
    glVertex3f(  0.5, -0.5, 0.5 );
    glVertex3f(  0.5,  0.5, 0.5 );
    glVertex3f( -0.5,  0.5, 0.5 );
    glVertex3f( -0.5, -0.5, 0.5 );

    // And 5 others like this...

    glEnd();

    // finally, update the rotation
    rotValue_ += 0.2f;
}
  • Called on any redraw event
    • Here the actual polygons are drawn
    • glClear - used to clear both the color buffer (basically clearing the screen to a colour set using glClearColor) and the depth buffer as well
    • Reset the MODLVIEW matrix (the default here) to the identity
    • Apply both a translation and rotation to the current (modelView) matrix
    • Finally, setup to draw quads and set the colour and vertex positions for all faces of the cube

And there we have it! This produces a basic spinning cube in front of the camera while using a full screen Qt window to display it. Next, mouse control...

Update: The code for this can be found in my github repo:

https://github.com/doc-sparks/Interface/tree/v0.1

Thursday 7 March 2013

Editing Bootable DVDs as ISO images

 When someone says ISO, I reach for my Q*Bert

A friend recently asked how to go about changing the contents of an install DVD with freely available tools that didn't have a size limit. This was something I hadn't explicitly done before and sounded like quite a fun thing to try (you may not entirely agree with the use of the word 'fun' there) so I thought I'd look into it and try to hack my Win7 install DVD a bit. For this I used Ubuntu (though I'm guessing almost any Linux install would work), ddmount, mkisofs and a script called geteltorito.pl (available here). I was going to use ISO Master but hit problems which I will get to later. If you're of the Windows persuasion, I'm not sure how to do this using free tools, however there's nothing to stop you creating a live boot CD/USB stick of a Linux distro, going into that and just going into some area on your existing HD.

Anyway, the first step was to create an ISO image of the DVD. This was fairly easy using:

 
markwslater@markwslater-System-Product-Name:~$ dd if=/dev/cdrom of=~/win7_image.iso
6298216+0 records in
6298216+0 records out
3224686592 bytes (3.2 GB) copied, 229.392 s, 14.1 MB/s
markwslater@markwslater-System-Product-Name:~$ ls -ltrh ~/win7_image.iso 
-rw-rw-r-- 1 markwslater markwslater 3.1G Mar  5 23:26 /home/markwslater/win7_image.iso

So we now have the ISO image which we could (in theory) start messing with through various tools. Apparently (though I didn't confirm this) most of the free ones or trials for Windows have an upper limit on the size of ISO you can mess with (around CD size by all accounts). Looking around I happened across ISO Master and thought this would do the job. Unfortunately, plugging the above ISO image in only ended up with a README that said:
 
This disc contains a "UDF" file system and requires an operating system
that supports the ISO-13346 "UDF" file system specification.

Which was not the most helpful. So, after a bit of searching I found the following thread that explained how to do things with basic Linux tools. The key thing is to preserve the master boot record of the ISO and recreate it with this intact. The aforementioned geteltorito.pl perl script did a grand job of this:

 
markwslater@markwslater-System-Product-Name:~$ ./geteltorito.pl  win7_image.iso > ~/boot.bin
Booting catalog starts at sector: 22 
Manufacturer of CD: Microsoft Corporation
Image architecture: x86
Boot media type is: no emulation
El Torito image starts at sector 734 and has 8 sector(s) of 512 Bytes
Image has been written to stdout ....
markwslater@markwslater-System-Product-Name:~$


It's then quite easy to mount the existing ISO image (read only), copy this elsewhere, change the permissions and do what's necessary:

 
mkdir windvd
sudo mount -t auto -o loop win7_image.iso /home/markwslater/windvdcp -r windvd windvd_rw
chmod u+w windvd_rw -R
echo stuff > windvd_rw/my_file.txt


So all that remains is to recreate the ISO with the boot.bin file created above:
 
mkisofs -udf -b boot.bin -no-emul-boot -hide boot.bin -relaxed-filenames -joliet-long -D -o ~/new_win7.iso  ~/windvd_rw


Burn this to a DVD however you wish and bob's-your-uncle, you have recreated the win7 install DVD after messing with it :)