@shijitht

October 21, 2010

Inline assembly basics

Filed under: C — Tags: , — shijitht @ 1:34 am

Inline assembly functionality allows embedding assembly code in C program. This is like an inline function, where the corresponding value gets substituted. Here assembler substitutes our assembly code in proper place with no change. GCC follows AT&T systax for assembly code.

AT&T syntax

  1. Register prefixed with % and $ for immediate/constant.(%eax and $10)
  2. Source operand comes first.(opcode source destination)
  3. Size of operand as a suffix to opcode. i.e. b -> byte, w -> word,
    l -> long.(movl)
  4. Indirect memory reference using parenthesis, ‘( ‘ and ‘ )’.(  (eax)  )

In C

syntax:   asm(” assembly code “);
function asm is used to write assembly code in C.

test.c
-------
#include<stdio.h>
int fun()
{
 asm("mov $24, %eax");
}
int main()
{
 int n = fun();
 printf("%d\n", n);
 return 0;
}

In test.c function fun has no return statement. But it returns 24. When a function returns, its return value is places in eax register. But we can explicitly set eax using asm. So a value can be returned without a return statement. The move instruction used is darkened above.

Operation which are very difficult or unable to perform in C can be achieved easily using inline assembly. Rotation of a block of bytes is done in a single step using  asm.( ror or rol ). But in C, it takes an effort. And all machine level instructions can be used, which can’t be produced with gcc. eg: logical and arithmetic shift. Architecture dependent coding and optimization is done using inline assembly. Speed of code can be further improved with hand written assembly.

October 20, 2010

Valgrind

Filed under: C, Commands, GNU/Linux, Tools — Tags: , , , , — shijitht @ 1:18 pm

Valgrind is a collection of tools to check the correctness of a program. The main tool in it is memcheck. It reports memory leak, out of bound writes, improperly initialized variables etc. This provides a report which pin points the correct location of the error. So this is a good tool to debug programs with unpredictable behavior and crash.

Using

Inorder to see the exact line number of error, compile the code with -g option and reports could be misleading if optimization above level 1 are used(-O1). The -g option compiles the code with debugging symbols enabled, this helps valgrind to locate line number.
Use this program prog.c

prog.c
-------
void fun()
{
    char *a = (char *)malloc(10 * sizeof(char));
    a[10] = 'a';
}
main()
{
    fun();
    return 0;
}

prog.c has two major errors,
1. a[10]  = ‘a’;
a[10]  is out of the allocated region. Writing to this region could produce mysterious behavior. This is called heap block overrun.
2. 10 byte block pointed by a is never freed. So on return to main, that block remains inaccessible and unusable. This is a serious memory leak.

Lets use valgrind to detect these errors,
Compile the code with -g option

$ cc -g prog.c

Generate report

$ valgrind –leak-check=yes   ./a.out
can use 2>&1 to redirect report to a file( $ valgrind –leak-check=yes  ./a.out > report   2>&1 )

Analyzing report

Various error messages and summaries can be found. error messages are generated in case of out of bound writes, here a[10].
The corresponding report is
==4836== Invalid write of size 4
==4836==    at 0x80483FF: fun(prog.c:6)
==4836==    by 0x8048411: main (prog.c:11)
==4836==  Address 0x419a050 is 0 bytes after a block of size 40 alloc’d
==4836==    at 0x4024F20: malloc (vg_replace_malloc.c:236)
==4836==    by 0x80483F5: fun(prog.c:5)
==4836==    by 0x8048411: main (prog.c:11)
4836 is the process id. First line shows, error is due to an invalid write of size 4. Below it is a complete stack trace. The error happened at line 6 of  prog.c. Read stack trace from bottom to up. Started from main, then a function call to fun, malloc and error at last. Error shows the address we tried to write is beyond the allocated 40 byte block. This information is quite useful to make the code correct.

The Leak summery show the memory leaks.
Here,
==4836== LEAK SUMMARY:
==4836==    definitely lost: 40 bytes in 1 blocks
==4836==    indirectly lost: 0 bytes in 0 blocks
==4836==      possibly lost: 0 bytes in 0 blocks
==4836==    still reachable: 0 bytes in 0 blocks
==4836==         suppressed: 0 bytes in 0 blocks
Second line shows the 40 byte block lost in function fun. Report includes other types of leaks also.

Valgrind checks these errors and leaks in runtime like a virtual machine executing each instruction of a code. So it is time consuming for large code. But the report generated is very much useful and can be used to correct mistakes which are otherwise very difficult to detect.

October 19, 2010

Compiler optimizations

Filed under: C, GNU/Linux — Tags: , , , , — shijitht @ 5:01 pm

To improve the performance, compiler optimizes the code while compilation. Compiler inline optimization and common subexpression elimination are discussed here. The assembly code is given for further clarifications. Compiler does these optimizations on the basis of a cost/benefit calculation.
Compiler used GCC 4.4.3

Code inlining

Code inlining embeds the functions body in the caller. This eliminates call and return steps and helps to put some extra optimization in both codes.
Lets see the difference.
Compile opt1.c with and without optimization to generate assembly code.

opt1.c
-------
int sqr(int x)
{
 return x*x;
}

main()
{
 printf("%d\n", sqr(10));
}

Without optimization
$ cc  -S  opt1.c  -o  wout_opt1.s
With optimization
$ cc  -S  -O3  opt1.c  -o  with_opt1.s

Compare both files. The function call to sqr in wout_opt1.s is replaced with its value in with_opt1.s. The corresponding  lines are darkened.

wout_opt1.s
-------------
main:
 pushl   %ebp
 movl    %esp, %ebp
 andl    $-16, %esp
 subl    $16, %esp
 movl    $10, (%esp)
 call    sqrc
 movl    %eax, 4(%esp)
 movl    $.LC0, (%esp)
 call    printf
 leave
 ret

with_opt1.s
-----------
main:
 pushl    %ebp
 movl    %esp, %ebp
 andl    $-16, %esp
 subl    $16, %esp
 movl    $100, 4(%esp)
 movl    $.LC0, (%esp)
 call    printf
 leave
 ret

But the code for sqr remains in both .s file because it could be referenced by some other functions where inline optimization can’t be applied. Only the linker can detect and remove unreferenced functions.
In inlining, the value of the function is found while compilation instead of runtime. Call instruction is replaced by a move instruction which loads the immediate value to the required location. An immediate value($100) equivalent to function sqr can be seen here and the call statement removed.

Common subexpression elimination

Compiler scans the code and finds identical subexpressions. These are evaluated only once and replaced with a single variable holding its value.
For example, take opt2.c

opt2.c
-------
main()
{
 int i, j, k, r;

 scanf("%d%d", &i, &j);

 k = i + j + 10;

 r = i + j + 30;

 printf("%d %d\n", k, r);

}

opt2.c has the subexpression i + j.

Compile opt2.c with and without optimization

Without
$ cc  -S  opt2.c  -o  wout_opt2.s
With
$ cc  -O3  -S  opt2.c  -o  with_opt2.s

wout_opt2.s
------------
main:
 pushl   %ebp
 movl    %esp, %ebp
 andl    $-16, %esp
 subl    $32, %esp
 leal    24(%esp), %eax
 movl    %eax, 8(%esp)
 leal    28(%esp), %eax
 movl    %eax, 4(%esp)
 movl    $.LC0, (%esp)
 call    scanf
 movl    28(%esp), %edx
 movl    24(%esp), %eax
 leal    (%edx,%eax), %eax
 addl    $10, %eax
 movl    %eax, 20(%esp)
 movl    28(%esp), %edx
 movl    24(%esp), %eax
 leal    (%edx,%eax), %eax
 addl    $30, %eax
 movl    %eax, 16(%esp)
 movl    16(%esp), %eax
 movl    %eax, 8(%esp)
 movl    20(%esp), %eax
 movl    %eax, 4(%esp)
 movl    $.LC1, (%esp)
 call    printf
 leave
 ret

with_opt2.s
------------
main:
 same as above
 call    scanf
 movl    24(%esp), %eax
 addl    28(%esp), %eax
 movl    $.LC1, (%esp)
 leal    30(%eax), %edx
 addl    $10, %eax
 movl    %edx, 8(%esp)
 movl    %eax, 4(%esp)
 call    printf
 leave
 ret

In wout_opt2.s, two variables are read as usual. The value i + j is calculated in two places to add with 10 and 30. leal  (%edx,%eax),  %eax is to add i and j. Evaluating the expression twice wastes CPU time.
In optimized with_opt2.s, the first value read is stored in eax. It gets added with the value read next. Now eax the value of i + j. leal adds 30 to it and stores in edx. addl adds 10 and eax.
Common subexpression elimination is a powerful technique to optimize code performance. Programmers can eliminate such subexpressions while coding.  But there will be compiler generated expressions for array index calculation, macro expansion etc. A programmer can’t do optimization in this level. These are the cases where a compiler does its trick to improve performance.

October 14, 2010

Building a GNU/Linux system from scratch and running it with UML

Filed under: GNU/Linux, Projects — Tags: , , , , — shijitht @ 12:23 am

Other than installing the system from the precompiled binaries, we could build it from scratch. The scratch means, building every element that makes our system, entirely from source code !!!. This is a !!! now, but in older days this was a necessary installation routine for linux lovers. Since we are new to this process proper documentation is needed to build the system. A fantastic Linux From Scratch(LFS) project is there to help us. Go to http://www.linuxfromscratch.org/. With proper description of each step, this project is the best to start for a beginner.

After system build, we can use it on a physical drive as per the documentation. Compile a proper kernel and write it on a drive to boot from it. But this article covers using uml. UML means User Mode Linux. Run linux on top of current linux kernel instead of bare hardware. We can test the new system, above a uml kernel.  While running a uml kernel, we can specify virtual resources such as root file system, swap, hardware etc.

Why build from source ?

You will have a question that, why build from source when a lot distros are available, which could be installed easily with a few mouse clicks ?. The aim of installing from scratch is to make a proper understanding of our system software internals. You can see the use of each package, its contents, size and the commands and scripts to build it. With intelligent package managers, we are unaware of the horrible dependency relations between packages. When you finish compiling the system, you can also get an awareness about the effort people put in making it.

UML

Uml kernel runs on top of system kernel in user space. Being a user process, privileged direct hardware access cant be made in uml kernel. So these are converted to system’s native calls. These calls are usually in arch folder of kernel. This make the creation of UML kernel easier because only the arch section differs. Only that folder needs rewrite. Another simple technique is used to execute processes inside uml system. An executing process is traced using ptrace. On a system call the process is stopped and it’s called address is replaced by the address of system call in real kernel. Then the stopped process is continued.

Building UML kernel

Follow these steps to create a uml kernel binary.
Download a stable kernel source from kernel.org.
Untar and cd to that folder.
Give these commands in the command line.
# make defconfig ARCH=um
will produce a .config file with default configuration for uml.
# make menuconfig
if desired
# make menuconfig ARCH=um
to produce a custom kernel
# make mrproper
# make mrproper ARCH=um
to get rid of all traces of whatever building you did, and start over.
# make ARCH=um
starts build
On completion, you can see a uml binary called “linux” of size ~25M.
Strip off debugging symbols to cut size.
To test this kernel start it with ubda=< FILE SYSTEM > option.

Creating file system

We can use a virtual file system to check the above kernel.
Use dd command to create a file of 6 GB.
# dd if=/dev/zero of=fs bs=1G count=6
will copy 1G from /dev/zero 6 times measuring a total of 6G.
Now create a file system of type ext3 using mkfs.ext3.
# mkfs.ext3 fs
give yes when prompted and it will produce a virtual file system of type ext3.
Use loopback to mount it
# mount -o loop fs /mnt/lfs
will mount fs -> /mnt/lfs
Loop back device /dev/loop0 is used for mounting this file. Through this device we can use the file as we are accessing a block device.
Next step is to make all programs inside fs which will help us use the system.

Creating files(programs)

Our system needs a lot of programs and tools for proper functioning. This include the init process, the first process which loads at boot. Then bash to interact with user and various tools to do tasks like compilation(gcc). The proper documentation for building is given at linuxfromscratch.org. So repeating them again will be a waste of time. Go to LFS site and follow the step by step instructions to build a working system. Do the version check as per the documentation to make sure you meet all the prerequisites. Also some errors might happen if texinfo packages is missing. You can come back, when you finish setting up system boot scripts.

Booting in uml

Make some changes to what you have done.
Edit /etc/fstab
When booting with uml we specify ubda as the root file system device.
So add it to fstab instead of what is there as root device.
/dev/ubda    /    ext3    defaults    1    1
Edit /dev/inittab
Comment all agetty lines and add this line instead
1:2345:respawn:/sbin/agetty tty0 9600
Now you are ready to boot with uml.

Power ON

Unmount the file system
# umount /mnt/lfs
Use the uml kernel produced(“linux”)  to start the system
# ./linux ubda=< PATH TO fs >
This will take you to a new virtual system build with your hands.

Conclusion

Building a Gnu/linux system from scratch is an experience which will take you to each and every corner of your system. Familiarizing with many commands and scripting techniques will increase knowledge. The role of each package is clearly understood while building from source. Now we know, what each package has and depends on. So later a much lighter system can be built with a custom configuration as per demands. Very small(~4M) systems can be built, which could be used in embedded systems. If we use uml, no restart is needed to test or boot a system. Also an exposure to virtualization can be gained with uml. LFS project has a documentation that any beginner can follow. Building a linux system from scratch is something that every linux enthusiast must do.

September 26, 2010

Vertex coloring problem app powered by Ruby On Rails

Filed under: Projects — Tags: , , , , — shijitht @ 8:39 pm

Rails is a powerful framework for developing dynamic web applications. It is written in Ruby. Ruby is a must to write application back-end. It follows Model View Controller(MVC) to organize application. This app is to solve vertex coloring problem using Rubyonrails.

The problem is, no two adjacent nodes in the graph should have the same color. The input is a graph and output its colored form. The output is shown in the browser using HTML 5 and JavaScript. Heroku provides free hosting for rails apps. So I also covered using git and heroku to deploy it on-line.

Ruby On Rails

Rails framework eases the creation of dynamic and rich web applications. It allows a developer to start from scratch or from inbuilt templates. Use scaffold to create various templates. Rails got a way of its own to develop applications. First we need to master it. If we wont read the manual, experiences with other frameworks and web development will lead us to trouble. Motto of rails to do more with very less effort.

MVC in rails

Splitting the application into MVC will isolate logic from view, code remains clean and will make it clear where different types of code belong for easier maintenance.

M – Model

Model represents the data and the rules to manipulate that data. Models are used for interaction with the database. i.e. One model for a table in our app. In rails interaction between application and database is achieved using active records.

Active records is an ORM(Object Relation Mapping) programming technique. ORM means mapping of objects between program and database for storage and processing. Actually rails uses an extension of active records. Unlike other mapping, we don’t have to specify database schema. Just edit a few lines in database configuration file like user, password, socket etc and also specify a migration file to make changes while updating. SQLite is the default database, but we can change it to MySql or whatever in need by changing the above two files.

V – View

It represents the user interface of application. They are HTML files with embedded ruby code .

C – Controllers

Controllers process the incoming requests from the web browser, communicate with models for data and pass that data to views for presentation.

Getting Started

Installation

  1. Install ruby from your distro repo
  2. gem from source tar
  3. install rails using gem.

Gem manages ruby packages and libraries. More detailed installation steps can be found at http://railstutorials.org/book/.

Setup

To start a new project

  1. $ rails new <project name>
  2. Use ‘bundle install’ to install all dependencies specified in Gemfile.
  3. $ rails generate controller home index
    This will create a basic template to start a project. To test it, start server.
  4. $ rails server
    starts server at http://localhost:3000. It shows a rail welcoming page.

To replace it with the new application index page follow these steps.

  • remove public/index.html
  • set default root route in /config/routes.rb
    i.e. Uncomment the root :to => line and change it to “home#index”

Development

The app folder holds the MVC files of the app. The controller have the controller file. It will have a default action, index. You can edit it to add new actions. To reflect addition of new actions, these should be added to to the routes.rb file. Each action can have methods. For each action, corresponding view should be placed in views folder. The URL is of the form http://localhost:3000/controller/action/method/.

Views folder hosts html files and layout. By default, /public/javascript/application.js and /public/stylesheet/application.css can be used to store js and css. To include a specific file, layout in views can be used. Give the file name in double quotes near stylesheet_link_tag and javascript_include_tag.

Params can be used to receive values posted using POST. eg: to receive a parameter with name area use params[ :area ] or params[ “area” ]. The processed values can be embedded to the html page using <%= and %> tags. The control or loop expressions are used inside <% and %> and to use a value of a variable use <%= and %>.

Deploying online

Heroku provides a free hosting for rails applications. To use it, create a heroku account from http://heroku.com/. For installation do,

  1. $ sudo gem install heroku
  2. Now move to your git folder for rail app.
  3. $ heroku create
    Give credentials when prompted and follow the link shown to go to your app page.
  4. $ git push heroku master
    This will upload app.

Now go to the previous link to test application online.

Conclusion

Rails is a very good framework to develop web applications in a shot time and with less effort. The MVC model keeps the design simple and database can be used without worrying about schema. In built template system gives more time for user specific development. And using git and heroku, application can be loaded online. To check my code and its online presentation go to:

Git – http://github.com/shijith/ColoringProblem-Rails/
Heroku – http://afternoon-wind-21.heroku.com/

September 20, 2010

Vertex coloring problem app in Google app engine

Filed under: Projects — shijitht @ 12:37 am

Google App Engine lets you create and deploy apps on the same machine which servers google apps like Gmail. As we all know, it is a cloud infrastructure. Sandbox is the name of the virtual environment(machine), for running our apps. Our aim is to create an app to solve graph(vertex) coloring problem.

The problem is, no two adjacent nodes in the graph should have the same color. The input is a graph and  output  its colored form. The output is shown in the browser and use of HTML 5, JavaScript and Django is gonna make it interesting.

Google App Engine

Goole App Engine allows your apps to run in the sandbox. Like said, it is a secure box. It got some restrictions like can’t write to filesystem, only read files uploaded by the app, use specified APIs to get access to resources and services and there is a dead line for responses. Violation of these limits will end in an exception. And the free account limits a user in other ways like the usual storage limit, connection limit. Go visit the site for more details.

The app engine now supports two runtime, python and java. I prefer python, so I gone a stick to it. The sandbox has a standard python interpreter. The libraries which conflicts with the above mentioned policies are removed. It provides a data model to store data and api for managing it. The web frameworks like Django are also supported. As the site has a more detailed documentation for theses things, lets move to what I have done.

Getting started

Google App Engine resides in a remote server. So there will be question about how to develop offline. The emulation of app engine can be produced offline using App Engine SDK. It is available online. Install it and conform the requirements like run time support are satisfied in your system.

Development

Start the project by creating a folder for it. The app consists of mainly three type of files. app.yaml, project.py and index.html. This is the case for a simple project. To bring order in a little bigger project, the css, js, image etc folders could be included and more html pages for better functionality. The app.yaml file should be updated to reflect these changes.

app.yaml is the configuration file of app engine. It specifies the runtime, version, url handling etc. The python script does all data processing and transactions. To provide an MVC, the html elements are separated from the script(model) and put in separate html files(view). The current situation gives a lot of web tools and frameworks to develop front end. And app engine supports most of the frameworks. The SDK comes with inbuilt webapp and Django support. I used a mix of HTML 5, JavaScript, CSS, Django as front-end and python, app engine as back-end.

Canvas element in HTML 5 provided the canvas and images were drawn with JavaScript. CSS styles the pages and Django allows an easy integration with server for the creation of dynamic pages. You can find the code at http://github.com/shijith/ColoringProblem. This app is hosted under http://coloringproblem.appspot.com. For an example, see an instance of coloring

Some ideas

While developing a clicking application like this, a certain doubt is; How could you know, whether the click was on a circle or not. The answer is, for a circle at (0,0)

x^2 + y^2 = r^2 .

Let the circle be at (x1,y1) and click at (x2,y2). A point inside the circle should satisfy,

(x2 - x1)^2 + (y2 - y1)^2 <= r^2 .

Conclusion

Google app engine provides a very fast and reliable server for our apps and at no cost. HTML 5 includes a lot of in build elements for the creation of heavy graphics applications with JavaScript. The recent developments in browser technology fastened (300X) JavaScript execution and hence providing for the overall performance. With CSS 3 more effects can be added with less effort like round curves or a box. App engine dashboard provides detailed error report which eases error detection and debugging.

August 26, 2010

Pidgin invisibility cloak

Filed under: Tools — Tags: , , , , — shijitht @ 1:54 pm

Go Invisible option in pidgin is an option only. It will set status to busy. But there is a solution for it.

Go online.
Enable the XMPP console in Tools > Plugins.
Now select Tools->XMMP Console->XMMP Console.
Check your gmail account is selected in the console.
Put these code in the second box and hit enter.
Don’t mind the trail of code in the monitor, but a hacker could find it pretty useful.

{{{
<presence>
<priority>5</priority>
</presence>
<presence type="unavailable">
<priority>5</priority>
</presence>
}}}

Now go invisible.
To reset change unavailable to available and put it in the console.
For more go to http://www.madhusudancs.info/invisible.

August 22, 2010

Brief note about using Git

Filed under: Commands, Tools — Tags: , , — shijitht @ 7:51 pm

This is a short note including everyday commands. You can find more details like installing git and more commands at http://help.github.com/.
Don’t forget $ man git.

Create a repo graphically and do the following to get started.

The first section is to set up your name and email as a committer.

“git commit” allows to commit locally with  comment specified using -m option.
Here $ git commit -m “first commit” will make your local repo a step ahead of remote repo.
Check status using $ git status .
At first we need to connect to the remote repo. Origin is the remote repo-name used here, I suggest github or whatever name say your real repo name.
Now push using git push. Here master branch pushed to origin(remote repo-name).
To start tracking/add a new file → $ git add <file name> .
To commit to the remote repo → $ git push <repo name>  <branch name>.

Default branch choice is master.
To see branches → $ git branch .
Add -a option if you wish to see remote branches also.
To add a new branch for parallel developing → $ git branch <branch name> .
Move to the new branch → $ git checkout <branch name> .
To push a new branch →
$ git push <remote repo name>  <local branch name> :<remote branch name>.

If you use both branch names same, the ” <local branch name>: ” is optional.
To delete a branch locally → $ git branch -d .
To delete remote branch → $ git push <repo name>   :<remote branch name> .
i.e. nothing pushed( local section blank ).
To add a tag → $ git tag <tag name>
To update remote repo → $ git push   – – tags
To remove tags → $ git push <repo name>  :<tag name> .
Creating a folder demands adding files under it and use git rm in case of file deletion. So better keep file names distinct.
Use git clone/fetch to create a local copy of a repo.
To remove a repo do it graphically.

Useful links :
http://www.gitready.com/
http://scottr.org/presentations/git-in-5-minutes/

August 21, 2010

GitHub key update delay

Filed under: Commands — Tags: , , , , — shijitht @ 10:35 pm

Changing the current rsa public key or adding a new key wont work as soon as you do it. The usual delay I get is about 10-30 min. So don’t get frustrated and change keys continuously. Keep patience and wait. Do
$ ssh git@github.com   in intervals until confirmation. The success message is

ERROR: Hi shijith! You've successfully authenticated, but GitHub does not
provide shell access

August 19, 2010

New Twitter “Tweet Button”

Filed under: WordPress — Tags: , , — shijitht @ 4:50 pm

tweetbutton

You can get a new tweet button after each post, just like “like button”. This will enable easy sharing of posts among readers.

Go to Appearance > Extra and enable the option for Tweet Button.😀

Older Posts »

Blog at WordPress.com.