Tech Corner

Ubuntu 8.10 gdm AutoStart at Boot

2010-06-04T16:26:00.000-07:00

/etc/rcS.d contains the scripts that are executed at boot time. These are in general symbolic links to the scripts in /etc/init.d

gdm start/stop script is found in /etc/init.d/. If you're missing the link to this script in /etc/rcS.d then it is a reason why your gdm does not start when your linux box boots.

ln -s /etc/init.d/[conf_file] /etc/rcS.d/[name_it].sh

How to Pass an Opened Socket (or File Descriptor) to a Forked Process in Python/Windows

2009-08-17T18:08:00.000-07:00

Client Side

The client side program creates 5 threads, and each thread communicates with the server.

import socket, threading
import ctypes

NUMBER_OF_CLIENT_THREADS = 5
HOST = 'localhost'
PORT = 7777
BUF_SIZE = 100

class Client(threading.Thread):

def run(self):

buf = ctypes.create_string_buffer(BUF_SIZE)

# open a conn. to the server
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect((HOST, PORT))

# send your message and recive a response
client.send('Hello, I am {0}'.format(self.getName()))
buf.value = '\0' * BUF_SIZE
data = client.recvfrom_into(buf, BUF_SIZE)
print '\tServer:', buf.value

# say your last words
client.send('Goodbye Server!')
buf.value = '\0' * BUF_SIZE
data = client.recvfrom_into(buf, BUF_SIZE)
print '\tServer:', buf.value

# exit
client.close()

def main():

thread_list = []

for i in xrange(NUMBER_OF_CLIENT_THREADS):
thread_list.append( Client() )
thread_list[i].setName("thread-"+repr(i))
thread_list[i].start()

for thr in thread_list:
thr.join()

if __name__ == "__main__":
main()

Server Side

Server forks a new child process for each connection, and child processes communicate with the client threads.

import socket, signal, os, multiprocessing, time
from ctypes import *

#
# Structures needed to duplicate a socket:
#
kernel32 = windll.kernel32
Ws2_32 = windll.Ws2_32
msvc = cdll.msvcrt

class GUID(Structure):
_fields_ = [("Data1", c_ulong),
("Data2", c_ushort),
("Data3", c_ushort),
("Data4", c_char * 8)]

class WSAPROTOCOLCHAIN(Structure):
_fields_ = [("ChainLen", c_int),
("ChainEntries", c_ulong * 7)]

class WSAPROTOCOL_INFO(Structure):
_fields_ = [("dwServiceFlags1",c_ulong),
("dwServiceFlags2",c_ulong),
("dwServiceFlags3",c_ulong),
("dwServiceFlags4",c_ulong),
("dwProviderFlags",c_ulong),
("ProviderId",GUID),
("dwCatalogEntryId",c_ulong),
("ProtocolChain", WSAPROTOCOLCHAIN),
("iVersion",c_int),
("iAddressFamily",c_int),
("iMaxSockAddr",c_int),
("iMinSockAddr",c_int),
("iSocketType",c_int),
("iProtocol",c_int),
("iProtocolMaxOffset",c_int),
("iNetworkByteOrder",c_int),
("iSecurityScheme",c_int),
("dwMessageSize",c_ulong),
("dwProviderReserved",c_ulong),
("szProtocol",c_char* 256)]

BUF_SIZE = 100

def WorkerProcess(conn_child):

buf = create_string_buffer(BUF_SIZE)

# communicate with the parent process to duplicate the opened socket with client
pid = os.getpid()
conn_child.send(pid)

wsock_prot_info = conn_child.recv()
# (AF_INET, SOCK_STREAM, IPPROTO_TCP, x, group, flag)
sock_no = Ws2_32.WSASocketA(2, 1, 6, byref(wsock_prot_info), 0, 0)

if(sock_no == c_uint(~0)):
print "Error in duplicating the socket"
conn_child.send(1)
exit(1)

conn_child.send(0) # everthing has gone OK!

# Now we have obtained the socket, go on to communicate with the client
while 1:

buf.value = '\0' * len(buf)
ret = Ws2_32.recv(sock_no, buf, len(buf), 0)
if ret == 0 or ret == -1:
break
print 'Client:' + buf.value

buf.value = '\0' * len(buf)
buf.value = "OK"
ret = Ws2_32.send(sock_no, buf, len(buf), 0)
if ret == -1: # SOCKET_ERROR
print "Error in sending data"
break

msvc._close(sock_no)

print "Worker Process Exiting..."
exit(1)

def main():

HOST = ''
PORT = 7777
wsock_prot_info = WSAPROTOCOL_INFO()

# 1. Initialize ZFS
print "Server is starting..."

# 2. Set up the server:
server = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
server.bind ( (HOST, PORT) )
server.listen ( 5 )

# 3. Listen for client connections:
while 1:

conn, addr = server.accept()
print 'New connection with', addr

#
# WE WANT TO PASS THE OPENED SOCKET TO THE NEWLY FORKED PROCESS:
#

# 1. open a pipe to realize comm. bw parent and the forked process
conn_parent, conn_child = multiprocessing.Pipe(True)

# 2. fork a process and hand the one side of the pipe to the child
new_worker = multiprocessing.Process(target=WorkerProcess, args=(conn_child,))
new_worker.start()

# 3. Child sends back its process id
cpid = conn_parent.recv()
print "New Child Process with pid {0} to handle the new client".format(cpid)

# 4. duplicate the opened socket and obtain prorocol info structure
if Ws2_32.WSADuplicateSocketA(conn.fileno(), c_ulong(cpid), byref(wsock_prot_info)) != 0:
print "Error in duplicating the socket"

# 5. Send this structure to the child, it'll use this info to create its own handle
conn_parent.send(wsock_prot_info)

# 6. Get the confirmation from the child
ret = conn_parent.recv()
if(ret!=0):
print "Error in handing in the socket to the child process"

# 7. Close connection with the client (child will handle that) and the pipe.
conn_parent.close()
conn.close()

if __name__ == "__main__":
main()

How to Compile Your Own Module

2009-03-26T14:11:00.000-07:00

Below is a sample module to compile (my_module.c):

#include linux/init.h
#include linux/module.h

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Oguzhan Ozmen");
MODULE_DESCRIPTION("My simple Hello World module");

static int module_init(void){

printk(KERN_ALERT "Hello World\n");

return 0; // success
}

static void __exit module_exit(void){

printk(KERN_ALERT "Goodbye\n");
}

module_init(module_init);
module_exit(module_exit);

Within the directory where my_module.c resides in, create a Makefile whose content is as follows:

obj-m += monitor_blockio.o

Then, generate the kernel object file (.ko) using the following command:

make -C [build_dir] M=$PWD modules

Here, [build_dir] stands for the directory which contains the necessary files to build new kernel modules:

- Makefile
- .config
- module.symVers (module symbol information)
- kernel header files (include/ and include/asm)

In general, [build_dir] is:

- /lib/modules/$(uname -r)/build OR
- /usr/src/packages/BUILD/kernel-$(uname -r)

(assuming that you are using a distro compiled kernel)

How to Generate a Patch File

2008-10-22T19:30:00.000-07:00

If you need to generate a patch file for a set of file(s) residing in a directory structure or just for one file, then you can use "diff" command. You can find a very nice article in Linux Magazine named "What's the diff?". Basically, this command finds out the difference between two files.

If you want to generate a patch for a directory structure:

diff -rau "old_dir" "modified_dir" > "patch_file"

r: recursive
a: treat all files as text.
u: Output NUM (default 3) lines of unified context.

In case of two file, ignoring -r would be enough.
a

How to Prevent "sudo" from Asking a Password

2008-10-22T10:36:00.000-07:00

Some commands like "echo 3 > /proc/sys/vm/drop_caches"* require superuser priviliges so (if you are just a user and you are a sudoer) you need to sudo.

"sudo" asks for a password. However, if you're "sudo"ing in a script then it's a problem: you wouldn't want to reveal your password as a text.

You can work around so that "sudo" doesn't require a password:

Edit "/etc/sudoers": ALL=(ALL) NOPASSWD: ALL

* NOTE:

echo 1 > drop_caches : just to free "pagecache" (memory which is used to cache the user data pages)
echo 2 > drop_caches: just to free dcache (which stores recently generated dentries which are created when a pathname is resolved) and inodes (which contain metadata for files).
echo 3 > drop_caches: remove all caches

AMPL/BONMIN (Linux/Ubuntu)

2008-10-05T22:20:00.000-07:00

IMPORTANT LINKS FOR BONMIN

Below are the websites which are helpful in explaining how to download/install BONMIN in a linux box:

(getting started) https://projects.coin-or.org/Bonmin/wiki/GettingStarted
https://projects.coin-or.org/BuildTools/wiki/downloadUnix

The latest version of BONMIN can be downloaded from

http://www.coin-or.org/download/source/Bonmin/

CAUTION

Although, it's claimed that BONMIN is tested with "gcc version 3.* and 4.* up to 4.3", I encountered a compiler/linker error:

ERROR: ....rodata' defined in discarded section `.gnu.linkonce...

I've been using gcc-3.4.6. After googling, I found that it's because of a bug which is fixed in gcc-4.*. After switching to gcc-4.0, the error is gone.

SUMMARY

After downloading and untarring the BONMIN file, the main directory contains all the necessary tools (e.g., Bonmin, IpOpt, Cbc, CoinUtils, etc.). However, some 3rd party programs which are required for installation and run are not included (because of licencing issues).

The directories, however, for these 3rd party programs are already available:

.../Bonmin-0.100.2/ThirdParty
HSL
Blas
Lapack
ASL

You only need to download the source codes into the corresponding directories:

.../Bonmin-0.100.2/ThirdParty/Blas$ ./get.Blas
.../Bonmin-0.100.2/ThirdParty/Lapack$ ./get.Lapack
.../Bonmin-0.100.2/ThirdParty/ASL$ ./get.ASL
For HSL: goto http://hsl.rl.ac.uk/archive/hslarchive.html

- Fill in "Registration Form". You don't need to activate your membership! After acquiring the username/passwd just go ahead!
- In the "Package(s) Required'' part, enter 'MA27'.
- Login the "HSL Archieve" with the acquired username and password.
- You will need the double precision versions of the MA27 (you don't need to download the dependencies such as ID05 and ZA02), MC19. In the HSL directory, you should save to files named ma27ad.f and mc19ad.f respectively.

Finally, create a new director called "build" under the main directory:

mkdir .../Bonmin-0.100.2/build

And, within the "build", execute the following commands

../configure -C
make
make test
make install

"make install" will create the below directories under "build":

bin
lib
share
include

"bin" contains the executable "bonmin" and the solvers it uses: Cbc, Clp, and Ipopt. "lib" contains the neccesary libarary files for "bonmin" to run.

Create a symbolic link named "bonmin" (which points at .../Bonmin-0.100.2/build/bin/bonmin) under the directory where "ampl" executable resides so that ampl can use BONMIN as a solver.

AMPL/Linux - Setting Environment Variables

2008-10-05T22:11:00.000-07:00

If your PATH environment variable doesn't contain the path to the solvers, AMPL complains that it couldn't find the solver:

ampl: option solver minos;
ampl: solve;
Cannot invoke minos: No such file or directory
exit code 4

You need to add the path to the solver executables in the PATH:

export PATH="$PATH":/home/user/.../Solvers

Similarly, if your model contains an external function you need to set the environment variable AMPLFUNC. It should be pointing at the shared object file (i.e., amplfunc.so), or the directory which contains this file.

export AMPLFUNC=/home/user/.../AMPL/amplfunc.so

Kernel Logging, Log Ring Buffer, printk(), syslog-ng

2008-09-16T09:24:00.000-07:00

Kernel Log (Ring) Buffer

Linux kernel generates log messages using printk(). These messages are stored in a "ring buffer". The size of this buffer is controlled by a kernel configuration parameter:

CONFIG_LOG_BUF_SHIFT

The default value of this parameter is 14, which means 2^14 bytes, thus 16KB. The size of the buffer can not be changed online so it should be modified (if you'd like to have a larger or smaller buffer) before compiling the kernel (more information on kernel compilation). (1)

printk() - print() function at the Kernel level

printk() is used to print messages at the kernel level. The size of the message can not be larger than 1KB. Below is a sample printk() statement:

printk( KERN_INFO "message\n");

One can classify kernel messages according to their importance/priorities. Priority of a print statement is given by a log level macro. There are 8 priority levels defined in the kernel:

KERN_EMERG - emergency the highest level
KERN_ALERT, KERN_CRIT, KERN_ERR, KERN_WARNING, KERN_NOTICE, KERN_INFO, and
KERN_DEBUG - debugging messages with the lowest priority

For more information on printk(): Linux Device Drivers, 3rd Edition.

How are Kernel Log Messages Exposed to User Space?

The log buffer is exposed to user through /proc/kmsg file. If the file is read one can catch the kernel log messages. In fact, there are available programs to display the ingredients of the file /proc/kmsg, and log the content of this file in a static file (e.g., klogd and syslogd deamons, and syslog-ng logging facility). /proc is a memory filesystem (it contains virtual files which reveals the current state of the running Linux kernel), and the content of /proc/kmsg are being overwritten. In addition, reading from /proc/kmsg is destructive; that's, once you read a line/message it's removed from the file.(2)

"syslog-ng" (system log new generation) is a widely used logging application in Linux systems. It can directly read from /proc/kmsg and log the messages into static file(s). "syslog-ng" can be regarded as the upgraded version of the old kernel deamon "syslogd".

syslog-ng

syslog-ng can be configured so that messages can be directed based on their priorities. That means you direct different level of messages to different files.

syslog-ng can be configured using the configuration file:

/etc/syslog-ng/syslog-ng.conf (in SuSE Linux, you need to change syslog-ng.conf.in file which is used to generate syslog-ng.conf automatically by SuSE config).

In this configuration file, you define:

the sources syslog-ng is using: for example, /proc/kmsg, /dev/log, etc.
filters to identify the priority of a message or the facility from which a message is originated (news, mail, etc.)
destination files to direct the messages belonging to certain group (based on filters)
Finally, with "log" statements, you combine {source, filter, destination} to specify where to log which messages.

You can find some sample configuration files here, and some more information on logging with syslog-ng here.

Tuning syslog-ng for Performance

In "Monitoring Block I/O at Linux FileSystem Level" project, I am logging information on each block I/O using printk() function and syslog-ng logging facility. In our experiments, we can produce a trace file (i.e., a log file) of size 1 GB within couple of hours. As a result, to be able to configure HOW we log kernel messages into a static trace file is critical, we don't want to hurt the performance by producing too many I/O operations at a high rate.

However, using syslog-ng, you can control how you log the messages into a file. While defining a destionation file, you can set

log_fifo_size: log buffer size (in terms of number of messages)
fsynch(no): by saying "no", syslog-ng will not issue fflush() for each of the message received from the source; otherwise, it'd be catastrophic for the performance.
flush_lines & flush_timeout: syslog-ng will flush either flush_lines many message are collected in the log buffer of the destination or flush_timeout is passed since the last flushing.

As a result, you can control the rate at which you write the syslog-ng destination files.

Example

Below is a simple example for a syslog-ng configuration file:

# defining a source: /proc/kmsg -> kernel messages
source my_source { file("/proc/kmsg" log_msg_size(1024)); };

# defining a filter: kernel messages with the level of KERN_DEBUG
filter f_myfilter { facility(kern) and priority(debug); };

# defining a destination file: it defines a log file (i.e., a.txt) which will have its
# own buffer with a size of 10000 messages. Messages are flushed from buffer
# to the log file if 8000 messages are collected in the buffer of 5 second is past
# since the last flushing.
destination my_destination { file("/home/user/a.txt" log_fifo_size(10000) fsync(no) flush_lines(8000) flush_timeout(5000) );};

# Finally, a logging point is defined using above source, filter, and destination:
log { source(my_source); filter(f_myfilter); destination(my_destination); };

Foot Notes:

(1) Some information may be kernel version dependant. I've considered Linux kernel 2.6.21 in this document.
(2) dmesg which also examines the kernel ring buffer is on the other hand non-destructive.

How to Assign PassWd to root

2008-09-06T05:58:00.001-07:00

If you haven't assigned a password to the superuser "root", you can do so by using the command:

sudo passwd root

It asks for a password, what you entered will be the initial password for the superuser root.

Compiling a Linux (Vanilla) Kernel

2008-09-05T12:13:00.000-07:00

Here's how I've compiled vanilla kernel for my SuSE 10.0 (1). A vanilla kernel is the one you can download from www.kernel.org,which does not include any modification by any distribution (e.g., Suse, Debian, etc.). Normally, a distribution adds its own patches, facilities to this core kernel.

Likewise, you can also add any patchset supported by the linux version you are about to compile. For example, ARA (adaptive readahead) is not a stock option as of version 2.6.21 but there is a patchset if you'd like to have ARA feature on your Linux box. Or else, you'd like to upgrade to an upper version you may apply the new version's patch, which are called prepatches.

Basically below are the steps you would normally go through when compiling a linux kernel source:

Get the kernel source from kernel.org
Untar the download file in /usr/src directory. This will create a new directory under /usr/src, which would be like linux-2.6.21.x
Create a symbolic link named "linux" to this new directory again under /usr/src:

/usr/src/linux -> /usr/src/linux-2.6.21.x

Apply the patches if you have any supported by the version you are about to compile

Untar the patch file under /usr/src/linux
I'd definitely recommend you first do a dry-run. A dry-run will not actually add the patch but kind of simulate it. You'd have a chance to see whether any failure occurred. You can find more information on how to apply patches to a kernel source tree here.

patch --dry-run -p1 < "patch_name"

Now, it's cofiguration time. Before compiling the kernel you need to configure it. Basically, you select which drivers to include, change any default parameter, etc.

It's a very common practice to use the already running kernel's configuration file because it is probably configured to include all the necessary drivers, etc. The running kernel's configuration file is found under the /boot directory. Hence, copy the existing config file (probably named as /boot/config-2.6.x.y) under /usr/src/linux:
cp /boot/config-`uname -r` /usr/src/linux/.config (2)
Use "make menuconfig" to start configuring your new kernel. This command brings up a menu. You need to first "load" the config file using the "Load an Alternate Configuration Final" menu item. This will load the old config file.
Afterwards, you can go through the menu tree and configure your kernel. For example, as mentioned in kernel log buffer subject, you may want to change the size of the kernel log ring buffer, which can only be modified at kernel compilation time. You can do it by Choosing "Kernel Hacking" menu item and then change the "Kernel Log Buffer Size" menu item in the new menu.
We have the source tree, we have configured the kernel parameters /drivers/etc; finally, it's time to build/compile the kernel. At this point, there is separate paths to follow depending on your Linux distribution. In Suse, you create an rpm file. In Ubuntu and Debian, you need to compile the kernel source into a deb package. "Linux HowTos and Tutorials" web site has a very nice set of documents for compiling kernel for each most popular linux distros.

SuSE way: make rpm
Ubuntu way: "make-kpkg clean" and then "make deb-pkg"
You can go to http://www.howtoforge.com/, and search how you can build a kernel for your Linux distro.

Normally, compilation takes half an hour to one or one and a half hours depending on the speed of your box. After building is done, it's time to install the newly built kernel.
Again in above source, you can find how to carry out installation process for your Linux distro.

Footnotes:

(1) You can learn the version of your distribution from /etc/issue
(2) uname command prints the system information. "uname -r" gives the version of the running kernel.

How to reach Windows partition from Linux, and vice-versa

2008-08-15T11:59:00.000-07:00

How to Mount a Windows NTFS file system partition in Linux:

fdisk -l to see which partition has the NTFS file system. For example, let's say, it's /dev/hda1
mkdir /mnt/windows create a directory to mount the NTFS filesystem.
mount /dev/hda1 /mnt/windows/ -t ntfs -o nls=utf8,umask=0222

-t option: type of the filesystem to be mounted
with -o option, you can specify several option seperated by commas.
with nls option, you can specify the io character set
using umask, set the file permission on the file system (the value is in octal). By default, only root has the access, above option gives permission to other users.

You can access your Windows files using the directory /mnt/windows (which may correspond to C:\, D:\, etc.)
To unmount the Windows NTFS partiton: umount /mnt/windows/

How to Access Files on your Linux Partition From Windows:

I find "explore2fs" very effective and simple if you'd like to access ext partitions. No installation needed.

How to Import User-Defined Functions to AMPL?

2008-07-30T18:16:00.000-07:00

AMPL is a mathematical programming language for describing optimization problems. There are quite a number of solvers that accept AMPL model as input. It's a powerful language but sometimes you may need to have your own function to include in your AMPL model. Just to note, if your model includes a user-defined function, it's not possible to use NEOS solvers.

NETLIB provides some explanation on how to import user-defined functions to AMPL but I found the related information rather scattered. Here, I'll try to walk through the process of importing a very simple user-defined function.

First of all, user-defined functions are imported to AMPL through "amplfunc.dll" shared library. If this file exists then AMPL loads the defined functions. AMPL looks for this dll file in the execution directory and the directory (or file) given as a command-line option using "-i".

The same shared library makes user defined functions availbale both to AMPL and to solvers. If AMPL finds a shared-library, it sets the $AMPLFUNC environment variable to inform the linked solvers.

A function can be in one of the 3 categories: deterministic, symbolic (string) and random. Since solvers directly igonore functions returnming strings and non-deterministic functions I'll write a deterministic function.

Below is a very simple function which sums up the given two arguments and sends back the result:

You need to download the header file from Netlib which inlcudes necessary definitions; for example, the defition of the structure "struct arglist" is found in this header file. You'll include it in your c program.
Then, you'll begin to code your function; note that you may have more than 1 function to import. In Visual C/C++, you need to explicilty export the function you want to use in AMPL:

http://msdn.microsoft.com/en-us/library/ms235636%28VS.80%29.aspx

__declspec(dllexport) real foo(arglist *al){

/* the return type is real, which is nothing but a double: see the definition in header file you downloaded */

/* Now, you'll get the arguments passed by AMPL */

real arg1 = al->ra[0];
real arg2 = al->ra[1];

/* return the result */
return (arg1+arg2);

}
Another function you have to export is "funcadd". It'll be used by AMPL to get the information about the user-defined function(s), in this case it's just foo.

__declspec(dllexport) void funcadd(AmplExports *ae){

/* 1st param: name of the function to import to AMPL.
2nd param: pointer to the function "foo"
3rd param: function type...you can get more info from the header file by checking enumarated type FUNCADD_TYPE.
4th param: Number of arguments passed to function "foo".
5th param: funcinfo
*/
addfunc("foo", (rfunc)foo, 0, 2, 0);
}

Again, NETLIB includes a more complicated sample: funcadd.c .
After creating dll, you can name it "amplfunc.dll" and copy it under the execution directory of AMPL.
Lastly, you have to declare the function in your model before using it:

....
function foo;
...
subject to X: ... foo(param1, var1)...;

TortoiseCVS - Does it crash when you try to update or commit?

2008-07-30T10:32:00.000-07:00

I found TortoiseCVS one of the best CVS clients to be used at a WindowsXP end. However, I had some hard times at the beginning. When I was trying to use CVS update/commit, it was crashing and XP's famous error reporting message box was poping up afterwards.

In my case, the cvs client couldn't connect to the server because of authorization issues. The solution was to add an environment variable:

CVS_RSH which points to the Plink executable (Plink is the one which tries to establish the client/server connection); for example, C:\Program Files\TortoiseCVS\TortoisePlink.exe.

Adding an environment variable in XP is an easy task:

Right Click on "My Computer" -> Properties -> Select "Advanced" tab -> Click on "Environment Variables" at the bottom of the window -> Add CVS_RSH as a system variable.