AMPL/Linux - Setting Environment Variables

If your PATH environment variable doesn't contain the path to the solvers, AMPL complains that it couldn't find the solver:

ampl: option solver minos;
ampl: solve;
Cannot invoke minos: No such file or directory
exit code 4

You need to add the path to the solver executables in the PATH:

export PATH="$PATH":/home/user/.../Solvers

Similarly, if your model contains an external function you need to set the environment variable AMPLFUNC. It should be pointing at the shared object file (i.e., amplfunc.so), or the directory which contains this file.

export AMPLFUNC=/home/user/.../AMPL/amplfunc.so

Kernel Logging, Log Ring Buffer, printk(), syslog-ng

Kernel Log (Ring) Buffer

Linux kernel generates log messages using printk(). These messages are stored in a "ring buffer". The size of this buffer is controlled by a kernel configuration parameter:

CONFIG_LOG_BUF_SHIFT

The default value of this parameter is 14, which means 2^14 bytes, thus 16KB. The size of the buffer can not be changed online so it should be modified (if you'd like to have a larger or smaller buffer) before compiling the kernel (more information on kernel compilation). (1)

printk() - print() function at the Kernel level

printk() is used to print messages at the kernel level. The size of the message can not be larger than 1KB. Below is a sample printk() statement:

printk( KERN_INFO "message\n");

One can classify kernel messages according to their importance/priorities. Priority of a print statement is given by a log level macro. There are 8 priority levels defined in the kernel:

• KERN_EMERG - emergency the highest level
• KERN_ALERT, KERN_CRIT, KERN_ERR, KERN_WARNING, KERN_NOTICE, KERN_INFO, and
• KERN_DEBUG - debugging messages with the lowest priority
  

For more information on printk(): Linux Device Drivers, 3rd Edition.

How are Kernel Log Messages Exposed to User Space?

The log buffer is exposed to user through /proc/kmsg file. If the file is read one can catch the kernel log messages. In fact, there are available programs to display the ingredients of the file /proc/kmsg, and log the content of this file in a static file (e.g., klogd and syslogd deamons, and syslog-ng logging facility). /proc is a memory filesystem (it contains virtual files which reveals the current state of the running Linux kernel), and the content of /proc/kmsg are being overwritten. In addition, reading from /proc/kmsg is destructive; that's, once you read a line/message it's removed from the file.(2)

"syslog-ng" (system log new generation) is a widely used logging application in Linux systems. It can directly read from /proc/kmsg and log the messages into static file(s). "syslog-ng" can be regarded as the upgraded version of the old kernel deamon "syslogd".

syslog-ng

syslog-ng can be configured so that messages can be directed based on their priorities. That means you direct different level of messages to different files.

syslog-ng can be configured using the configuration file:

• /etc/syslog-ng/syslog-ng.conf (in SuSE Linux, you need to change syslog-ng.conf.in file which is used to generate syslog-ng.conf automatically by SuSE config).
  

In this configuration file, you define:

• the sources syslog-ng is using: for example, /proc/kmsg, /dev/log, etc.
  
• filters to identify the priority of a message or the facility from which a message is originated (news, mail, etc.)
• destination files to direct the messages belonging to certain group (based on filters)
• Finally, with "log" statements, you combine {source, filter, destination} to specify where to log which messages.

You can find some sample configuration files here, and some more information on logging with syslog-ng here.

Tuning syslog-ng for Performance

In "Monitoring Block I/O at Linux FileSystem Level" project, I am logging information on each block I/O using printk() function and syslog-ng logging facility. In our experiments, we can produce a trace file (i.e., a log file) of size 1 GB within couple of hours. As a result, to be able to configure HOW we log kernel messages into a static trace file is critical, we don't want to hurt the performance by producing too many I/O operations at a high rate.

However, using syslog-ng, you can control how you log the messages into a file. While defining a destionation file, you can set

• log_fifo_size: log buffer size (in terms of number of messages)
• fsynch(no): by saying "no", syslog-ng will not issue fflush() for each of the message received from the source; otherwise, it'd be catastrophic for the performance.
• flush_lines & flush_timeout: syslog-ng will flush either flush_lines many message are collected in the log buffer of the destination or flush_timeout is passed since the last flushing.
  

As a result, you can control the rate at which you write the syslog-ng destination files.

Example

Below is a simple example for a syslog-ng configuration file:

# defining a source: /proc/kmsg -> kernel messages
source my_source { file("/proc/kmsg" log_msg_size(1024)); };

# defining a filter: kernel messages with the level of KERN_DEBUG
filter f_myfilter { facility(kern) and priority(debug); };

# defining a destination file: it defines a log file (i.e., a.txt) which will have its
# own buffer with a size of 10000 messages. Messages are flushed from buffer
# to the log file if 8000 messages are collected in the buffer of 5 second is past
# since the last flushing.
destination my_destination { file("/home/user/a.txt" log_fifo_size(10000) fsync(no) flush_lines(8000) flush_timeout(5000) );};

# Finally, a logging point is defined using above source, filter, and destination:
log { source(my_source); filter(f_myfilter); destination(my_destination); };

Foot Notes:

(1) Some information may be kernel version dependant. I've considered Linux kernel 2.6.21 in this document.
(2) dmesg which also examines the kernel ring buffer is on the other hand non-destructive.

How to Assign PassWd to root

If you haven't assigned a password to the superuser "root", you can do so by using the command:

sudo passwd root

It asks for a password, what you entered will be the initial password for the superuser root.

Compiling a Linux (Vanilla) Kernel

Here's how I've compiled vanilla kernel for my SuSE 10.0 (1). A vanilla kernel is the one you can download from www.kernel.org,which does not include any modification by any distribution (e.g., Suse, Debian, etc.). Normally, a distribution adds its own patches, facilities to this core kernel.

Likewise, you can also add any patchset supported by the linux version you are about to compile. For example, ARA (adaptive readahead) is not a stock option as of version 2.6.21 but there is a patchset if you'd like to have ARA feature on your Linux box. Or else, you'd like to upgrade to an upper version you may apply the new version's patch, which are called prepatches.

Basically below are the steps you would normally go through when compiling a linux kernel source:

1. Get the kernel source from kernel.org
2. Untar the download file in /usr/src directory. This will create a new directory under /usr/src, which would be like linux-2.6.21.x
3. Create a symbolic link named "linux" to this new directory again under /usr/src:
• /usr/src/linux -> /usr/src/linux-2.6.21.x
  
4. Apply the patches if you have any supported by the version you are about to compile
• Untar the patch file under /usr/src/linux
• I'd definitely recommend you first do a dry-run. A dry-run will not actually add the patch but kind of simulate it. You'd have a chance to see whether any failure occurred. You can find more information on how to apply patches to a kernel source tree here.
  
• patch --dry-run -p1 < "patch_name"
5. Now, it's cofiguration time. Before compiling the kernel you need to configure it. Basically, you select which drivers to include, change any default parameter, etc.
   
   It's a very common practice to use the already running kernel's configuration file because it is probably configured to include all the necessary drivers, etc. The running kernel's configuration file is found under the /boot directory. Hence, copy the existing config file (probably named as /boot/config-2.6.x.y) under /usr/src/linux:
   

   cp /boot/config-`uname -r` /usr/src/linux/.config (2)

6. Use "make menuconfig" to start configuring your new kernel. This command brings up a menu. You need to first "load" the config file using the "Load an Alternate Configuration Final" menu item. This will load the old config file.
   Afterwards, you can go through the menu tree and configure your kernel. For example, as mentioned in kernel log buffer subject, you may want to change the size of the kernel log ring buffer, which can only be modified at kernel compilation time. You can do it by Choosing "Kernel Hacking" menu item and then change the "Kernel Log Buffer Size" menu item in the new menu.
   
   
7. We have the source tree, we have configured the kernel parameters /drivers/etc; finally, it's time to build/compile the kernel. At this point, there is separate paths to follow depending on your Linux distribution. In Suse, you create an rpm file. In Ubuntu and Debian, you need to compile the kernel source into a deb package. "Linux HowTos and Tutorials" web site has a very nice set of documents for compiling kernel for each most popular linux distros.
• SuSE way: make rpm
• Ubuntu way: "make-kpkg clean" and then "make deb-pkg"
• You can go to http://www.howtoforge.com/, and search how you can build a kernel for your Linux distro.
  
  
8. Normally, compilation takes half an hour to one or one and a half hours depending on the speed of your box. After building is done, it's time to install the newly built kernel.
   Again in above source, you can find how to carry out installation process for your Linux distro.
   

Footnotes:

(1) You can learn the version of your distribution from /etc/issue
(2) uname command prints the system information. "uname -r" gives the version of the running kernel.

How to reach Windows partition from Linux, and vice-versa

How to Mount a Windows NTFS file system partition in Linux:

1. fdisk -l to see which partition has the NTFS file system. For example, let's say, it's /dev/hda1
   
2. mkdir /mnt/windows create a directory to mount the NTFS filesystem.
   
3. mount /dev/hda1 /mnt/windows/ -t ntfs -o nls=utf8,umask=0222
• -t option: type of the filesystem to be mounted
• with -o option, you can specify several option seperated by commas.
• with nls option, you can specify the io character set
• using umask, set the file permission on the file system (the value is in octal). By default, only root has the access, above option gives permission to other users.
  
4. You can access your Windows files using the directory /mnt/windows (which may correspond to C:\, D:\, etc.)
5. To unmount the Windows NTFS partiton: umount /mnt/windows/

How to Access Files on your Linux Partition From Windows:

I find "explore2fs" very effective and simple if you'd like to access ext partitions. No installation needed.

How to Import User-Defined Functions to AMPL?

AMPL is a mathematical programming language for describing optimization problems. There are quite a number of solvers that accept AMPL model as input. It's a powerful language but sometimes you may need to have your own function to include in your AMPL model. Just to note, if your model includes a user-defined function, it's not possible to use NEOS solvers.

NETLIB provides some explanation on how to import user-defined functions to AMPL but I found the related information rather scattered. Here, I'll try to walk through the process of importing a very simple user-defined function.

First of all, user-defined functions are imported to AMPL through "amplfunc.dll" shared library. If this file exists then AMPL loads the defined functions. AMPL looks for this dll file in the execution directory and the directory (or file) given as a command-line option using "-i".

The same shared library makes user defined functions availbale both to AMPL and to solvers. If AMPL finds a shared-library, it sets the $AMPLFUNC environment variable to inform the linked solvers.

A function can be in one of the 3 categories: deterministic, symbolic (string) and random. Since solvers directly igonore functions returnming strings and non-deterministic functions I'll write a deterministic function.

Below is a very simple function which sums up the given two arguments and sends back the result:

1. You need to download the header file from Netlib which inlcudes necessary definitions; for example, the defition of the structure "struct arglist" is found in this header file. You'll include it in your c program.
   
   
2. Then, you'll begin to code your function; note that you may have more than 1 function to import. In Visual C/C++, you need to explicilty export the function you want to use in AMPL:
   
   http://msdn.microsoft.com/en-us/library/ms235636%28VS.80%29.aspx
   
   __declspec(dllexport) real foo(arglist *al){
   
   /* the return type is real, which is nothing but a double: see the definition in header file you downloaded */
   
   /* Now, you'll get the arguments passed by AMPL */
   
   real arg1 = al->ra[0];
   real arg2 = al->ra[1];
   
   /* return the result */
   return (arg1+arg2);
   
   }
   
   
3. Another function you have to export is "funcadd". It'll be used by AMPL to get the information about the user-defined function(s), in this case it's just foo.
   
   __declspec(dllexport) void funcadd(AmplExports *ae){
   
   /* 1st param: name of the function to import to AMPL.
   2nd param: pointer to the function "foo"
   3rd param: function type...you can get more info from the header file by checking enumarated type FUNCADD_TYPE.
   4th param: Number of arguments passed to function "foo".
   5th param: funcinfo
   */
   addfunc("foo", (rfunc)foo, 0, 2, 0);
   }
   
   Again, NETLIB includes a more complicated sample: funcadd.c .
   
   
4. After creating dll, you can name it "amplfunc.dll" and copy it under the execution directory of AMPL.
   
   
5. Lastly, you have to declare the function in your model before using it:
   
   ....
   function foo;
   ...
   subject to X: ... foo(param1, var1)...;
   
   

TortoiseCVS - Does it crash when you try to update or commit?

I found TortoiseCVS one of the best CVS clients to be used at a WindowsXP end. However, I had some hard times at the beginning. When I was trying to use CVS update/commit, it was crashing and XP's famous error reporting message box was poping up afterwards.

In my case, the cvs client couldn't connect to the server because of authorization issues. The solution was to add an environment variable:

• CVS_RSH which points to the Plink executable (Plink is the one which tries to establish the client/server connection); for example, C:\Program Files\TortoiseCVS\TortoisePlink.exe.

Adding an environment variable in XP is an easy task:

• Right Click on "My Computer" -> Properties -> Select "Advanced" tab -> Click on "Environment Variables" at the bottom of the window -> Add CVS_RSH as a system variable.