Rel = "icon" href = "../../ Common / Images / LF-16.png" TYPE = "Image / PNG">
LF343, UNIXBASICS: why does this not work !? how to find and fix fish !? how to find and fix faults in linux applications.
<- | Home | MAP | Index | Search
News | Archives | LINKS | About LF
This Document Is Available in: English Deutsch Francais Russian Turkce
By Guido Socher (Homepage) About The Author:
GUIDO LIKES The Possibilities That An Open Source System Like Linux Offers To Investigate Problems. You Can Really Find The Root Cause Given That You Are Motivated to Invest Time.
CONTENT:
IntroductionLogSstracegdb and Core FilesConclusionTalkback Form for this article
Why does this not work !? how to find and fix faults in linux applications.
Abstract:
Everybody claims that it is easy to find and fix bugs in programs written under Linux. Unfortunately it is very hard to find documents explaining how to do that. In this article you will learn how to find and fix faults without first learning how an application internally Works.
_________________________________________________
Introduction From a user perspective there is hardly any difference between closed and open source systems as long as everything runs without faults and as expected. The situation changes however when things do not work and sooner or later every computer user will come to the point where things Do Not Work.
In a closed source system you have usually only two option:
Report the fault and pay for the fix Re-install and pray that it works now Under Linux you have these options too but you can also start and investigate the cause of the problem. One of the main obstacles is usually that you are not the author of the failing program and that you have really no clue how it works internally.Despite those obstacles there are a few things you can do without reading all the code and without learning how the program works internally.
Logs The Most Obvious and Simplest Thing You Can Do Is To Look At File In / Var / Log / ... What You Find in Those Files And What The names of those logs files area is configurable. / Var / log / message is usually The file you want to look at. Bigger Applications May Have Their Own Log Directories (/ var / log / httpd / / / var / log / exim ...).
.
Logging works such that the designer of an program can add a syslog line to his code This is much like a printf except that it writes to the system log In this statement you specify a priority and a facility to classify the message..:
#include
Void OpenLog (const char * ident, int option, int facility);
Void syslog (int priority, const char * format, ...);
Void CloSelog (Void);
Facility Classifies The Type of Application Sending The Message.
Priority Determines The Importance of The Message. Possible
VALUES in Order of Importance Are:
Log_emerg
LOG_Alert
Log_crit
Log_err
Log_warning
LOG_NOTICE
LOG_INFO
Log_debug
With this C-interface any application written in C can write to the system log Other languages do have similar APIs Even shell scripts can write to the log with the command:.. Logger -p err "this text goes to / var / log / Messages
A Standard Syslog Configuration (File /etc/syslog.conf) SHOULD HAVE AMONG OTHERS A LINE THAT LOOKS LIKE THIS:
# Log anything (Except mail) of level info or higher.
# Don't log private authentication messages.
* .info; mail.none; authpriv.none / var / log / messages
THE "* .INFO" WILL log_info or higher. To see more information in / var / log / Messages You can change this to "* .debug" and restart syslog (/etc/init.d/syslog restart ).
The Procedure to "Debug" An Application Would Therefore Be as Follows.
1) Run Tail -f / var / log / message and then start the application which
Fails from a Different Shell. Maybe you get already some hints
Of what is going wrong.
2) IF step 1) Is Not Enough Then Edit /etc/syslog.conf and
Change * .info to * .debug. Run "/etc/init.d/syslog rest" and
REPEAT STEP 1).
The problem with this method is that it depends entirely on what the developer has done in his code. If he / she did not add syslog statements at key points then you may not see anything at all. In other words you can find only problems where The Developer Did Already Foresee That Could Go Wrong.
Strace An Application Running Under Linux CAN EXECUTE 3 TYPE OF FUNCTION:
Functions somewhere in its own code Library functions System calls Library functions are similar to the application's own functions except that they are provided in a different package. System calls are those functions where your program talks to the kernel. Programs need to talk to the kernel if they need to access you computer's hardware That is:. write to the screen, read a file from disk, read keyboard input, send a message over the network etc ... These system calls can be intercepted and you can therefore follow the communication between Application and the kenergy.
A common problem is that an application does not work as expected because it can not find a configuration file or does not have sufficient permissions to write to a directory. These problems can easily be detected with strace. The relevant system call in this case would BE CALLED "Open".
You use strace like this:
Strace Your_Application
Here is an esample:
# straape / usr / sbin / uuCico
Execve ("/ usr / sbin / uuCico", ["/ usr / sbin / uuCico", "-s", "uucpssh", "-x", "11"],
[/ * 36 VARS * /]) = 0
Uname ({sys = "linux", node = "brain", ...}) = 0
BRK (0) = 0x8085e34
Mmap2 (NULL, 4096, Prot_read | Prot_Write,
Map_private | map_anonymous, -1, 0) = 0x40014000
Open ("/ etc / ld.so.preload", o_rdonly) = -1 Enoent (no sudh file or directory)
Open ("/ etc / ld.so.cache", o_rdonly) = 3
FSTAT64 (3, {ST_MODE = S_IFREG | 0644, ST_SIZE = 70865, ...}) = 0
Mmap2 (NULL, 70865, Prot_read, Map_Private, 3, 0) = 0x40015000
CLOSE (3) = 0
Open ("/ lib / libnsl.so.1", o_rdonly) = 3
READ (3, "/ 177ELF / 1/1/1/0/0/0/0/0/0/0/0/0/3/0/3/0/1/0/0/0/300; / 0 "..., 1024) = 1024
FSTAT64 (3, {ST_MODE = S_IFREG | 0755, ST_SIZE = 89509, ...}) = 0
Mmap2 (NULL, 84768, Prot_read | Prot_exec, Map_Private, 3, 0) = 0x40027000
MProtect (0x40039000, 11040, prot_none) = 0
Mmap2 (0x40039000, 4096, prot_read | prot_write, map_private | map_fixed, 3, 0x11)
= 0x40039000
MMAP2 (0x4003A000, 6944, Prot_read | Prot_Write,
Map_private | map_fixed | map_anonymous, -1, 0) =
0x4003a000
CLOSE (3) = 0
Open ("/ lib / libc.so.6", o_rdonly) = 3
READ (3, "/ 177ELF / 1/1/1/0/0/0/0/0/0/0/0/0/3/0/3/0/1/0/0 / 0`X / 1/000 "..., 1024)
= 1024
FSTAT64 (3, {ST_MODE = S_IFREG | 0755, ST_SIZE = 1465426, ...}) = 0
Mmap2 (NULL, 1230884, Prot_read | Prot_exec, Map_Private, 3, 0) = 0x4003C000
MProtect (0x40163000, 22564, prot_none) = 0
Mmap2 (0x40163000, 12288, prot_read | prot_write,
Map_private | map_fixed, 3, 0x126) = 0x40163000
Mmap2 (0x40166000, 10276, prot_read | prot_write,
Map_private | map_fixed | map_anonymous, -1, 0) = 0x40166000
CLOSE (3) = 0
Munmap (0x40015000, 70865) = 0
BRK (0) = 0x8085e34
BRK (0x8086e34) = 0x8086e34
BRK (0) = 0x8086e34
BRK (0x8087000) = 0x8087000
Open ("/ usr / conf / uucp / config", o_rdonly) = -1 Enoent (no sudh file or directory)
RT_SIGAction (Sigint, Null, {SIG_DFL}, 8) = 0
Rt_sigAction (sigint, {0x806a700, [],
SA_RESTORER | SA_INTERRUPT, 0X40064D58}, NULL, 8) = 0
Rt_sigAction (Sighup, Null, {SIG_DFL}, 8) = 0
Rt_sigAction (Sighup, {0x806A700, [], SA_RESTORER | SA_INTERRUPT, 0X40064D58}, NULL, 8) = 0
RT_SIGAction (SIGQUIT, NULL, {SIG_DFL}, 8) = 0
Rt_sigAction (SIGQUIT, {0x806A700, [],
SA_RESTORER | SA_INTERRUPT, 0X40064D58}, NULL, 8) = 0
RT_SIGAction (Sigterm, Null, {SIG_DFL}, 8) = 0
Rt_sigAction (Sigterm, {0x806A700, [],
SA_RESTORER | SA_INTERRUPT, 0X40064D58}, NULL, 8) = 0
Rt_sigAction (SIGPIPE, NULL, {SIG_DFL}, 8) = 0
Rt_sigAction (sigpipe, {0x806a700, [],
SA_RESTORER | SA_INTERRUPT, 0X40064D58}, NULL, 8) = 0
GetPid () = 1605
Gtrlimit (rlimit_nofile, {RLIM_CUR = 1024, RLIM_MAX = 1024}) = 0
Close (3) = -1 EBADF (Bad File Descriptor)
Close (4) = -1 EBADF (Bad File Descriptor)
Close (5) = -1 EBADF (Bad file descriptor)
Close (6) = -1 EBADF (Bad File Descriptor)
Close (7) = -1 EBADF (Bad file descriptor)
Close (8) = -1 EBADF (Bad file descriptor)
Close (9) = -1 EBADF (Bad file descriptor)
FCNTL64 (0, f_getfd) = 0
FCNTL64 (1, f_getfd) = 0
FCNTL64 (2, f_getfd) = 0
Uname ({sys = "linux", node = "brain", ...}) = 0
umask (0) = 022
Socket (PF_Unix, Sock_Stream, 0) = 3
Connect (3, {SA_FAMILY = AF_UNIX,
PATH = "/ var / run / .nscd_socket"}, 110) = -1 Enoent (no so file or directory)
CLOSE (3) = 0
Open ("/ etc / nsswitch.conf", o_rdonly) = 3fstat64 (3, {ST_MODE = S_IFREG | 0644, ST_SIZE = 499, ...}) = 0
Mmap2 (NULL, 4096, Prot_read | Prot_Write,
Map_private | map_anonymous, -1, 0) = 0x40015000
Read (3, "# /etc/nsswitch.conf:/n# $ header:" ..., 4096) = 499
READ (3, ", 4096) = 0
CLOSE (3) = 0
Munmap (0x40015000, 4096) = 0
Open ("/ etc / ld.so.cache", o_rdonly) = 3
FSTAT64 (3, {ST_MODE = S_IFREG | 0644, ST_SIZE = 70865, ...}) = 0
Mmap2 (NULL, 70865, Prot_read, Map_Private, 3, 0) = 0x40015000
CLOSE (3) = 0
Open ("/ lib / libnss_compat.so.2", o_rdonly) = 3
READ (3, "/ 177ELF / 1/1/1/0/0/0/0/0/0/0/0/0/3/0/3/0/1/0/0/0/300 / 25 "..., 1024)
= 1024
FSTAT64 (3, {ST_MODE = S_IFREG | 0755, ST_SIZE = 50250, ...}) = 0
Mmap2 (NULL, 46120, Prot_read | prot_exec, map_private, 3, 0) = 0x40169000
MProtect (0x40174000, 1064, prot_none) = 0
Mmap2 (0x40174000, 4096, prote_read | prot_write,
Map_private | map_fixed, 3, 0xa) = 0x40174000
CLOSE (3) = 0
Munmap (0x40015000, 70865) = 0
Uname ({sys = "linux", node = "brain", ...}) = 0
BRK (0) = 0x8087000
BRK (0x8088000) = 0x8088000
Open ("/ etc / passwd", o_rdonly) = 3
FCNTL64 (3, f_getfd) = 0
FCNTL64 (3, f_setfd, fd_cloexec) = 0
FSTAT64 (3, {ST_MODE = S_IFREG | 0644, ST_SIZE = 1864, ...}) = 0
Mmap2 (NULL, 4096, Prot_read | Prot_Write,
Map_private | map_anonymous, -1, 0) = 0x40015000
_llseek (3, 0, [0], seek_cur) = 0read (3, "root: x: 0: 0: root: / root: / bin / bash / n" ..., 4096) = 1864
CLOSE (3) = 0
Munmap (0x40015000, 4096) = 0
Getuid32 () = 10
getEuid32 () = 10
Chdir ("/ var / spool / uucp") = 0
Open ("/ usr / conf / uucp / sys", o_rdonly) = -1 Enoent (no sudh file or directory)
Open ("/ var / log / uucp / debug", o_wronly | o_append | o_creat | o_nOctty, 0600) = 3
FCNTL64 (3, f_getfd) = 0
FCNTL64 (3, f_setfd, fd_cloexec) = 0
FCNTL64 (3, f_getfl) = 0x401 (Flags O_Wronly | O_APpend)
FSTAT64 (3, {ST_MODE = S_IFREG | 0600, ST_SIZE = 296, ...}) = 0
Mmap2 (NULL, 4096, Prot_read | Prot_Write,
Map_private | map_anonymous, -1, 0) = 0x40015000
_llseek (3, 0, [0], seek_cur) = 0
Open ("/ var / log / uucp / log", o_wronly | o_append | o_creat | o_nockte, 0644) = 4
FCNTL64 (4, f_getfd) = 0
FCNTL64 (4, f_setfd, fd_cloexec) = 0
FCNTL64 (4, f_getfl) = 0x401 (Flags O_Wronly | O_APPEND)
What Do We See Here? Let's Look E.G Look at The Following Lines:
Open ("/ etc / ld.so.preload", o_rdonly) = -1 Enoent (no sudh file or directory)
Open ("/ etc / ld.so.cache", o_rdonly) = 3
The program Tries to read /etc/ld.so.preload and fails1.so.cache. Here it succeeds and gets file descriptor 3 Allocated. Now the Failure to Read / etc / ld. so.preload may not be a problem at all because the program may just try to read this and use it if possible. In other words it is not necessarily a problem if the program fails to read a file. It all depends on the design of The Program. Let's Look At All The Open Calls in The PrintOut from Strace: Open ("/ USR / conf / uucp / config", o_rdonly) = -1 Enoent (no sudh file or directory)
Open ("/ etc / nsswitch.conf", o_rdonly) = 3
Open ("/ etc / ld.so.cache", o_rdonly) = 3
Open ("/ lib / libnss_compat.so.2", o_rdonly) = 3
Open ("/ etc / passwd", o_rdonly) = 3
Open ("/ usr / conf / uucp / sys", o_rdonly) = -1 Enoent (no sudh file or directory)
Open ("/ var / log / uucp / debug", o_wronly | o_append | o_creat | o_nOctty, 0600) = 3
Open ("/ var / log / uucp / log", o_wronly | o_append | o_creat | o_nockte, 0644) = 4
Open ("/ etc / ld.so.preload", o_rdonly) = -1 Enoent (no sudh file or directory)
Open ("/ etc / ld.so.cache", o_rdonly) = 3
The Program Tries Now To Read / USR / Conf / UuCP / Config. OH! This is Strange I Have The Config File In / etc / uucp / config! ... and there is no line where the program attempts to open / etc / UUCP / Config. This is the fault. Obviously The program for the Wrong location of the configuration file.
As you see strace can be prary useful. The problem is this ketput of strait ket...................
gdb and core files Sometimes it happens that a program just dies out of the blue with the message "Segmentation fault (core dumped)". This means that the program tries (due to a programming error) to write beyond the area of memory it has allocated. Especially in cases where the program writes just a few bytes to much it can be that only you see this problem and it happens only once in a while. This is because memory is allocated in chunks and sometimes there is accidently still room left for the extra bytes.When this "Segmentation fault" happens a core file is left behind in the current working directory of the program (normally your home directory). This core file is just a dump of the memory at the time when the fault happened. Some shells provide facilities for controlling whether core files are written Under bash, for example, the default behavior is not to write core files at all In order to enable core files, you should use the command..:
# ulimit -c unlimited
# ./lshref -i index.html, index.htm test.html
Segmentation Fault (Core Dumped)
EXIT 139
The Core File Can Now Be Used with the GDB Debugger to Find Out What Was Going Wrong. Before You Start GDB You Can Check That You Are Really Looking At The Right Core File:
# file core.16897
Core.16897: ELF 32-BIT LSB Core File Intel 80386, Version 1 (Sysv), SVR4-Style,
From 'lshref'
OK, lshref is the program that was crashing so let's load it into gdb. To invoke gdb for use with a core file, you must specify not only the core filename but also the name of the executable that goes along with that core file.
# gdb ./lshref core.23061
GNU GDB Linux (5.2.1-4)
CopyRight 2002 Free Software Foundation, Inc.
GDB Is Free Software, Covered by The GNU General Public License, And You '
Type "Show Copying" to see the conditions.
There Is Absolutely No Warranty for GDB. Type "Show Warranty" for Details.
Core Was generated by `./lshref -i index.html, index.htm test.html '.
Program Terminated with Signal 11, Segmentation Fault.
Reading symbols from /lib/libc.so.6...done.
Loaded Symbols for /Lib/Libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded Symbols for /Lib/ld-Linux.so.2
# 0 0x40095e9d in strcpy () from /lib/libc.so.6
(GDB)
Now We know what The program is crashing while it Tries to do a strcpy. The question. The code where straces in the code where strcpy is used.
In General There Will Now BE 2 Possibilities To Find Out Where Exactly In The Code It Goes WRONG.
Recompile the code with debug information (gcc option -g) Do stack trace in gdb The problem in our case is strcpy is a library function and even if we would re-compile absolutely all code (including libc) it would still tell us that that IT fails at a given line in the c library.
What we need is a stack trace which will tell us which function was called before strcpy was executed. The command to do such a stack trace in gdb is called "backtrace". It does however not work with only the core file. You have to RE-Run The Command in GDB (Reproduce the FAULT):
gdb ./lshref core.23061
GNU GDB Linux (5.2.1-4)
CopyRight 2002 Free Software Foundation, Inc.
GDB IS Free Software, Covered by the gnu general public license, and you are
Welcome to change IT and / or or distribute copies of it under certain conditions.
Type "Show Copying" to see the conditions.there is Absolutely no warranty for gdb. Type "Show Warranty" for details.
Core Was generated by `./lshref -i index.html, index.htm test.html '.
Program Terminated with Signal 11, Segmentation Fault.
Reading symbols from /lib/libc.so.6...done.
Loaded Symbols for /Lib/Libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded Symbols for /Lib/ld-Linux.so.2
# 0 0x40095e9d in strcpy () from /lib/libc.so.6
(GDB) BackTrace
# 0 0x40095e9d in strcpy () from /lib/libc.so.6
Cannot Access Memory At Address 0xBffFeb38
(GDB) Run ./lshref -i index.html, index.htm test.html
Starting program: / home / guido / lshref ./lshref -i index.html, index.htm test.html
Program received Signal SigSegv, Segmentation Fault.
0x40095e9d in strcpy () from /lib/libc.so.6
(GDB) BackTrace
# 0 0x40095e9d in strcpy () from /lib/libc.so.6
# 1 0x08048d09 in string_to_list ()
# 2 0x080494c8 in main ()
# 3 0x400374ed in __libc_start_main () from /Lib/libc.so.6
(GDB)
NOW We can see, function main () called string_to_list () is called. We go to string_to_list () and look at the code:
Char ** string_to_list (char * String) {
Char * dat;
CHAR * CHPTR;
Char ** array;
INT i = 0;
Dat = (char *) Malloc (Strlen) 5000;
Array = (char **) Malloc (sizeof (char *) * 51);
STRCPY (DAT, STRING);
This Malloc Line Looks Like A Typo. Probably It Should Have Been:
Dat = (char *) Malloc (Strlen 5000);
We change it, re-compile and ... hurra ... it works.
Let's look at a second example where the fault is not detected inside a library but in application code. In such a case the application can be compiled with the "gcc -g" flag and gdb will be able to show the exact line where the fault IS detected.here is a simple example.
#include
#include
Int Add (int * p, int a, int b)
{
* p = a b;
Return (* P);
}
Int main (void)
{
INT I;
INT * P = 0; / * a null pointer * /
Printf ("Result IS% D / N", Add (p, 2, 3));
Return (0);
}
We Compile IT:
gcc -wall -g -o exMp exmp.c
Run it ...
# ./exmp
Segmentation Fault (Core Dumped)
EXIT 139
GDB EXMP CORE.5302
GNU GDB Linux (5.2.1-4)
CopyRight 2002 Free Software Foundation, Inc.
GDB IS Free Software, Covered by the gnu general public license, and you are
Welcome to change IT and / or or distribute copies of it under certain conditions.
Type "Show Copying" to see the conditions.
There Is Absolutely No Warranty for GDB. Type "Show Warranty" for Details.
Core was generated by `/exmp '.
Program Terminated with Signal 11, Segmentation Fault.
Reading symbols from /lib/libc.so.6...done.
Loaded Symbols for /Lib/Libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded Symbols for /Lib/ld-Linux.so.2
# 0 0x08048334 in add (p = cannot access memory at address 0xbffe020
AT EXMP.C: 6
6 * p = a b;
GDB TELLS US Now That The Fault Was Detected At line 6 and this Pointer "P" Pointed to Memory Which CAN NOT BE Accessed.
WE LOOK AT THE ABODE AND IS OF Course A Simple Made-Up Example Where P IS A NULL POINTER AND You CAN NOT INY DATA IN A NULL POINTER. Easy to FIX ...
Conclusion
We have seen cases where you can really find the cause of a fault without knowing too much about the inner workings of a program.I have on purpose excluded functional faults, eg a button in a GUI is in the wrong position but it works. In Those Cases You Will Have to Learn About The Inner Workings of The Program. This Will Generally Take Much More Time and There Is No Recipe On How To Do That.
However the Simple Fault Finding Techniques Shown Here Can Still Be Be Applied In Many Situations.
Happy Troubleshooting!
Talkback Form for this article:
Talkback Page
BACK to the index of this issue
WebPages Maintained by the Linuxfocus Editor Team © Guido Socher, FDL Linuxfocus.org
Translation Information:
En -> -: Guido Socher (HomePage)
2004-08-16, Generated by LfParser Version 2.47
