Jim Meyering [Wed, 17 Aug 2011 08:27:29 +0000 (10:27 +0200)]
fts: do not exhaust memory when processing million-entry directories
Before this change, processing (via rm -rf, find, du, etc.) an N-entry
directory would require about 256*N bytes of memory. Thus, it was
easy to construct a directory too large to be processed by any of
those tools. With this change, fts' maximum memory utilization is
now limited to around 30MB.
* lib/fts.c (FTS_MAX_READDIR_ENTRIES): Define.
(fts_read): When we've processed the final entry (i.e., when
->fts_link is NULL) and fts_dirp is non-NULL, call fts_build
using the parent entry to read any remaining entries. Dispatch
depending on what fts_build returns:
- NULL+stop, aka failure: stop
- NULL otherwise: move up in the dir hierarchy
- non-NULL: handle this new entry
(fts_build): Declare and use new local, continue_readdir.
Prepare to be called from fts_read, when the entries
from a partially-read directory have just been exhausted.
In that case, we'll skip the opendir and instead use the parent's
fts_dirp and derive dir_fd from that.
Finally, in the readdir loop, if we read max_entries entries,
exit the loop ensuring *not* to call closedir. This is required
so that fts_dirp can be reused on a subsequent call.
Prompted by Ben England's report of memory exhaustion in find
and rm -rf vs. NFS: https://bugzilla.redhat.com/719749.
Jim Meyering [Wed, 17 Aug 2011 07:20:41 +0000 (09:20 +0200)]
fts: add/use new struct member, fts_dirp
We are about to use this to manage any directory with
too many entries to read all of them into memory at once.
To do that, we'll need to save the DIR* pointer in each
affected FTSENT struct.
* lib/fts_.h: Include <dirent.h>.
(struct FTSENT) [fts_dirp]: New member.
* lib/fts.c (closedir_and_clear): Define.
Use it in place of closedir so that we are sure to
clear the new fts_dirp member when done with it.
(fts_alloc): Initialize the new member.
(fts_lfree): Free, if needed.
Jim Meyering [Tue, 16 Aug 2011 18:20:30 +0000 (20:20 +0200)]
maint: fts: give __opendir2 a new parameter and rename
* lib/fts.c (__opendir2): Give it a new parameter, Pdir_fd, rather
than surreptitiously using sole caller's "dir_fd".
(fts_opendir): Rename from __opendir2.
Paul Eggert [Thu, 18 Aug 2011 07:46:17 +0000 (00:46 -0700)]
regex: port to Stratus OpenVOS
* lib/regex_internal.h (internal_function) [!_LIBC]: Simply
define to empty, rather than attempting nonportable optimizations.
Problem reported by Paul Green in:
http://lists.gnu.org/archive/html/bug-diffutils/2011-08/msg00047.html
and fix suggested by Eric Blake in:
http://lists.gnu.org/archive/html/bug-gnulib/2011-08/msg00143.html
Eric Blake [Wed, 17 Aug 2011 23:51:47 +0000 (17:51 -0600)]
getcwd: fix test failures on mingw
The GPL getcwd replacement now kicks in for mingw thanks to the
signature check, but does not have to do anything. However,
because the code was not taking an early exit for ERANGE when
a buffer size was given, it instead tried to second-guess
mingw's cwd algorithm, which doesn't work. After fixing that,
the tests still failed, even though mingw getcwd doesn't have
any problems with long paths (since they can't be created in
the first place).
* lib/getcwd.c (__getcwd): Early exit for ERANGE.
* tests/test-getcwd.c (test_abort_bug, test_long_name): Don't fail
test if long directory cannot be created, and allow mingw errno.
Eric Blake [Wed, 17 Aug 2011 22:14:12 +0000 (16:14 -0600)]
getcwd-lgpl: fix m4 to match relaxed test for BSD
Commit c6dc8f1 changed test-getcwd-lgpl to allow BSD behavior
where getcwd(NULL,1) allocates the proper size, in addition to
the glibc behavior of allocating exactly one byte then failing
with ERANGE. But the m4 test was still rejecting the former
behavior.
However, merely fixing the test to avoid probing the behavior
would be a regression of commit 8585ce2, since it was the only
reason why rpl_getcwd was being compiled on mingw. Hence, we
have to add a signature check in its place to keep mingw happy.
* m4/getcwd.m4 (gl_FUNC_GETCWD_NULL): Relax probe.
(gl_FUNC_GETCWD_SIGNATURE): New macro.
(gl_FUNC_GETCWD_LGPL, gl_FUNC_GETCWD): Use it.
* doc/posix-functions/getcwd.texi (getcwd): Tweak mentions of
signature problem.
Eric Blake [Wed, 17 Aug 2011 20:13:35 +0000 (14:13 -0600)]
getcwd: fix compilation on mingw64
Mingw 64 added <direct.h> as a place that declared getcwd. It is
a non-standard header, but since we already include it in the
replacement for <sys/stat.h>, we must also include it in the
replacement for <unistd.h> to avoid compilation problems.
* lib/unistd.in.h (includes) [mingw]: Include <direct.h> for
getcwd.
Reported by Marc-André Lureau.
Ben Pfaff [Tue, 16 Aug 2011 05:33:25 +0000 (22:33 -0700)]
relocatable-prog: fix link error
* modules/relocatable-prog (configure.ac) [RELOCATABLE]: Also
invoke AC_LIBOBJ([relocatable]). This invocation was previously
in the gl_RELOCATABLE_LIBRARY macro. That invocation was moved
into modules/relocatable-lib without noticing that
modules/relocatable-prog also invokes gl_RELOCATABLE_LIBRARY and
also needs to build relocatable.c.
Paul Eggert [Fri, 12 Aug 2011 07:18:19 +0000 (00:18 -0700)]
getaddrinfo: fix sh typo in gai_strerrorA decl checking
* m4/getaddrinfo.m4 (gl_GETADDRINFO): Fix typo in the generated
shell code: it contained a 'break' that was not in a loop.
Apparently the macro assumed that AC_CHECK_DECLS is implemenented
via a shell-language loop; this may have been true in old Autoconf
versions, but it's not true in Autoconf 2.68. I found this bug
when testing coreutils git on Solaris 8, whose shell complains
about the syntax error.
Paul Eggert [Thu, 11 Aug 2011 17:46:26 +0000 (10:46 -0700)]
po/Makefile.in.in: fix make -q problem
* build-aux/po/Makefile.in.in (check-macro-version): Remove this
rule, since there's no file named 'check-macro-version' and its
use as a file breaks make -q.
(all): Don't depend on check-macro-version.
(CHECK_MACRO_VERSION): New macro.
(stamp-po): Use it.
Paul Eggert [Thu, 11 Aug 2011 18:20:28 +0000 (11:20 -0700)]
configmake: fix make -q problem
* modules/configmake (configmake.h): Update configmake.h's time stamp
even if the file does not change. Otherwise, 'make -q' fails.
Problem reported by Simon Josefsson in
<http://lists.gnu.org/archive/html/bug-gnulib/2011-08/msg00088.html>.
Paul Eggert [Wed, 10 Aug 2011 19:36:13 +0000 (12:36 -0700)]
base64: fix off-by-one buffer size bug
Problem and (trivial) fix reported by Gijs van Tulder in
<http://lists.gnu.org/archive/html/bug-gnulib/2011-08/msg00083.html>.
* lib/base64.c (base64_decode_alloc_ctx): Allocate one more byte.
* tests/test-base64.c (main): Catch the bug.
Eric Blake [Tue, 9 Aug 2011 21:18:31 +0000 (15:18 -0600)]
fseeko: fix bug on glibc
Commit 1074f45959f had a bug on glibc that caused the unit test to
fail, but instead of figuring why the failure was happening, we
instead added a partial hack to avoid the problem in the scenario
covered by the testsuite.
https://lists.gnu.org/archive/html/bug-gnulib/2009-01/msg00171.html
But that hack doesn't cover all scenarios, and papers over the
real issue: if you fopen()/ftell()/fseek(,0,SEEK_END)/ftell(),
then the stream should be positioned at the same place as the
underlying fd, even though no I/O occurred, and without requiring
an fflush() in the middle.
* lib/fseeko.c (fseeko): Set stream offset to match fd offset.
Reported by John W. Eaton.
Eric Blake [Mon, 8 Aug 2011 15:35:14 +0000 (09:35 -0600)]
largefile: fix typo that regressed large file support
With no configure.ac-early, and two configure.ac sections, gnulib-tool
only picked up the second (empty) section, and thus completely skipped
any AC_SYS_LARGEFILE call altogether.
Bruno Haible [Fri, 5 Aug 2011 01:01:01 +0000 (03:01 +0200)]
pathmax: Leave PATH_MAX undefined on the Hurd, and a constant otherwise.
* lib/pathmax.h (PATH_MAX): Leave it undefined on GNU/Hurd.
* lib/chdir-long.h: Include pathmax.h.
* lib/clean-temp.c (PATH_MAX): Remove code that is done by pathmax.h.
* lib/getcwd.c: Include pathmax.h instead of <limits.h>.
(PATH_MAX): Remove code that is done by pathmax.h.
* lib/canonicalize.c (PATH_MAX): Provide a fallback value.
* lib/tmpfile.c: Add a comment.
* m4/pathmax.m4 (gl_PATHMAX): Don't test for pathconf.
* modules/chdir-long (Depends-on): Add pathmax.
* modules/getcwd (Depends-on): Add pathmax.
* tests/test-getcwd.c (test_abort_bug): Avoid syntax error when PATH_MAX
is not defined.
* doc/posix-headers/limits.texi: Mention the pathmax module.
* NEWS: Mention the change.
Jim Meyering [Tue, 2 Aug 2011 12:54:34 +0000 (14:54 +0200)]
maint.mk: relax the default _gl_TS_function_match regexp
* top/maint.mk (_gl_TS_function_match): Don't require at least one
space between function name and "(" in an "extern" declaration.
That would fail to match a decl with no space there: extern void foo();
git-version-gen: document that EXTRA_DIST must include .version
* build-aux/git-version-gen: In the how-to-use comment, document
that EXTRA_DIST must include .version. Otherwise, "make distcheck"
will fail when run from an unpacked distribution tarball.
Bruno Haible [Sun, 31 Jul 2011 20:39:32 +0000 (22:39 +0200)]
wctype-h: Add support for Minix.
* m4/wctype_h.m4 (gl_WCTYPE_H): Test whether towlower is declared. Set
REPLACE_TOWLOWER.
* modules/wctype-h (Makefile.am): Substitute REPLACE_TOWLOWER.
* lib/wctype.in.h (towlower, towupper): Test REPLACE_TOWLOWER, not
REPLACE_ISWCNTRL.
Bruno Haible [Sun, 31 Jul 2011 17:00:23 +0000 (19:00 +0200)]
errno: Port to Minix.
* m4/errno_h.m4 (gl_HEADER_ERRNO_H): Test also whether ENETRESET and
ECONNABORTED are defined.
* lib/errno.in.h (ENETRESET, GNULIB_defined_ENETRESET, ECONNABORTED,
GNULIB_defined_ECONNABORTED): New macros.
* lib/strerror-override.h (strerror_override): Test also
GNULIB_defined_ENETRESET, GNULIB_defined_ECONNABORTED.
* lib/strerror-override.c (strerror_override): Handle also ENETRESET,
ECONNABORTED.
* doc/posix-headers/errno.texi: Mention the Minix problem.
Bruno Haible [Sun, 31 Jul 2011 15:05:59 +0000 (17:05 +0200)]
Work around declaration collisions on Minix.
* m4/mbsinit.m4 (gl_FUNC_MBSINIT): If mbsinit is declared but not
defined, set REPLACE_MBSINIT.
* m4/mbrtowc.m4 (gl_FUNC_MBRTOWC): If mbrtowc is declared but not
defined, set REPLACE_MBRTOWC.
* m4/mbrlen.m4 (gl_FUNC_MBRLEN): If mbrlen is declared but not defined,
set REPLACE_MBRLEN.
* m4/mbsrtowcs.m4 (gl_FUNC_MBSRTOWCS): If mbsrtowcs is declared but not
defined, set REPLACE_MBSRTOWCS.
* m4/wcrtomb.m4 (gl_FUNC_WCRTOMB): If wcrtomb is declared but not
defined, set REPLACE_WCRTOMB.
* m4/wcsrtombs.m4 (gl_FUNC_WCSRTOMBS): If wcsrtombs is declared but not
defined, set REPLACE_WCSRTOMBS.
Jim Meyering [Wed, 27 Jul 2011 18:27:48 +0000 (20:27 +0200)]
maint.mk: avoid sc_prohibit_always-defined_macros failure in coreutils
* top/maint.mk (gl_extract_significant_defines_): Now that
SA_RESETHAND and SA_RESTART are #defined (albeit conditionally) in
gnulib/lib/signal.in.h, and now that we recommend to
define-if-undefined those two symbols in application code,
we must filter them out of the "significant" list.
This avoids a "make syntax-check" failure in coreutils.
Eric Blake [Tue, 26 Jul 2011 19:58:26 +0000 (13:58 -0600)]
include-next, warnings: support older autoconf
Older autoconf did not know how to peer through quoting of the
m4 macro created by AS_VAR_PUSHDEF; while newer autoconf has
been patched to allow typical autoconf quoting rules to still
apply, compatibility with autoconf 2.59 requires the unquoted
use of the macro name for all uses prior to AS_VAR_POPDEF.
absolute-header.m4 and warn-on-use.m4 already got this right.
* m4/include_next.m4 (gl_NEXT_HEADERS_INTERNAL): Use
AS_VAR_PUSHDEF in a way that works with older autoconf.
* m4/warnings.m4 (gl_WARN_ADD): Likewise.
Reported by Daniel P. Berrange.
Bruno Haible [Mon, 25 Jul 2011 22:01:49 +0000 (00:01 +0200)]
fclose: Move AC_LIBOBJ invocation to module description.
* m4/fclose.m4 (gl_REPLACE_FCLOSE): Remove macro.
(gl_FUNC_FCLOSE): Instead of gl_REPLACE_FCLOSE, just set REPLACE_FCLOSE
to 1.
* modules/fclose (configure.ac): Invoke AC_LIBOBJ.
Bruno Haible [Mon, 25 Jul 2011 21:47:31 +0000 (23:47 +0200)]
close: Move AC_LIBOBJ invocation to module description.
* m4/close.m4 (gl_REPLACE_CLOSE): Remove macro.
(gl_FUNC_CLOSE): Instead of gl_REPLACE_CLOSE, just set REPLACE_CLOSE to
1.
* modules/close (configure.ac): Invoke AC_LIBOBJ.
Bruno Haible [Mon, 25 Jul 2011 21:38:11 +0000 (23:38 +0200)]
open: Move AC_LIBOBJ invocation to module description.
* m4/open.m4 (gl_REPLACE_OPEN): Remove macro.
(gl_FUNC_OPEN): Instead of gl_REPLACE_OPEN, just set REPLACE_OPEN to 1.
* modules/open (configure.ac): Invoke AC_LIBOBJ and gl_PREREQ_OPEN.
Paul Eggert [Mon, 25 Jul 2011 14:38:52 +0000 (07:38 -0700)]
close: use gl_REPLACE_FCLOSE only if defined
* m4/close.m4 (gl_REPLACE_CLOSE): Use gl_REPLACE_FCLOSE only if it
is defined. The close module doesn't depend on the fclose module
any more, so gl_REPLACE_CLOSE's existence cannot be assumed. See
<http://lists.gnu.org/archive/html/bug-gnulib/2011-07/msg00392.html>.
I reproduced the problem with "./gnulib-tool --test close sys_socket".
Bruno Haible [Sun, 24 Jul 2011 17:05:25 +0000 (19:05 +0200)]
doc: Mention new glibc headers and functions.
* doc/glibc-headers/gshadow.texi: New file.
* doc/glibc-functions/endsgent.texi: New file.
* doc/glibc-functions/fgetsgent.texi: New file.
* doc/glibc-functions/fgetsgent_r.texi: New file.
* doc/glibc-functions/getsgent.texi: New file.
* doc/glibc-functions/getsgent_r.texi: New file.
* doc/glibc-functions/getsgnam.texi: New file.
* doc/glibc-functions/getsgnam_r.texi: New file.
* doc/glibc-functions/putsgent.texi: New file.
* doc/glibc-functions/setsgent.texi: New file.
* doc/glibc-functions/sgetsgent.texi: New file.
* doc/glibc-functions/sgetsgent_r.texi: New file.
* doc/glibc-functions/malloc_info.texi: New file.
* doc/glibc-functions/preadv.texi: New file.
* doc/glibc-functions/pwritev.texi: New file.
* doc/glibc-functions/register_printf_modifier.texi: New file.
* doc/glibc-functions/register_printf_specifier.texi: New file.
* doc/glibc-functions/register_printf_type.texi: New file.
* doc/glibc-functions/pthread_attr_getaffinity_np.texi: New file.
* doc/glibc-functions/pthread_attr_setaffinity_np.texi: New file.
* doc/glibc-functions/pthread_getaffinity_np.texi: New file.
* doc/glibc-functions/pthread_getname_np.texi: New file.
* doc/glibc-functions/pthread_mutex_consistent_np.texi: New file.
* doc/glibc-functions/pthread_mutexattr_getrobust_np.texi: New file.
* doc/glibc-functions/pthread_mutexattr_setrobust_np.texi: New file.
* doc/glibc-functions/pthread_setaffinity_np.texi: New file.
* doc/glibc-functions/pthread_setname_np.texi: New file.
* doc/glibc-functions/pthread_sigqueue.texi: New file.
* doc/glibc-functions/pthread_timedjoin_np.texi: New file.
* doc/glibc-functions/pthread_tryjoin_np.texi: New file.
* doc/glibc-functions/qsort_r.texi: New file.
* doc/glibc-functions/quick_exit.texi: New file.
* doc/glibc-functions/syncfs.texi: New file.
* doc/gnulib.texi: Include them.
(Glibc gshadow.h, Glibc sys/uio.h): New sections.
* doc/posix-functions/psiginfo.texi: Fix info about glibc version.
* doc/posix-functions/pthread_mutex_consistent.texi: Likewise.
* doc/posix-functions/pthread_mutexattr_getrobust.texi: Likewise.
* doc/posix-functions/pthread_mutexattr_setrobust.texi: Likewise.
* doc/glibc-functions/execvpe.texi: Likewise.
Bruno Haible [Sun, 24 Jul 2011 10:20:41 +0000 (12:20 +0200)]
fsusage: Enable large volume support on AIX >= 5.2.
* m4/fsusage.m4 (gl_FILE_SYSTEM_USAGE): If 'struct statvfs64' has a
larger f_blocks field than 'struct statvfs', define STAT_STATVFS64
instead of STAT_STATVFS.
* lib/fsusage.c (get_fs_usage) [STAT_STATVFS64]: Use statvfs64.
Paul Eggert [Fri, 22 Jul 2011 17:07:18 +0000 (10:07 -0700)]
fsusage: port to MacOS X 10.7 with 4 TiB file systems
* m4/fsusage.m4 (gl_FILE_SYSTEM_USAGE): Reject statvfs
implementations that use only 32 bits to count blocks.
On typical hosts with 1024-byte blocks, this fails with file
systems as small as 4 TiB. Problem reported by Herb Wartens
<http://debbugs.gnu.org/9140> and this should also fix a similar
problem reported by Tim Spriggs <http://debbugs.gnu.org/7355>.
file-has-acl: use acl_extended_file_nofollow if available
* lib/acl-internal.h (HAVE_ACL_EXTENDED_FILE): New macro.
(acl_extended_file): New macro.
* lib/file-has-acl.c (file_has_acl): Use acl_extended_file_nofollow.
* m4/acl.m4 (gl_FUNC_ACL): Check for acl_extended_file_nofollow.
This addresses http://bugzilla.redhat.com/692823.
Bruno Haible [Thu, 21 Jul 2011 22:16:46 +0000 (00:16 +0200)]
Declare system functions in a way that works with C++.
* m4/fdopendir.m4 (gl_FUNC_FDOPENDIR) [C++]: In the test program,
declare fdopendir as extern "C".
* m4/frexpl.m4 (gl_FUNC_FREXPL_WORKS) [C++]: In the test program,
declare frexpl as extern "C".
* m4/getaddrinfo.m4 (gl_GETADDRINFO) [C++]: In the test program,
declare gai_strerror as extern "C".
* m4/getdomainname.m4 (gl_FUNC_GETDOMAINNAME) [C++]: In the test
programs, declare gai_strerror as extern "C".
* m4/getlogin_r.m4 (gl_FUNC_GETLOGIN_R) [C++]: In the test program,
declare getlogin_r as extern "C".
* m4/ioctl.m4 (gl_FUNC_IOCTL) [C++]: In the test program, declare ioctl
as extern "C".
* m4/ldexpl.m4 (gl_FUNC_LDEXPL_WORKS) [C++]: In the test program,
declare ldexpl as extern "C".
* m4/logb.m4 (gl_FUNC_LOGB) [C++]: In the test programs, declare logb
as extern "C".
* m4/ls-mntd-fs.m4 (gl_LIST_MOUNTED_FILE_SYSTEMS) [C++]: In the test
program, declare getmntinfo as extern "C".
* m4/stpncpy.m4 (gl_FUNC_STPNCPY) [C++]: In the test program, declare
stpncpy as extern "C".
* m4/strerror_r.m4 (gl_FUNC_STRERROR_R_WORKS) [C++]: In the test
program, declare __xpg_strerror_r as extern "C".
* m4/strndup.m4 (gl_FUNC_STRNDUP) [C++]: In the test program, declare
strndup as extern "C".
* m4/sys_select_h.m4 (gl_HEADER_SYS_SELECT) [C++]: In the test program,
declare memset and bzero as extern "C".
Reported by Sam Steingold <sds@gnu.org>.
timer-time: A new module to check for timer_settime()
* m4/timer_time.m4: Check for the POSIX function.
* modules/timer-time: Add the new module.
* MODULES.html.sh (Compat checks for POSIX:2008 functions):
Mention it.
* doc/posix-functions/timer_create.texi: Add timer-time as
the module, and update the platforms where unavailable.
* doc/posix-functions/timer_delete.texi: Likewise.
* doc/posix-functions/timer_gettime.texi: Likewise.
* doc/posix-functions/timer_settime.texi: Likewise.
Paul Eggert [Wed, 20 Jul 2011 00:57:20 +0000 (02:57 +0200)]
pthread_sigmask: assume POSIX threads if --avoid=threadlib
* m4/pthread_sigmask.m4 (gl_FUNC_PTHREAD_SIGMASK): If gl_THREADLIB is
not defined, assume POSIX threads and look for pthread_sigmask in
$LIBS, without changing $CPPFLAGS.
Bruno Haible [Tue, 19 Jul 2011 21:43:05 +0000 (23:43 +0200)]
strstr: Update cross-compilation guess.
* m4/strstr.m4 (gl_FUNC_STRSTR): On glibc > 2.12 with x86 or x86_64
CPUs, guess no, in view of glibc
BZ #12100 <http://sourceware.org/bugzilla/show_bug.cgi?id=12100>.
Suggested by Eric Blake. Reported by Reuben Thomas.
getopt-gnu: suppress core dumps from detection code
* m4/getopt.m4 (gl_GETOPT_CHECK_HEADERS): Use the nocrash module
to suppress core dumps that may well occur on glibc systems.
These core dumps might not be cleaned up automatically, or could
trigger some system core dump handling logic.
* modules/getopt-gnu: Depend on nocrash.