Please consider a donation to the Higher Intellect project. See https://preterhuman.net/donate.php or the Donate to Higher Intellect page for more info.

Whattohide.shar

From Higher Intellect Vintage Wiki
Jump to navigation Jump to search
#! /bin/sh
# This is a shell archive, meaning:
# 1. Remove everything above the #! /bin/sh line.
# 2. Save the resulting text in a file.
# 3. Execute the file with /bin/sh (not csh) to create the files:
#	README
#	update-all-to-hidden
#	whattohide
#	update-to-hidden
#	ChangeLog
# This archive created: Fri Jun 30 13:37:32 2000
export PATH; PATH=/bin:$PATH
if test -f 'README'
then
	echo shar: will not over-write existing file "'README'"
else
sed 's/^X//' << \SHAR_EOF > 'README'
X
XThese comments and the scripts and utility
Xprogram mentioned are freely available.
X
XThey are useful to folks who build applications
Xwith large numbers of DSOs. 
XNothing here can or should be used to affect
Xsystem DSOs.  Only to change the build
Xof DSOs you build for your application.
XIf the number of DSOs you build for your application is
Xgetting to be more than 10 you might consider using
Xthe facilities here.
XIf the number of DSOs you build for your application is
Xnear or above 100 you should definitely use the following.
X
XReport problems or suggestions to me, [email protected]
X
X
X
[email protected] Feb 2000
X$Revision: 1.2 $ 
X$Date: 2000/06/30 20:36:32 $
X=========================
XDSO: a shared library. DSO is the
Xname used in IRIX documentation. 
XDSO is shorter to type :-)
X
X*****The Problem
X(back of the envelope style calculation, based
Xon knowledge of run time lookup internals):
XLookup N syms by name with 1 DSO to search:  2N
XLookup N syms by name in N DSOs by name,
Xfinding 1/2 way thru list on average:     (2N/2)*2N
XQuadratic cost! Bad!
X
X*****The Summary 
XIf one has more than a few DSOs
Xlookup can become expensive.
XThis can cause application startup time to be excessive and
Xnoticeable when the numbers get big enough.
XFor 70000 symbols and 100 DSOs the time gets to 
Xseveral seconds.
XEven for a single DSO, if the lookups are
Xactually not necessary, there is time wasted.
X
X*****Justifying the Problem Calculation:
XIterating thru the list of DSOs takes cycles.
XFinding if a symbol is in a given DSO is fast (since
Xthe Elf hash table is effective, it is an O(1)
Xoperation) but lets call that
Xthe same cost as getting to a single DSO in the list.
XSo each by-name lookup in a single DSO
Xhas cost 1 to find the DSO, and 1 to find the name.
XN names to look up in 1 DSO, cost is thus 2N.
X
X*****A Possible Solution:
XMany of the names so lookup up are probably resolved
Xin the *same* dso that the call is in.
XSuch names can be converted to be 'hidden' symbols
Xand hidden symbols are *never* looked up by name.
XEvery name hidden gets out of the quadratic time bind!
X
XBuild the DSOs and app (no changes from present build)
XCreate a list of the DSOs and the executable for input
Xto  a script.
Xls -1 is the best approach: you want the 
Xexecutable and your DSOs, not
Xsystem DSOs, on the list.
X
XCreate the list of hideable symbols (script name:whattohide)
X
X******Hide for testing purposes
XNow use  the list of DSOs and the list of hideable symbols
Xto hide the hideable symbols (scripts update-all-to-hidden  
Xand  update-to-hidden and the program 'optionalsym').
X(don't bother hiding symbols in the executable).
X
XThere is no formal guarantee that using 'optionalsym'
Xproduces a valid DSO (in fact in once sense the result
Xis incorrect). However, all existing versions of
Xrld(1) account for this incorrectness and the
Xresult should work.
X
X
X******Hide for shipment of your application
XFor build of a real product, the use the ld option
X-hiddens_file <filename>
Xwhere the <filename> contains the list of symbols to be hidden.
XThis generates a DSO with the hidden symbols etc
Xproperly laid out so the DSO is correct (unlike optionalsym)
X
X
XIf the total list of filenames too large, the -hiddens_file <filename>
Xcould expand ld to the point it runs out of virtual memory.
XIn this case, do something like the following
X  nm -Bg <DSONAME> |grep ' T ' | awk ' { print $3 } '  >temp-sto1
X  cat temp-sto1 <TOHIDELIST> |sort |uniq -d >temp-sto2
Xborrowed from the update-to-hidden script
Xand use "-hiddens_file  temp-sto2", which has just the names
Xthat this DSO should hide.
X
X
X*****When you have functions that must not be hidden
XSome applications allow users to add new DSOs that refer
Xback to the base DSOs or executable.  In that case, the
Xlist of 'symbols to hide' generated by the scripts
Xwill have some things that the script cannot know must
Xbe left visible.  
X
XThe solution: simply add the required stuff to the set 
Xof symbols referred to externally (modify the whattohide
Xscript to add in those symbols as extra outside-dso references).
X
XNow this solution is simple for C (and reasonably so for Fortran)
Xbut since, for C++, one might want an entire class's
Xfunctions to be visible, C++ is harder.
XYou may need to compile certain headers and use the demangler
X(/usr/lib/c++/c++filt) to allow you to match up
Xnames-to-be-left-visble by the class name.
XAt this time no code for such is provided here.
XThere is no generally agreed upon way to mark classes as
Xto-be-exported so it's not clear there is a single
Xapproach.
X
X
X*****General comments
XIf you leave out a DSO or the executable in
Xthe process of building the initial DSO list
Xthen when all is done the app may not work (rld(1)
Xmay say it cannot find symbols at app run time)!
XIf you miss an executable or a DSO in the
Xhiding the result will still work but
Xsome of the savings will be missed.
X
X*****Testing for correct execution:
XIt's always wise to test your app with LD_BIND_NOW 
Xset to 1, as that is the quickest way to verify that
Xthere are no name resolution surprises.
XAs opposed to having a user reporting an app crash
Xbecause rld could not resolve a symbol due to using
Xa code path that did a call not used in testing.
XLD_BIND_NOW ensures all the calls actually have code
Xsomewhere to resolve them!  
XThis still requires full testing though, as you must
Xbe sure all dlopens are done...
X
X
SHAR_EOF
fi # end of overwriting check
if test -f 'update-all-to-hidden'
then
	echo shar: will not over-write existing file "'update-all-to-hidden'"
else
sed 's/^X//' << \SHAR_EOF > 'update-all-to-hidden'
X#!/bin/ksh
X#
X# Usage:  update-all-to-hidden <filename> <names to hide>
X# where <filename> contains a list of DSOs, one per line,
X# full or relative path, that you want some symbols hidden in.
X# where <names to hide> contains  a list of names to hide,
X# one per line, across all the DSOs.
X# The script whattohide produces such a list.
X#
X# $Revision: 1.3 $
X# $Date: 2000/06/30 20:37:00 $
X#set -x
Xif [ $# != 2 ]
Xthen
X   echo "Usage: update-all-to-hidden <file-with-dsos> <hide-list>"
X   exit 1
Xfi
X
Xwhile read dsoname
Xdo
X	ksh update-to-hidden $dsoname  $2
Xdone <$1
SHAR_EOF
fi # end of overwriting check
if test -f 'whattohide'
then
	echo shar: will not over-write existing file "'whattohide'"
else
sed 's/^X//' << \SHAR_EOF > 'whattohide'
X#!/bin/ksh
X# $Revision: 1.6 $
X# $Date: 2000/02/16 18:15:20 $
Xif [ $# != 1 ]
Xthen
X   echo "Usage: whattohide file-with-dsos-listed"
X   exit 1
Xfi
X
Xo=hides-list
Xt=temp-hides
X# This script takes as input a  file with
X# a list of pathnames to the DSOs in an application.
X# Plus the executable name.
X# It produces a list of names which can be 'hidden'
X# and can thus get us out of the time bind.
X# If the list of names is substantial using the
X# list is worthwhile.
X#
X# System DSOs can be left out of the list (but it's not
X# harmful to leave them in).
X# Leaving out DSOs that are  part of the app
X# will most likely break the application!
X#
X# If you miss an executable or a DSO in the
X# rebuild the result will still work but
X# some of the savings will be missed.
X#
X# This script creates temp files in the local directory.
X#
X
X#==========================================
Xl=$1
X#set -x
Xrm -f $t-allsyms
Xrm -f $t-U
Xrm -f $t-T
Xrm -f $t-defwhere
Xrm -f $t-symdetails
Xrm -f $t-wherelist
Xrm -f $t-all
Xrm -f $t-hide
Xrm -f $t-alreadyhidden
Xrm -f $t-hid
Xrm -f $t-prot
Xrm -f $t-dup
X
X# Find all the global symbols
Xwhile read file
Xdo
X  nm -Bg $file >>$t-allsyms
Xdone < $l
X
X# Create a list of undefined symbols
Xgrep ' U ' $t-allsyms >$t-U
X# Create a list of defined text symbols (which could
X# be hidden or protected.... check below)
Xgrep ' T ' $t-allsyms >$t-T
X
X# We only deal with text as only those can we be
X# quite sure it is ok to simply hide.
X# Common and Data are problematic.
X
X# The number of dups is interesting. Might
X# suggest a build problem?
Xcat $t-T | awk ' { print $3 }' |sort |uniq -d >$t-dup
Xecho "duplicate names file entry count" `wc -l $t-dup`
X
X# We want to remove things whch are already
X# marked as 'HIDDEN' from our hides list. elfdump -Dt tells us this
X# (once we process it later)
X
X
X
X# effectively sorts U (undef) before T (definition, text)
Xcat $t-U $t-T |uniq >$t-all
X
X# Emit only those not marked as undefined
X#
Xcat >$t.1.awk <<\_EOF
X/ U / { undef[$3] = 1 }
X/ T / { if ( undef[$3] != 1 ) {print $3 ; }  }
X/ D / { if ( undef[$3] != 1 ) {print $3 ; }  }
X/ A / { if ( undef[$3] != 1 ) {print $3 ; }  }
X_EOF
Xawk -f $t.1.awk <$t-all >$t-hide
X
X
X# Emits SVR4 style name so we can see HIDDEN PROTECTED
Xwhile read file
Xdo
X  elfdump -Dt $file >>$t-symdetails
Xdone < $l
X
X# Things that are already hidden should not
X# be in the list! make lists of these
Xgrep  -e '.*[	 ]*HIDDEN[	 ]*' <$t-symdetails >$t-hid
Xgrep  -e '.*[	 ]*PROTECT[	 ]*' <$t-symdetails >$t-prot
X
X# Create a new file with things we want to hide and protect first
X# Do this in specific order, effectively sorting by
X# category.
Xawk ' {print "HIDDEN " $8 }'  <$t-hid >$t-thc
Xawk ' {print "PROTECT " $8 }'  <$t-prot >>$t-thc
Xawk ' {print "CHECK " $1 }'  <$t-hide >>$t-thc
X
X
X# Emit  as hideable only those not hidden or protected
X# Removes hidden and protected symbols.
X# Depends on input ordering to work correctly.
Xcat >$t.2.awk  <<\_EOF
X/^HIDDEN /  { hid[$2] = 1; }
X/^PROTECT / { hid[$2] = 1; }
X/^CHECK /   { if ( hid[$2] != 1 ) { print $2; } }
X_EOF
Xawk -f $t.2.awk  <$t-thc > $o
X
X
Xecho "number of externs that should be hidden " `wc -l $o`
X
Xexit 0
SHAR_EOF
fi # end of overwriting check
if test -f 'update-to-hidden'
then
	echo shar: will not over-write existing file "'update-to-hidden'"
else
sed 's/^X//' << \SHAR_EOF > 'update-to-hidden'
X#!/bin/ksh
X#
X# Given a DSO name, and the file hides-list with names to hide,
X# change those names to STO_HIDDEN
X#
X# Usage: update-to-hidden <DSOname> <hideslist name>
X#
X# $Revision: 1.3 $
X# $Date: 2000/06/30 20:37:00 $
X#set -x
X
Xif [ $# != 2 ]
Xthen
X   echo "Usage: update-to-hidden <dso pathname> <hide-list>"
X   exit 1
Xfi
X
Xn=$1
Xh=$2
X
Xrm -f temp-sto1
Xrm -f temp-sto2
X
Xnm -Bg $n |grep ' T ' | awk ' { print $3 } '  >temp-sto1
X
Xcat temp-sto1 $h |sort |uniq -d >temp-sto2
X
Xwhile read name
Xdo
X	  optionalsym -u STO_HIDDEN -y $name $n
Xdone <temp-sto2
X
X#Marked hidden now, we trust.
SHAR_EOF
chmod +x 'update-to-hidden'
fi # end of overwriting check
if test -f 'ChangeLog'
then
	echo shar: will not over-write existing file "'ChangeLog'"
else
sed 's/^X//' << \SHAR_EOF > 'ChangeLog'
X
X30 Jun 2000 [email protected]
X  Cheng Liao contributed fixes to typographical errors in
X  usage messages and an 'if'.
X
X  I, davea, contributed some more words in the README for a
X  special case: externally-visible-stuff.
SHAR_EOF
fi # end of overwriting check
#	End of shell archive
exit 0