ssh fatal: buffer_get: trying to get more bytes than in buffer

The issue

You are using ssh to login to a server with ssh key authentication and you get connection closed. On the server you are logging into the syslog shows as messages like

Oct 17 11:30:02 myserver sshd[27687]: [ID 800047 auth.crit] fatal: buffer_get: trying to get more bytes than in buffer

The fix

Check your authorized_keys file on the remote server. Use ssh-keygen -l -f  ~/.ssh/authorized_keys

ssh-keygen -l -f ~/.ssh/authorized_keys
buffer_get: trying to get more bytes than in buffer

The above shows there is at least one  key in your file that is the wrong format – usually because it is  split over several lines rather than being just one long line. (note it could be any key in the file – not the one you are using from your server ) Once you fix the key then confirm with ssh-keygen that all is well – it should return a md5 checksum.
ssh-keygen -l -f authorized_keys
md5 1024 5d:35:7e:ad:3d:e6:70:6d:6f:1d:76:1a:46:ee:c1:c9 authorized_keys

Now retry your ssh access

UTF-8 AIX and the 

I setup a GPG encryption transfer with a 3rd party to transfer XML files.

On decrypting the GPG XML file I noticed when I vi’d the file I had  at the beginning of the file – everything else in the file was displayed OK.

In the end it turned out it was all to do with character sets – the default AIX charcter set is ISO8859-1 and the file was sent from a server that was using UTF-8

The  I was seeing is the BOM Byte Order Mark – Wikipedia has a useful page  that shows what characters are displayed on a ISO8859-1 system for the various UTF encodings.

To convert to ISO8859-1 you can use iconv

Install websm client on Linux to access HMC

Install websm client on Linux

From your Linux box copy over /usr/websm/pc_client/wsmlinuxclient.exe  from your AIX LPAR.

For Centos x86_64 I needed additional RPMs :-

yum groupinstall “GNOME Desktop Environment”     ( only needed if you don’t have X11 installed )
yum install libXmu.i386
yum install libXp.i386

Note the libs have to be the i386 version even though the Linux is x86_64

Install websm

./wsmlinuxclient.exe -console

If you accept the defaults then after teh install satrt it up :-

/opt/websm/bin/wsm

If it bombs out with errors like :-

java.lang.UnsatisfiedLinkError: /opt/websm/_jvm/bin/libawt.so: libXp.so.6: canno
t open shared object file: No such file or directory
Then install the missing library RPM ( it needs to be the i386 version )

AIX bug on 5.3 TL11 SP6 directory corruption

After patching to AIX 5.3 TL11 SP6 with fileset bos.mp , bos.mp64 5.3.11.6 a directory became unusable and corrupt , the only solution was to unmount the filesystem and mount it again. fsck did not report any errors. I managed to re-create the error :-

$ cd /test
$ ls -l
total 0
-rw——-    1 root     system            0 Jul 22 10:59 one
drwxr-xr-x    2 root     system          256 Jul 22 10:59 onedir
-rw-r–r–    1 root     system            0 Jul 22 10:59 three
drwxr-xr-x    2 root     system          256 Jul 22 10:59 threedir
-rw-r–r–    1 root     system            0 Jul 22 10:59 two
drwxr-xr-x    2 root     system          256 Jul 22 11:00 twodir
$ mv -f one twodir
mv: 0653-401 Cannot rename one to twodir/one:
The file access permissions do not allow the specified action.
$ ls -l
ls: 0653-341 The file ./one does not exist.
ls: 0653-341 The file ./twodir does not exist.
total 0
drwxr-xr-x    2 root     system          256 Jul 22 10:59 onedir
-rw-r–r–    1 root     system            0 Jul 22 10:59 three
drwxr-xr-x    2 root     system          256 Jul 22 10:59 threedir
-rw-r–r–    1 root     system            0 Jul 22 10:59 two
$

$ cd twodir
ksh: twodir: 0403-036 The specified directory is not valid.
$ find twodir
find: 0652-019 The status on sixdir is not valid.
$
You end up with a directory you can do anything with –  and any sub directories are also unavailable. No data is actually lost – if you unmount and remount the filesystem all the data is available again.

The fix is to install APAR IZ96928  which requires a reboot.

AIX not showing the correct oslevel after patching

After patching AIX to 5.3 TL11 SP3 oslevel -r still showed the OS at TL 09 :-

# oslevel -r
# 5300-09

To check what filesets are below the TL you patched to first find all TL levels you have installed :-

# oslevel -r -q
Known Recommended Maintenance Levels
————————————
5300-11
5300-10
5300-09
5300-08
5300-07
5300-06
5300-05
5300-04
5300-03
5300-02
5300-01
5300-00

Next check what filesets are below our highest level e.g. TL11

# oslevel -r -l 5300-11
Fileset                                 Actual Level           Recommended ML
—————————————————————————–
Java5.sdk                               5.0.0.1                5.0.0.235
ifor_ls.html.en_US.base.cli             5.3.7.0                5.3.8.0
#

Here you can see two filesets are at the wrong level. So we need to fix this. Java is a seperate download for AIX you can get it here http://www.ibm.com/developerworks/java/jdk/aix/service.html

Check if you have the 32 or 64 bit version with lslpp -l | grep -i java and download the latest fix – select your highest TL level you have installed when you download. Install using the usual  installp / smiity method.

Re run oslevel -r -l 5300-11
Fileset                                 Actual Level           Recommended ML
—————————————————————————–
ifor_ls.html.en_US.base.cli             5.3.7.0                5.3.8.0

Now only one fileset is below level. A Google shows that there is a bug with the update process for this fileset – simply run the patch process again and it will update to the correct level.

Now oslevel is correct :-

# oslevel -r
5300-11
#

Installing Verisign SSL certificate on IBM HTTP server

Installing a Verisign SSL site certificate on IBM HTTP server

If you have an Apache certificate e.g. it was requested with an openssl signing request rather than using ikeyman then you first need to convert it to PKCS12 format which can then be imported into the IBMHTTPServer6 keystore.

openssl pkcs12 -export -out new_key_pair_filename.p12 -inkey private_key_filename.key -in certificate_filename.crt

You will get prompted for a password – you must use the same password as you have on the keystore you want to import it into.

Move the file to /usr/IBMHTTPServer6/bin
If you used strong encryption to generate the signing key request ( and you would have done ) then you may have to install the unrestricted JCE policy files.

To check :-

/usr/IBMHTTPServer6/java/jre/bin/keytool -list -v -keystore /usr/IBMHTTPServer6/bin/wbis104m.p12 -storetype pkcs12 -storepass passwd

If it barfs with java errors like :-

keytool error (likely untranslated): java.io.IOException: Private key decryption error: (java.security.InvalidKeyException: Illegal key size)

keytool error (likely untranslated): java.io.IOException: Private key decryption error: (java.lang.SecurityException: Unsupported keysize or algorithm parameter
s)

You need to install the unrestricted JCE policy files.

Download the zip file from https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?source=jcesdk ( you need an IBM ID – this is a free registration )

unzip and after making copies of the orginals copy over the new local_policy.jar US_export_policy.jar files to /usr/IBMHTTPServer6/java/jre/lib/security

Rerun the keytool command above ( ensuring you use the full path to the keytool command ) to confirm it lists the certificate details without Java errors.

Now add it into the keystore

You need to be able to use X as the ikeyman program is GUI only.

su to root , export XAUTHORITY and DISPLAY to those of the user you su’d from.

e.g.

export XAUTHORITY=/home/fred/.Xauthority

export DISPLAY=localhost:10.0

cd /usr/IBMHTTPServer6/bin

./ikeyman

Key Database File – Open

Key Database type CMS

Location /usr/IBMHTTPServer6/keys/

File Name key.kdb

You will be prompted for the password

Now import the certificate you converted to pkcs12 format above

Ensure Personal Certificates is selected then click on Export/Import

Select Import Key

Key file type PKCS12

File Name the file name of the converted pkcs12 format above

Location where you put the file

Click OK – you will be prompted for a password – use the one you set when you did the conversion ( which should also be the same as the keystore password you are putting it in )

If you get a message “The specified database has been corrupted” ensure you have installed the unrestricted JCE policy files above. If you have to install them you need to exit ikeyman and restart it again.

You should now get a dialog asking if you would like to change any of these labels before completeing the import process

Click on the label ( which is probably a very long string ) and then change it to something like prod-cert ( this is the name you will use in the httpd.conf file )

Click apply

Click OK ( you may have to scroll to the right to see the OK button )

If you now get an error An attempt to import the certificate has failed.

All the signer certificates must exist in the key database

This probably means that you need to install the Verisign intermediate signers certificate.

https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&id=AR657

Assuming it is a standard Verisign site certifiacate ( class 3 ) then go here :-

http://www.verisign.com/support/verisign-intermediate-ca/secure-site-intermediate/index.html

Cut and paste the certificate into a file and save with a .arm extension

Go into ikeyman and open the keystore as above

Select Signer Certificates

Click add

Data type Base64-encoded ASCII data

Certificate file name the name of the arm file you created above

Location the location of the arm file

Click OK

Enter a label for the certificate – choose something like Verisign intermediate CA cert

Click OK

Now select Personal Certificates and import the converted PKCS12 SSL certificate using the intructions as before.

Adding the certificate to the httpd.conf file

vi /usr/IBMHTTPServer6/conf/httpd.conf

search for SSLServerCert and change the name of the certificate to the name you chose when you added the certificate to the key store e.g. prod-cert

Restart apache

Finding the date x number of days ago in the shell

If you need to find what the date was X number of days ago in AIX ( 5.3 or greater ), or Linux  :-

First convert the number of days to hours e.g. if you want to find the date 90 days ago you need 90 * 24 = 2160

echo `TZ=GMT+2160 date +%d/%m/%y`

You can of course use any of the % formats to get the output you require.

To compare two dates ( very useful in AIX to see if a user has not logged in for 90 days for example ) use the %s to get the number of seconds since 1st Jan 1970 :-

echo `TZ=GMT+2160 date +%s`

lsuser  -a time_last_login username   ( needs to run as root )

Then compare the two numbers and if the 90 days ago number is gretaer than the output of the lsuer command then that user has not logged in for gretaer than 90 days

AIX making the identification light flash on a PCI card

AIX making the identification light flash on a PCI card

When you need to check a cable on a PCI card or even replace a PCI card on an IBM Pseries running AIX it is helpful to identify the card by flashing the identification light on the card to distinguish it from the others.
First find the slot with the card in :-

lsslot -c pci
# Slot                   Description                         Device(s)
U7879.001.DQQQWTY-P1-C1  PCI-X capable, 64 bit, 133MHz slot  fcs1
U7879.001.DQQQWTY-P1-C3  PCI-X capable, 64 bit, 133MHz slot  ent0
U7879.001.DQQQWTY-P1-C4  PCI-X capable, 64 bit, 133MHz slot  ent1
U7879.001.DQQQWTY-P1-C5  PCI-X capable, 64 bit, 133MHz slot  fcs0

diag
enter to continue
Task Selection
Scroll down to Identify and attention Indicators
Scroll down until you get to the slot of the PCi card then hit enter and you get
a + besides the slot, hit F7 and then you get an I , this means the
indicator light on the card is flashing, use F7 enter to toggle it on and off.


AIX sshd start up problem

After rebooting a server sshd did not start. I tried starting using startsrc -s sshd , it came back with a PID but did not start. Looking in the errorlog with errpt -a showed src had a problem starting sshd.

OK, I thought no problem I will just start sshd from the command line to see why it fails to start so I did /usr/sbin/sshd

It came back with bad argument line 53 /etc/ssh/sshd_config

I checked the config file and sure enough at line 53 was :q!

Obviously some one had attempted to quit out of vi without saving and managed to mangle the file instead!

Removing the :q! solved sshd starting correctly.