ActiveMQ: QueueSize vs InFlightCount

Project Sparta #1 - tame the yellow elephant

Mission #1. set up multi-node setup for hadoop 0.20, hadoop 0.23
- document installation steps
- set up basic accounts, queues
- set up some authorization scheme
- play around with operational stuff - define, allocate and monitor quotas


Hardware:

0.20:
athena-nn, athena-jt, athena-s[1-3]


0.23:
zeus-nn, zeus-jt, zeus-s[1-3]

Re-enable "remember passwords" on Chrome (Mac)

This is pretty stupid of Chrome, given the fact that it has done everything else so smartly. Most of us have the habit of letting our browser save our mail and fb passwords; and when a friend or someone else logs in to his account using this browser, and the browser asks to save his password, we promptly click "not now" button that Firefox gives. But on chrome, the buttons are "save password" and "never for this site", and if you click the other button, your own saved passwords are also lost.

Hmm.. now what, go to the password manager and remove that exception, right? On Windows, yes; on Mac no. Go to the wrench icon, preferences; select the center tab, click "show saved passwords". This opens up the "Keychain Access" tool on Mac. And guess what, it does not have the exception list there.

Solution:

1. Close Chrome

2. Goto to ~/Library/Application Support/Google/Chrome/Default

3. sqlite3 Login\ Data "select origin_url, blacklisted_by_user from logins"

You'll see that for the site for which you clicked "never for this site", the blacklisted_by_user value is 1. Just reset those values to 0

4. sqlite3 Login\ Data "update logins set blacklisted_by_user=1 where origin_url like '%google%'"

Now start your chrome, and save your passwords again.

mysql client out of memory

Error:
Out of memory (Needed 1453192 bytes)
DBD::mysql::st execute failed: MySQL client ran out of memory at file

Solution:
Set this flag in the DB handle -
$rdbh->{'mysql_use_result'} = 1;

It forces the API to use mysql mysql_use_result() instead mysql_store_result().
Basically, instead of buffering all the matches, it returns line by line.
Note that this is the same as --quick option available in the mysql client.

This causes the mysql query thread state to change to "writing to net";
mysql> show processlist\G
...
State: writing to net
...

apache jammed with lots of 408s

408 - Request Timeout
A 408 status code means that the client did not produce a request quickly enough. A server is set to only wait a certain amount of time for responses from clients, and a 408 status code indicates that time has passed

http://www.addedbytes.com/articles/http-status-codes-explained/

Solution:
ReadTimeOut 10 (default is 120 seconds)

chapter3: system access

passwordless ssh setup
- ssh-keygen
- ssh-copy-id
- ssh-agent, ssh-add

sudo access
- format: user/alias machines=(runas) commands

syslog
- options: /etc/sysconfig/syslog
- for remote logging, add '-r' i.e. SYSLOGD_OPTIONS="-m 0 -r"

syslog.conf
- format: facility.priority location
- facility = kern, mail, user etc
- priority = crit, warn
- location = /var/log/messages, @192.168.0.3

cron
- format
- cron.allow supersedes cron.deny
- anacron

printers
- commands: lpadmin, lpq, lp
- conf files: /etc/cups/cupsd.conf /etc/cups/printers.conf
- use system-config-printer to make a default, text only printer

ntp
- commands: ntpq -p -n

Chapter2 - rpm and yum

Commands to be aware of:

yum

• yum list [all|installed|available|updates] [regex]
  
• yum info pkgname
• yum provides filename (lists the rpms that contain that file. 'yum provides filename | grep installed -B3' is more useful)
• yum install pkgname | yum localinstall pkgname.rpm | yum groupinstall group
• yum search

createrepo

• createrepo -v /dir
  
• createrepo -v --update /dir
• Creates the following files in /dir/repodata/
  • repomd.xml - md5 checksum of other files. client cache is updated when this file changes
  • primary.xml.gz - list of all rpms, and files provided by rpms
  • filelists.xml.gz - list of all files provided by rpms, used by 'yum provides'
  • other.xml.gz - other meta info of packages

rpm

• install-upgrade-remove
  
• rpm -ivh pkg.rpm: installs. -v verbose, -h prints hash progress bar
• rpm -Uvh pkg.rpm: upgrades pkg.rpm if it exists, installs if it doesn't exist
• rpm -Fvh pkg.rpm: upgrades pkg.rpm if it exists, skips if it doesn't exist
• rpm -i --replacepkgs: install same package again
• rpm -i --oldpackage: downgrade package
• rpm -e pkg: removes pkg

• query
  
• rpm -qa: list all packages
• rpm -ql pkg: list all files of installed pkg
• rpm -qc pkg: list all config files of installed pkg
• rpm -qd pkg: list all doc files of installed pkg
• rpm -qi pkg: pkg info
  
• rpm -qf filename: name the package that installed this file
• rpm -q --changelog pkg: print changelog
• rpm -q --scripts pkg: print pre and post scripts
• rpm -qa --last: print installation time
  
• rpm -qap pkg.rpm: list all files of pkg.rpm
• rpm -qcp | -qdp pkg.rpm

• verify
• rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
  
• rpm -V pkg: verifies package (of any modifications to its files)
• rpm -Va : verify all pkg
• rpm -Vp pkg.rpm
  
• rpm --checksig pkg.rpm

Chapter1 - Installation and Virtualization

Installation:

• In the training rooms, they set up a DHCP server, that also hosts a kickstart file
• For practice, I've installed RHEL 5.4 on my desktop using the regular DVD boot
• The kickstart file for this install is stored in /root/anaconda-ks.cfg
• A kickstart file contains 3 sections:
  • Commands: The response for all questions asked in the installation
  • Packages: List of pkgs to be installed
  • Scripts: pre and post scripts to be run before and after the installation

Virtualization:
Install Xen hypervisor:
$ yum -y install kernel-xen xen virt-manager
Reboot, select the Xen kernel in Grub menu

Start services:
$ service xend start
$ service libvirtd start

Make sure the services start on boot:
$ chkconfig xend on
$ chkconfig libvirtd on

Create LV for VM:
$ lvcreate -L 10G -n vserver sys
$ virt-install --paravirt --ram 1024 --disk vol=/dev/sys/vserver --location /mnt/iso
OR
$ virt-install --paravirt --ram 1024 --disk vol=/dev/sys/vserver --location ftp://ftplocation/image -x ks=kickstart_file

Graphical tool - virt-manager
$ yum - y install virt-manager

Create Yum repository from ISO file

The Linux flavor used here is RHEL5.

1. Mount the ISO image to /mnt/iso/
$ mkdir /mnt/iso
$ mount -o ro,loop /path/to/rhel-server-5.4-x86_64-dvd.iso /mnt/iso

or add this line to /etc/fstab
/path/to/rhel-server-5.4-x86_64-dvd.iso /mnt/iso iso9660 ro,loop 0 0
$ mount -a

2. Bootstrap - install createrepo
$ cd /mnt/iso/Server (it may be in some other folder)
$ rpm -Uvh createrepo*

3. Trouble with creating a repo of this mounted directory:
$ createrepo /mnt/iso/
Directory /mnt/iso/. must be writable.

4. Solution: Create another directory. Add a symlink to the mounted directories that contains the RPMs. Convert this new directory as the repo
$ mkdir /var/repo
$ ln -s /mnt/iso/Server /var/repo/rpms
$ ln -s /mnt/iso/VT /var/repo/rpms-vt

$ createrepo /var/repo
3040/3040 - rpms/zsh-html-4.2.6-3.el5.x86_64.rpm
Saving Primary metadata
Saving file lists metadata
Saving other metadata

5. Create repo conf file
$ vi /etc/yum.repos.d/myrepo.repo
[rhel-myrepo]
name=myrepo
baseurl=file:///var/myrepo
enabled=1
gpgcheck=0

6. Test the new repo
$ yum list installed

Setting up serial console

[ From: http://tx.downloads.xensource.com/downloads/docs/user/ ]

2.5.2 Serial Console (optional)

Serial console access allows you to manage, monitor, and interact with your system over a serial console. This can allow access from another nearby system via a null-modem (``LapLink'') cable or remotely via a serial concentrator.

You system's BIOS, bootloader (GRUB), Xen, Linux, and login access must each be individually configured for serial console access. It is not strictly necessary to have each component fully functional, but it can be quite useful.

For general information on serial console configuration under Linux, refer to the ``Remote Serial Console HOWTO'' at The Linux Documentation Project: http://www.tldp.org

2.5.2.1 Serial Console BIOS configuration

Enabling system serial console output neither enables nor disables serial capabilities in GRUB, Xen, or Linux, but may make remote management of your system more convenient by displaying POST and other boot messages over serial port and allowing remote BIOS configuration.

Refer to your hardware vendor's documentation for capabilities and procedures to enable BIOS serial redirection.

2.5.2.2 Serial Console GRUB configuration

Enabling GRUB serial console output neither enables nor disables Xen or Linux serial capabilities, but may made remote management of your system more convenient by displaying GRUB prompts, menus, and actions over serial port and allowing remote GRUB management.

Adding the following two lines to your GRUB configuration file, typically either /boot/grub/menu.lst or /boot/grub/grub.conf depending on your distro, will enable GRUB serial output.

serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout=10 serial console

Note that when both the serial port and the local monitor and keyboard are enabled, the text ``Press any key to continue'' will appear at both. Pressing a key on one device will cause GRUB to display to that device. The other device will see no output. If no key is pressed before the timeout period expires, the system will boot to the default GRUB boot entry.

Please refer to the GRUB documentation for further information.

2.5.2.3 Serial Console Xen configuration

Enabling Xen serial console output neither enables nor disables Linux kernel output or logging in to Linux over serial port. It does however allow you to monitor and log the Xen boot process via serial console and can be very useful in debugging.

In order to configure Xen serial console output, it is necessary to add a boot option to your GRUB config; e.g. replace the previous example kernel line with:

   kernel /boot/xen.gz dom0_mem=131072 com1=115200,8n1

This configures Xen to output on COM1 at 115,200 baud, 8 data bits, 1 stop bit and no parity. Modify these parameters for your environment.

One can also configure XenLinux to share the serial console; to achieve this append ``console=ttyS0'' to your module line.

2.5.2.4 Serial Console Linux configuration

Enabling Linux serial console output at boot neither enables nor disables logging in to Linux over serial port. It does however allow you to monitor and log the Linux boot process via serial console and can be very useful in debugging.

To enable Linux output at boot time, add the parameter console=ttyS0 (or ttyS1, ttyS2, etc.) to your kernel GRUB line. Under Xen, this might be:

  module /vmlinuz-2.6-xen0 ro root=/dev/VolGroup00/LogVol00 \
console=ttyS0, 115200

to enable output over ttyS0 at 115200 baud.

2.5.2.5 Serial Console Login configuration

Logging in to Linux via serial console, under Xen or otherwise, requires specifying a login prompt be started on the serial port. To permit root logins over serial console, the serial port must be added to /etc/securetty.

To automatically start a login prompt over the serial port, add the line:

c:2345:respawn:/sbin/mingetty ttyS0

to /etc/inittab. Run init q to force a reload of your inttab and start getty.

To enable root logins, add ttyS0 to /etc/securetty if not already present.

Your distribution may use an alternate getty; options include getty, mgetty and agetty. Consult your distribution's documentation for further information.