China gets horny
Today I woke up to several alerts from Linode informing me that one of my VPS nodes was exceeding the Disk I/O threshold that I had set. Curiously this VPS is used as a HTTP web proxy and whilst it gets about 300-400 visitors per day (mainly china) this morning I was seeing over 800 visitors in Google Analytics.
Attempting to ssh to the server failed with timeouts although the PHP web application was still responding to requests over HTTP fine. I suspect sshd was failing to reverse-lookup my IP address in any reasonable amount of time, or perhaps IP Tables – (Note to self: Look in to why that happened). Thankfully Linode provide out of band / console access via SSH and AJax so all was not lost.
Looking at the Network rrdgraph it shows that the server was approaching 7Mbit/s of HTTP traffic and almost 50GB had been consumed today alone. Whilst the server seemed to handle the load without problem (minus ssh access) consuming 50GB+ per day would quickly max out my monthly data transfer allowance with Linode – this wasn’t acceptable. I modified the firewall to accept HTTP/HTTPS traffic from my IP only in order to investigate and the load suddenly stopped and SSH was alive again.
Initially I had suspected that some sort of automated bot was using ehproxy.info to do automated scans and attacks but a closer inspection of the traffic showed an even number of distributed IPs (all from China – as Google Analytics confirms) all clicking various porn sites. I guess everyone in China was feeling horny this afternoon!
Further analysis of the access.log shows that the server (Linode XenU VPS with 720MB of ram) was handling 62 hits sec (2428863/39600) and lighttpd was dealing with the load no problem. Pretty good considering this is a pure PHP application utilising php-cgi.
For the record, the top five IP addresses were:
Hits : IP address
Show limits of a running process in Linux
A rather simple but often asked question was put forward to me today: How can I see the maximum amount of file descriptors my running process can open? (without killing the process!)
Typically one would say ‘check ulimit -n’ but lets say that a thread-driven or event-driven application like varnish or lighttpd is configured with an arbitrary amount of open file descriptors and you want to verify that they have taken effect before the application crashes.
A simple way to check this (atleast on Linux 2.6.26-1 or later) is to run:
svr1:~# awk '/Max open files/{ print $4}' /proc/$(pgrep -n lighttpd)/limits
1024
As you can see the above command returned the value of max open files for the running process. This means you can be sure that your lighttpd or varnish application will not suddenly die after being starved of file descriptors!
I have included the entire output of the limits table for the lighttpd process for completeness:
svr1:~# cat /proc/$(pgrep -n lighttpd)/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited ms Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 5824 5824 processes Max open files 1024 1024 files Max locked memory 32768 32768 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 5824 5824 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
Error: Device 0 (vif) could not be connected. Hotplug scripts not working
Are you running Xen / “xm create” and you get this error?
Try this (RHEL/CentOS): service haldaemon start
- and have a nice day!
Breaking your mac with the hwprefs command
The hwprefs(1) command shows you low level information about your Mac, It’s CPU, the Memory and OS X. It also lets you control low level CPU and memory options on your mac. You can also do things like disable a CPU and disable the L1/L2/L3 CPU caches. Similar to psradm(1) in Solaris / sparc64. Be warned!
hwprefs 4.5.0 usage: hwprefs [options] parameter hwprefs [options] parameter=value OPTIONS -v verbose mode -h display this help message PROCESSOR COUNT cpu_count {1..N} number of physical processors available to MacOS X cpu_enable {1..N} enable a specific processor cpu_disable {1..N} disable a specific processor cpu_ht {true, false} processor hyperthreading PROCESSOR CACHES cpu_l1_cache {true, false} processor l1 cache cpu_l2_cache {true, false} processor l2 cache cpu_l3_cache {true, false} processor l3 cache POWER SAVING cpu_nap {true, false} processor nap INFO AND STATUS (READ ONLY) os_class displays OS class {Cheetah, Puma, Jaguar, Smeagol, Panther, Tiger} os_type displays operating system type machine_type displays machine type memory_size displays system memory cpu_type displays processor type and version cpu_freq displays processor clock frequency cpu_bus_freq displays processor bus frequency memctl_type displays memory controller type ioctl_type displays io controller type
Linux growing partitions on the fly with blockdev(1)
One thing I wished that Linux could do was to dynamically grow a partition online. This is something I was accustom to in HP-UX, AIX and even Solaris. It’s a pretty common operation to do in an enterprise environment with SAN LUNs. Lets say I have a web server running Apache on Linux. The htdocs dir sits on a dedicated SAN LUN and is slowly filling up. This server is a production box. Everything is using LVM. Before we can expand the FS, LV, or even the VG we need to grow the physical parition that the VG lives on. We expand the physical LUN size on the SAN, and fdisk the partition in Linux but linux still does not see the updated partition table size (without a reboot) – this is not good!
However, do not fear! I come across the very command to address this problem: blockdev –rereadpt /dev/sdX
Sigh… I wish I came across this command years ago!
DMT tool for Speedtouch
I’ve just come across this awesome little utility called “DMT” for Speedtouch and similar routers. Speedtouch routers have a telnet interface that let you access a fairly powerful command line interface. You can configure advanced aspects such as IDS, SNMP, etc. DMT is essentially a GUI to the telnet interface providing a complete (and awesome) overview to your Speedtouch router. Checkout the screenshot below.
DMT can be downloaded from http://www.kitz.co.uk/routers/DMTv7.htm
Home network / patchsee cables!
I’ve just moved in to my new home and I decided that I will get rid of my old long cat5e network cables and order specific length Patchsee Cat6a network cables. Whilst I don’t actually need these cables, they’re aesthetically pleasing and technically superior. I highly recommend them!
Patchsee Cat6A UTP cable
Where I work…
I recently I joined a leading e-commerce company as a UNIX system administrator.
This is where I work, and this is what I do:
Data Centre Patch Panels…
I spent Friday wiring up RJ-45 patch panels to go in to our new lovely APC racks. Usually, we pay a contractor an extortionate amount of money to do this, but there is a recession, you know! It’s a fairly simple thing to do, cut once, measure twice, learn the wiring standard, bla bla – but boy does my hand hurt after doing this all day!
