Blog in isolation
Digging in a Habari sandpit

• About
• Archives
• Contact

Thursday 18 March 2010

This is the way, step inside...

the mechanics of IT

Last year, I visited a customer in Swindon. When I arrived at the office, I noticed some water dripping from the underside at the front of my car. I didn't have time to investigate or get my hands dirty so I subconsciously hoped it was just condensation from the air conditioning unit and tried to forget all about it.

After work, I managed to navigate Swindon's world famous roundabout of roundabouts and finally locate my hotel. When I parked, I noticed the temperature gauge was reading high - very high. Sure enough, I got out of the car to see steam pouring out of the bonnet and was greeted by the familiar smell of anti-freeze.

I still didn't want to get my hands dirty so I opened the bonnet and just stared at the problem, hoping that would miraculously fix it. Then I checked in to my room and rang the AA. I introduced myself to the operator as 'Hi - my name is Norman and I'm an alcoholic.'

The AA man duly arrived before 'The One Show' had even finished. We both stood looking at the engine knowingly before he asked: 'Now, Sir, what seems to be the problem ?'

'Well, I think it's pretty minor and easy to fix. It's just the top hose has perished and needs replacing.'

'Oh I see, Sir. Why do you think that ?'

'Well, when the engine was running, steam was pouring out of the top corner of the radiator - right there where the water hose joins.'

'OK, Sir. Thanks for that. Please could you just turn the engine over for me so I can take a look myself ?'

Engine on. Water and steam billowing out. Smell of anti-freeze. Temperature gauge rising.

'Whoa ! That's fine, Sir. Engine off now, please. Well it's not your hose, Sir. The problem is over here. It's your bleed screw, Sir. Look (flips the screw from the middle of the radiator). This bleed screw has sheared off in half. Quite a common problem on this model. Seen it a couple of times now.'

I felt a little sheepish (but very relieved that a solution was in sight) and the man from the AA, James, filled the radiator with 13 pints of cold water and I followed him to a local garage where we parked and he kindly drove me back to the hotel so I could wash my hands.

The next morning, I walked into the customer's offices to be excitedly greeted with 'Oh good - glad you're here. We have got problems. Serious problems. Performance problems. On production.'

'Ah OK - what seems to be the problem ?'

'Well it's the database, you see. It's the hard disk - 99% busy. Partition 27b on logical volume 7 is overheating. Oh and another thing that might be relevant - the hit ratio is right down at 72.7%'

'OK - thanks for that. Do you think you could just start the system up for me and I'll take a look myself ?'

the importance of end users

Cary Millsap posts a brilliant article about his approach to performance troubleshooting that resonated loudly with me.

When I first started working at Siebel, a standard review was a production health check that consisted of meetings with key project staff (Siebel Administrator, DBA, systems and network admins, project manager, developers) coupled with some standard checks on key Web, Siebel and Oracle configuration parameters.

At the tail end of one engagement where I had precious little to note or report on, I asked to meet with an end user for a brief chat. The response was surprising and not dissimilar to the standard retorts Cary describes.

• 'Talk to a user ? Why on earth would you want to do that ?' (delivered with suspicious frown)
• 'The user won't understand your questions.' (patronising)
• 'The users don't understand the business requirements.' (surprising and worrying)
• 'The users are too busy.'
• 'Can't you just talk to a supervisor instead ?'

Of course, most of these responses are instinctive, defensive measures, immediate responses to what is probably a rather unusual request and, to be fair and in the interests of balance, I have been introduced to end users by development staff who do know and obviously have a healthy relationship with the user community.

If and when I succeeded in getting an audience with an actual user, I didn't actually chat at first. Instead I introduced myself and simply asked if I could watch the individual use the system for 10 minutes.

It was often very enlightening just to sit quietly and observe the business process (typically, handling an inbound call at a call centre), the subset of available screens accessed by the agent, typical searches executed to locate data, actions they expected to be slow, actions they needed to be fast, common tasks they did frequently, lengthy interactive queries that were truly batch reports as well as interactions with other applications.

onwards and upwards

After over five varied and enjoyable years working for Siebel (and then Oracle) in Expert Services, I am changing jobs. However, I will continue to work for Oracle as part of the group responsible for 'Social CRM'.

Although this was a internal transfer, my interview process was quite unusual. Instead of being forced to massage my CV into Microsoft Word format and send an email attachment to a faceless recruitment agency, I was able to publish my CV using Google Docs and the whole interview process was conducted by telephone.

A colleague had already thoughtfully pointed my prospective manager to a blog entry that was loosely relevant and he also was keen to review more technical oriented content on my Siebel related blog.

As an aside, you can only imagine the feelings and thoughts that flooded into my brain when my colleague uttered the immortal words: 'Hey Andy, I've given your <prospective new manager> a pointer to a couple of articles on your blog.'

The role will mean a few changes for me:

• I will now revert to having difficulty explaining to friends and relatives what I actually do.
• I will probably spend less time in airport lounges and more time with my family. Norma has already expressed grave reservations about this element of my career change.
• My manager will be in a different continent and timezone.
• I will be contributing to a corporate blog for the first time which will necessitate a slight shift in subject matter, terminology and slightly less of my dry, cynical, off the wall, British sense of humour although I do intend to maintain this personal blog.

I am currently tidying up a few loose ends in Expert Services and embarking on a emotional, sell-out, European farewell tour to all my favourite Siebel customers so I won't actually be starting my new role until mid-October but it will be an exciting change for me and I'm really looking forward to it.

UKOUG agenda

Monday 3 December

Get up very early and drive to Birmingham.

09:40-10:30 '30 years at Oracle' - Tom Kyte. I own a couple of Kyte's excellent books and various sources report he is an excellent speaker.

10:40-11:00 Visit a few stands. A quick game of 'spot the colleague' and ask 27 different companies 'How can XYZ help me grow my business ?'

11:15-12:00 'Siebel Keynote' - David Mills. Possible sales and marketing fluff alert. Need to sit at the back adjacent to an aisle to allow a potential rapid escape to 'Oracle RAC versus Oracle Data Guard - which should I use for Disaster Recovery and which should I use for High Availability ?'.

12:25-13:20 'Under the Covers of Oracle BI Suite Enterprise Edition Plus' - Mick Bull/Lisa Dobson. I would like to learn more about Oracle's Business Intelligence tools. Plus I have an innate weakness for presentations titled 'Under The Covers...'

13:15-14:15 'TimesTen: Anatomy of an In-Memory Database' - Chris Jenkins (Oracle). Curious to hear more about this technology.

14:15-15:15 'Remote Hand Held SFA solutions need housekeeping - Ian Keleher/Nicola Burrows (Gallaher). A verbose and hardly compelling title but I have worked with this customer so I might get a mention on the Credits slide.

15:20-16:25 'Oracle 10g: RAC Tuning Tips' - Joel Goodman

16:45-17:30 'Siebel Marketing and Marketing Analytics' - Ben Wales

17:35-18:35 '11g new features for DBAs' - Tom Kyte. This has better be worth it. I am missing 'Siebel Networking', free beer and prawn volavons for this !

Back to hotel to dump marketing literature, USB memory sticks and complimentary gifts from Quest Software.

Shower, brush teeth, apply deodorant and slip into something more comfortable. Head over to the Pitcher and Piano to gatecrash the Oracle bloggers meetup.

membership form for BAAG

I own up. It's a fair cop. It is 27 years and 5 months since my last confession. Since then, I have sinned, Holy Father. In fact, I have committed an absolutely heinous crime. Please forgive me for I have submitted multiple random guesses.

May the high priests (and priestesses) of the BAAG movement, please have mercy on my wretched soul.

Recently, I have been guilty of supporting and even proposing a multitude of 'any guesses', all of which were proposed as possible solutions to solve a critical system problem on a production system:

1. I failed to cough and splutter in an effort to stifle my laughter when a manager suggested that 500 European users adopted shift working to clear the backlog.
2. I didn't shout down a ludicrous proposal to reboot the Siebel Enterprise every 4 hours in order to maintain some level of service.
3. I wrote on a whiteboard that consideration should be given to reinstating the previous version of the application even though this in itself was risky, time-consuming and unlikely to address the root cause.
4. I even proposed patching to Oracle 10.2.0.3 with no supporting evidence whatsoever.
5. I shuffled nervously and blushed when the customer asked 'Who do we escalate to when you fail to fix this problem ?'
6. I watched in silence as the SAN man was stood against a wall and pelted with questions as senior management all pointed fingers in his direction.
7. I failed to stand up for righteousness and technical purity as runaway, rogue sessions were maliciously and arbitrarily terminated by an Oracle DBA.
8. I failed to suggest running a trivial SQL to determine index fragmentation and stood by as 74 indexes were needlessly reorganised.
9. I watched helplessly as a systems administrator claimed the problem was 'definitely in the underlying disk I/O subsystem' simply because 8,000 operations a second were being performed.
10. I failed to raise my eyes skywards and embark on a spontaneous two hour training session with a DBA who claimed the 'buffer cache hit ratio was fine' and the problems only started once users were allowed onto the system.

My only defence is that all of the above 'any guesses' created enough of a diversionary smokescreen to buy me enough time, alone in a darkened room, to analyze multiple Statspack reports, reproduce the majority of the problems in SQL*Plus and then prove the behaviour was improved when statistics on empty tables were dropped and two additional indexes created.

This lucky 'guess' miraculously restored performance and stability. So you see, 'any guess' is not always such a bad strategy, after all.