wasabi2k.com

You may have noticed the ads for Telecom Big Time on TV recently. Flat rate! No Caps! Traffic Managed! uh oh.

First off, flat rate never ever works for long. Second, Traffic Management is never a good thing. It is normally sold as keeping it fair for everyone and only affecting those dastardly pirates. In reality in usually means making everyone suffer.

I had a colleague at work switch to Big Time. He had nothing but positive things to say. Slow to 200k from 5PM-12AM, but then full rate, perfectly usable during the day etc. He had pulled down something like 100 gig in a week or so. He also said that his SSL usenet downloads were unaffected. As I had recently hit my 40 Gig cap on my Telecom Pro plan, I thought I would give it a go.

One point: no static IPs on Big Time!

I filled out the online form (hooray for not having to call!) and the next day I was on Big Time. I did a speed test at 8:00AM that morning and I got ok speeds (8Mb down, 0.6 up) which were below my previously plan speeds (12Mb down, 1.4 up). I left for work excited at the prospect of massive downloads. Getting home that evening I fired up a few downloads using Free Download Manager, a multi part download tool. My speeds hit around 900k down using HTTP and HTTPS. Fantastic!

Youtube videos also loaded really quickly. Apparently due to some fancy caching going on at Telecom’s end.

However, about half an hour later bad things started to happen.

• My downloads slowed to 100k, then 60k.
• I started having problems with my download accelerator. Somehow the HTTP/S traffic was being mangled to the point that my app was reporting that the server didn’t support resuming downloads, which meant no multi part downloads.
• Single stream downloads would range from about 150k to an awesome 3k/sec. This is regardless of protocol (HTTP or HTTPS) or source (US, NZ)
• The same single stream downloads would periodically stop completely. Couple with the HTTP mangling meant no resuming.

In short, my internet connection was an inconsistent, unpredictable pile of shit. Some examples:

• My stepson jumped on to play Battlefield Heroes. A new version was released so we left it to download. It took over 5 hours.
• I went to Windows Update my laptop, 186MB of updates (Office 2007 SP2) took over 2 hours.
• My wife’s VPN connection dropped every 10 minutes.

Long story short, I changed plan on the 21st of October. I changed back on the 26th. I would rather pay the extra 2c a MB than sit with this pathetic excuse for an internet connection.

To be fair, I did download 30GB of stuff over the 4 or 5 days I was on this plan. Also Telecom were very prompt with changing me back and forth. It is great to be able to do this all online without having to call them.

In conclusion, Big Time delivers exactly what you expect from a traffic shaped flat rate service. terrible, terrible performance.

Introduction

The best projects are those that teach you something new. I’ve recently been tasked with decomissioning a RADIUS Server. Not a major task when you first think about it, but the plot thickens.

1. This RADIUS server is running Netware 6.
2. RADIUS is provided by BorderManager.
3. It is used to authenticate ADSL Routers, CDMA Devices and dial up connections all over the country.
4. It authenticates against E-Directory.
5. That particular E-Directory tree is not replicated anywhere.
6. Noone knows the usernames and passwords for the ADSL Routers around the country, so we can’t change user credentials on them.
7. Noone knows the shared secret for the RADIUS Clients.
8. Once connected users connect a PPTP VPN to a Windows Server which authenticates against AD.
9. Some connections have multiple users behind them.

Investigation
Oh man this is going to be fun. I’ve got no Netware or Borderware experience and the people who setup this system are long gone. There is no record of anything and we have no idea as to the number of connections in use, what they are used for and who uses them.

The first step is to figure out our current state. To do this we have to do some pretty heavy log analysis. Our data sources are:

1. RADIUS Debug logs: Shows all Access-Request, Accounting and other RADIUS Requests for a 6 month period.
2. VPN logs: Shows all PPTP VPN connections for a 6 month period.
3. Export of E-Directory accounts.
4. Export of Active Directory accounts.

The initial stats from these were a great start. We were able to export a list of unique connections over the past 6 months, with an idea of frequency of use. We were then left with a list of RADIUS connections (ADSL, CMDA and Dial-Up) and a list of VPN Connections, both with a fair amount of contact info such as name or telephone number. The next hurdle was mapping WHICH VPN users used which RADIUS connection.

We first matched the obvious ones, those connections with E-Directory and AD accounts with the same names. We were still left with a large number of unmatched accounts. Had the RADIUS connections had static IPs matching these would have been trivial. Unfortunately they were all dynamic, meaning that one address could be mapped to any number of RADIUS connections over the time period. After much hacking at the data I ended up importing it into Access so I could try and perform some SQL magic. Much googling later I ended up with the following query:

SELECT RADIUSLogs.Date AS RADIUSLogs_Date, RADIUSLogs.Time AS RADIUSLogs_Time, RADIUSLogs.[User-Name], RADIUSLogs.[Framed-IP-Address], VPNLogs.Date AS VPNLogs_Date, VPNLogs.Time AS VPNLogs_Time, VPNLogs.Username, VPNLogs.RemoteIP

FROM RADIUSLogs INNER JOIN VPNLogs ON (RADIUSLogs.Date=VPNLogs.Date) AND (RADIUSLogs.[Framed-IP-Address]=VPNLogs.RemoteIP)

WHERE VPNLogs.Time>=dateadd(”n”,-15,RadiusLogs.Time) And VPNLogs.Time<=dateadd(”n”,+15,RadiusLogs.Time) And VPNLogs.Time>RadiusLogs.Time

ORDER BY RADIUSLogs.[User-Name], VPNLogs.Username;

which joins the RADIUS and VPN logs by Date and IP Address and shows results where the VPN connection was connected within 15 minutes of the RADIUS connection connecting, bearing in mind that connected RADIUS connections sent accounting requests every ten minutes. This query can then be used to populate a Pivot Table mapping RADIUS connections to VPN connections, giving me a list of VPN users per RADIUS connection. This is by no means full proof. I can think of several situations in which the results returned will be invalid, but it is a first step. The output of this was run by a business user who helped identify those entries that didn’t make much sense and they were removed. We now had a list of RADIUS connections and the VPN users behind them.

RADIUS Migration
At this stage we decided to migrate the RADIUS functionality to a Windows IAS Server. This would mean that authentication would be performed against Active Directory rather than E-Directory. By a stroke of luck a decent number of the E Directory accounts used were already synchronised with AD using Identity Manager. Unfortunately a large number were not, which left us in a difficult situation. As stated we are unable to change the usernames and passwords used on the ADSL routers as we don’t have passwords for them and now we have accounts that we don’t have passwords for.

I did a bit of Googling. It is possible to capture RADIUS traffic, but the password is encrypted with a shared secret. Then I found the Wireshark Wiki page on RADIUS (http://wiki.wireshark.org/Radius). Wireshark can quite happily decrypt RADIUS if you configure the RADIUS shared secret within the app. To do this:

1. Open Wireshark. Go to Edit, Preferences
2. Expand Protocols, then go to RADIUS.
3. Populate the Shared Secret field.

So now all I needed was the RADIUS shared secret. This is stored in BorderManager and cannot be exported. I tried various asterisk reveal apps with no luck. Luckily the company who maintains the RADIUS proxy servers which forward the authentication requests (Telecom) were able to give this to me.

Next problem, how do I capture network traffic on a Novell server? I can’t install Wireshark locally (it may be possible but god knows how!). Network Team to the rescue here. I got a colleague to mirror the port that the RADIUS server was on to an unused NIC on an old server. This means that all traffic to and from the RADIUS server also shows up on this unused NIC. I installed Wireshark, unbound TCP/IP from the spare adapter (just in case it caused issues) and started capturing.

It is impossible to try and watch unfiltered traffic so I started with the following display filter: radius

This shows all RADIUS traffic. Simple and effective. However it soon became apparent that I was only interested in the

Access-Request packets, not the accounting ones. So after a bit of playing I had the following: radius.code==1. RADIUS code 1 is Access-Request, which is the packet that contains the username and password.

Now I could catch every authentication request sent to the Novell RADIUS server and hopefully see the username and password used. I crossed my fingers and waited.

It worked. As you can see from the  packet below we can now see the username and decrypted password!

We left Wireshark Capturing for a week and managed to capture all but 5 of the passwords we needed. Phone calls to the remaining sites asking them to reboot their routers got the rest.

All that is left to do is configure IAS on the Windows Server and perform the change. I’ll write about those in another post soon. In the meantime I’m going to be sitting here feeling happy with myself for learning something new.

I’m in the middle of a Network Segmentation project at the moment, which involves splitting a previously flat network into various subnets/vlans for security and management reasons. End of the day it involves changing IP addresses on servers.

I recently completed the segmentation of the VMWare environment, which includes around 50 hosts and a VirtualCentre server. I ran into some issues which took a bloody long time to fix, so I thought I’d share my experiences.

First off you will need the following:

1. Out of band management (such as iLO or Console Access) to your ESX hosts. SSH is NOT out of band management. You WILL lose IP connectivity to your host during these changes.
2. root logins for your esx hosts.
3. Knowledge of your network, specifically what VLAN your new ESX host Service Console is going to be on and what IPs/Gateways to use. For this guide I am assuming that the Service Console will be on a tagged VLAN presented to the ESX host. If you don’t know what that means then ask your network person.
4. An outage window for the VirtualCentre move. You will have to disconnect and reconnect ALL your ESX hosts after this change, which will mean they won’t be able to be managed for a while.
5. If you do not have licenses for VMotion this is going to be a lot harder and more disruptive. For the purposes of this guide I will assume you DO have VMotion available.

Before we start, a word about name resolution. VMWare is very reliant of DNS for operation of its various services. It is necessary for you to know how your name resolution works. In my case, each host used a HOSTS file (/etc/hosts) to resolve names. You may use this method, or a DNS server. Find this out now.

Also before we start, disable HA on your clusters if you can, this will stop any migrations while you are trying to disconnect/reconnect hosts. You can do this by right clicking your cluster and clicking Edit Settings, then removing the checkbox from VMWare HA.

Section 1: Change VirtualCentre IP

1. Log on to Server hosting VirtualCentre. Stop VirtualCentre service.
2. Change IP Address/Gateway
3. Change port VLAN assignment if required
4. Reconnect to Server.
5. Run ipconfig /flushdns, ipconfig /registerdns and nbtstat -RR
6. Start VirtualCentre Service.
7. Test connectivity from your VirtualCentre server to your hosts (ping).
8. Connect to VirtualCentre using Virtual Infrastructure client.
9. Confirm it loads cluster information successfully (lists Clusters/Hosts etc, disregard their status for now, if they show as offline that’s OK. Don’t worry your VMs are still running).

At this point you will need to update either your /etc/hosts file on each server or make sure that your dns server returns the correct address when you lookup the name of your VirtualCentre server. If not, update the A record.

Your hosts will probably all be showing up as offline at this stage. Don’t worry. We’ll soon fix that.

1. In the Virtual Infrastructure client, right click the first host in your cluster and select disconnect. Wait for this to complete, then right click again and select connect. Once this is complete the host will show up online and be able to be managed again.
2. Repeat for each host in the Cluster. Again, this will NOT pause or stop any VMs running on the affected host.

Once you’ve disconnected and reconnected all your hosts your VI Client should look all happy campers again, with all your hosts online. If you are not going to change the ESX Host IPs you’re done. Re-enable HA and test VMotion, Cloning etc.

Section 2: Change ESX Host IP

1. Open your VI Client and connect to your VirtualCentre Server.
2. Right click on your first host and select “Enter Maintenance Mode”. At this point all the running VMs should be VMotioned off to another server automatically. If this doesn’t occur you will need to manually migrate them by clicking each VM and selecting Migrate, then selecting a destination. Don’t move them all to one host or you will probably hit major performance issues.
3. Once the host is in Maintenance Mode, right click it and select Remove. Once this is complete the ESX host should no longer be in the console.
4. Connect to the Host using iLO or a Keyboard and Mouse.
5. Push Alt-F1 to get a console, you should be presented with a username prompt.
6. Login as root.
7. Run the following command, where XX is the VLAN that your new Service Console IP address is on: ‘esxcfg-vswitch -p “Service Console” -v XX vSwitch0′. This will change the VLAN that your Service Console listens on.
8. Run the following command, where X.X.X.X is the new IP address of your Service Console: ‘esxcfg-vswif -i X.X.X.X vswif0′. This will change the IP address of the Service Console.
9. Add a new default route using the following command, where X.X.X.X is the default gateway for your new service console: ‘route add default gw X.X.X.X’.
10. Test connectivity to the VirtualCentre server using ping.
11. You can now log out of the console using Ctrl-D and close your iLO.
12. At this stage you need to update any DNS/Hosts files to make sure the Host’s name resolves correctly to the new IP address (especially on the VirtualCentre server).
13. Go back to your VI client, right-click your Cluster and select “Add Host”.
14. Put in the hostname, username and password for the host you just changed the IP address on.
15. The host should be discovered and added to the cluster, and should still be in maintenance mode.
16. Click on the host, and in the right pane select Configuration then Networking. Click Properties on vSwitch0.
17. Select your Service Console and click Edit. When prompted select “Continue modifying this connection”
18. VLAN ID, IP address and Subnet mask should all be populated, but Default Gateway will be blank, Click Edit.
19. Fill out the Default Gateway field under Service Console, then click OK and exit the configuration.
20. Right click your host and select “Exit Maintenance Mode”. Once this is complete your host is ready to go.
21. Repeat this process for all your other hosts, making sure to update either your DNS server or hosts file each time you change an IP.
22. Once all your hosts are done, you will need to test that your Cluster is still functioning.

If you do not remove and re-add the host, the VI client will still show it and it will appear healthy. However some automated operations, such as Cloning or VMotion will fail with cryptic, unhelpful messages. Removing and re-adding the hosts is the ONLY way to successfully resolve this.

So there it is. In my case I didn’t discover the need to remove and re-add hosts until after I had made all my changes (only disconnecting and reconnecting rather than removing). This meant that I had to do all my changes 2 or 3 times.

Save yourself some time and follow the process.

So Google finally came out with a browser. Step 2 on their world domination plans is complete.

I have to say I like it so far. The uncomplicated interface is wonderful. There is no title bar which seems to make a huge difference. Most of my screen real estate is used by the web pages I’m visiting, which is as it should be. The general look is clean and cheerful, it makes IE8 look a tad cluttered.

Pages load quickly, but it’s been a while since I’ve had anything to complain about in that regard.  Javascript performance is supposedly improved but again, I can’t remember the last time I had issues with the speed of Javascript rendering.

Tabs are sandboxed/jailed which is a good idea in theory. It remains to be seen how secure their implementation really is. I’ve yet to make use of their phishing/blacklist features. I’d be interested to know how these lists are maintained and how open to poisining they are, much as DNSBLs are for mail. This feature can be turned off.

Task Manager (I wonder is Microsoft will complain about the name) shows all your browser tabs and processes along with their associated CPU/Network/Memory usage. Handy.

 

Google Chrome Task Manager

 

 

Where does Chrome fall down? The usual suspects. NTLM proxy authentication. Getting prompted for my credentials going through the proxy is not acceptable. Manageability. As with firefox there is no easy way for an administrator to lock down/configure Chrome using policies. Through from the looks of things Chrome uses IE’s proxy settings, so that should help in that regard. I imagine the other settings are stored in either the registry or config files so it’s scripts ahoy. Finally, deployment. The installation is performed by running a 500k installer which pulls down the app and installs it, not really workable in an enterprise environment. To be fair this is a beta and I don’t see google targeting the enterprise space at this early stage but fixing these 3 things would go a long way to easing acceptance in the workplace.

It is also a very simple browser. The options tab is no-nonsense, with a grand total of 3 tabs and not a lot of settings. In using and looking at it I haven’t really been comparing it to IE and Firefox as much as I have in using either of those products.

Using Google Chrome you get the impression that there is more to this browser launch than we all think. The minor, but significant changes they have made to the UI hint at a move towards making the browser more of the focal point. To put it another way by removing the title bar and increasing screen real estate the Google Chrome browser has more presence that your average application window. If you are using Google Docs/Gmail/Calendar etc then the browser you use effectively becomes your OS. We have already seen linux distros which use Firefox as the Desktop, I think Google Chrome is going to be doing much the same thing in the future.

Overall, a clean, lightweight, no-nonsense browser that has quite happily replaced IE8 on my home PC as my main browser.

In my reading I have come across several places that claim that NFS offers much better throughput for streaming video content than traditional SMB. I have a Mediaportal (http://www.team-mediaportal.com) PC at home which usffers from occasional stuttering so I thought I’d give it a go.

A quick google for Windows NFS Server brings up a load of crap. There are a number of commercial NFS server packages for windows which all cost money. There are also the occassional free ones about too, but these seem to be either horrendously bad or written for Windows 95. What you really want is Microsoft Windows Services for Unix, which can be had from here.

At 200 off meg it’s not a small download, so be prepared. Once downloaded extract it and start your install. For the Mediaportal/GP PVR client you will want to install the NFS Client only, for your box holding all the media you will want to install the NFS server and prerequisites.  Now the fun part.

For authentication purposes you can either copy the /etc/passwd and /etc/group files from an existing linux box to your PCs or you can setup NIS. NIS is basically the central authentication functions of AD for linux. Seeing as I have a linux VM I decided to go down that route as I couldn’t be bothered with copying files all over the place.

So on my debian VM I did the following:

apt-get install nis, select NIS domain name (can be whatever).  (I had to ctrl-c out of it as it hung at starting)
edit /etc/defaults/nis, change NISSERVER=MASTER and CLIENT=FALSE
/usr/lib/yp/ypinit -m
/etc/init.d/nis stop

(there is a much better written and more comprehensive guide at here.)

With that done, go back to your windows boxes. When prompted say use a NIS server. Finish setup.

You will now have a new Start Menu Entry for Windows Services for Unix. Open this up and go to User Mapping. What you need to do here is map a Windows Username to a Unix/Linux UID. For example, I have a linux user (wasabi2k) which I map to MEDIABOX\Jeff. I create that mapping on MEDIABOX. I then create a mapping between wasabi2k and WASABI2K\Jeff on my main PC (WASABI2K) which has all my media. So as far as NFS is concerned when I access the NFS share on WASABI2K from MEDIABOX I am Linux UID 1000 (wasabi2k), which has permissions for that share.

Now that you have mapped some users you can share some folders with NFS on the PC that you installed the NFS server on. Right click on a folder in explorer and you will see a new NFS Sharing tab. The default permissions give Read-Only access, which is perfect for what we want.

Once shared, go back to your client and map a drive, except use the format server:/share. For example my movies are at wasabi2k:/movies (as opposed to \\wasabi2k\movies). When you first access/map a drive it will tell you who you are authenticated as and ask if you wish to continue with that user or change.

So after about half an hour I managed to get my PCs talking NFS. I setup my shares in MediaPortal and it worked straight away. Is there any difference in performance? Well.. I’d like to say yes but I haven’t really noticed. It doesn’t seem to skip anymore which is great but I’ve only tested it for about an hour. At the end of the day it was actually just fun setting up something different on a Windows box.

I am currently working on a server consolidation project for a customer. The usual story, customer sets up a VMWare environment, loves it, then proceeds to create a new server whenever someone has a good idea for an application. End result, 1000 staff, 500 servers, ridiculous licensing and support costs.

Our job is to consolidate these servers, which is a tough call. Servers include SQL, IIS, various applications (Projectwise, Knowledge Exchange, Provision etc). The server I am writing about happened to be SQL.

I have a fair bit of experience with SQL. I learnt a lot about RDBMS’ in my high school years playing with PHP/MySQL. I then transferred a lot of those concepts over to MS SQL in my previous jobs, but I have never had any formal training with SQL, hence my vagueness on some concepts.

We successfully moved the database in question to our new consolidation server. Nothing fancy, SQL 2000 SP4 to SQL 2000 SP4. The application was updated, which involved some ehavy editing of a large excel spreadsheet then rewriting a small VB6 utility which uploads excel checklists to a share on a server (which was hard coded in the app, now reads from an ini file). New SQL agent jobs were created in place of the old ones, insane DTS tasks (30+ steps) were updated and all went well. Until month end.

One of the SQL Agent jobs starts a stored procedure which sums data and generates averages into other tables, which is then used for reporting. Come the first of the month the nice monthly totals of around 50-70 a month were replaced with 2000 items for January 08 and 0 for the rest of the year. Oh dear.

The stored procedure which did all this in fact runs 6 more stored procedures, all of which do some pretty funky LEFT OUTER JOINs and all sorts of things that were well beyond my command of SQL. I spent a considerable amount of time going through everything I knew to try and understand the problem. Luckily we still had the old server and database (and several backups) so there was no risk of data loss.

I eventually found that there were several views in the database which worked fine on the old server but not on the new one, where they generated errors relating to conversions generating invalid values for field types (The conversion of char data type to smalldatetime data type resulted in an out-of-range smalldatetime value). Google suggested incorrect regional settings, which were dutifuly checked. Then I came across the following webpage: http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=74074 which put me onto the right track.

Executing “DBCC useroptions” on the old server showed the language of the user as British English and dateformat at dmy. Executing it on the new server showed English and mdy. I used sp_configure to set both to British English for the account used to run the stored procedure and voila, all my reports came back beautifully.

Some further reading suggests that hitting issues like this is usually the result of sloppy or lazy coding on the behalf of the original coder. The issue here appeared to be that the date fields were being compared to another table which contained obviously human entered “readable” dates and from this a specific date code generated. To do this the original date was first stripped of the time portion (no problem) then converted to a string, manipulated and compared, then converted back to a datetime field. Overall a nightmare to try and figure out.

End of the day the app is all working again. 1 down, 499 to go.

One of our clients uses a hosted exchange service. You manage their account using a tool they have developed called Q Administrator.

Recently I was asked to modify an account so I fired up Q Administrator and was told I had no access to manage companies. Annoyed, I decided to upgrade the app to see if that made any difference.

Bad idea. New version logs in and crashes with a 0xc000005 mscomctl.ocx error. Crap. Try various workarounds with no luck. Dependency Walker points at a missing dll, dwmapi.dll. This opens an entirely new kettle of fish. Apparently some Microsoft DLLs (that come with IE7 apparently?) are coming out with dependencies on dwmapi.dll, which is only included on Windows Vista machines. oh dear. This means us lowly XP peasants can’t use apps that use those DLLs.

To test this I chucked the app on VM that had IE6. Sure enough, no issues. What a prick. This seems to be getting more and more widespread and Microsoft have yet to come up with a fix. In the meantime, try and keep a machine or VM somewhere without IE7 on it.

So if your VB apps mysteriously start crashing with 0xc000005 errors to do with mccomctl.ocx, point the finger at IE7.

We’ve recently deployed Exchange 2007 at our work. It was a fun process as Exchange 2007 is not a simple upgrade. It is completely different in pretty much every way.

 I will write an update on Exchange 2007 another day. Today i want to bring to your attention a fun bug that made it to production.

 Shortly after release Update Rollout 1 was released, you can find it at: http://support.microsoft.com/?kbid=930809. The specific problem I mentioned is http://support.microsoft.com/kb/932487/.

The Microsoft Exchange Information Store service stops unexpectedly when the Exchange Server 2007-based server replicates the Public folder. Yep you read that correctly. Turn on diagnostic logging for Replication DS Updates and then try and replicate a public folder.

Bam. Information Store goes down and corrupts everything in it. That includes the Mailbox Store.

 What you see in Event Viewer:

Watson report about to be sent to dw20.exe for process id: 6124, with parameters: E12N, c-rtl-AMD64, 08.00.0685.024, store.exe, mdbrest.dll, 4df20, c0000005, bf0b, 08.00.0685.024Faulting application store.exe, version 8.0.685.24, faulting module mdbrest.dll, version 8.0.685.24, fault address 0×000000000004df20.

Then your infostore crashes. If you don’t mount your public folder store or have replciation going you’re fine. So if that’s you, apply the Rollup package or disable diagnostic logging. That way you won’t be up till 3am recovering mailbox stores.

So we’ve been running OCS for a month now and it’s time for a summary and a look at what we got out of it.

1. Instant Messaging is a fun tool, but we are not utilising it enough between our offices
2. Presence information can be very useful but only if it is kept up to date. Many suers simply mark themselves as busy so as not to be disturbed.
3. Use of the Communication Server features is not common. Despite providing documentation and training the majority of users are apathetic.
4. LiveMeeting is a fantastic tool.
5. The requirement to deploy the software is a pain as remote users and offices are often not technical enough to source and install the application.
6. Communicator Web Access is excellent for technicians on the road who need to contact engineers in the office for help. However it can be very slow on medium to low speed connections.
7. The requirement for CWA (Communicator Web Access) and your Front End Servers to be on separate machines is very frustrating. Having to dedicate an entire server to providing CWA makes it both a pain to maintain and manage and difficult to justify.

  So while it was great fun deploying it we have yet to see any business changing benefits. Stay tuned as next week we’ll be looking at Systems Operation Manager 2007 in a distributed domain environment! I love new products.