OABGen encountered file error ffffffff
The problem started with users reporting that their Offline Address Book (OAB) was missing some recent new starters, actually they said Global Address List (GAL) but that misconception is another story for another day. Upon investigation it seemed that the issue was a global issue. So where to start… I first thought right let’s think through the process for Exchange 2010 to generate the OAB and for Outlook clients to pick this up. This process is as follows:
- One Exchange server is the OAB generation server. This server will create a fresh OAB plus a set of delta files depending on the schedule set within Exchange
- The Client Access Servers (CAS) will poll this generation server every 8 hours. The list of CAS servers that do this polling is configurable or can be set to all CAS servers in the environment
- The Outlook client will look for OAB updates every 24 hours
So, I knew that Outlook wasn’t getting updates, so I checked a number of the 100+ CAS servers, who all seemed to have an old OAB. Right moving onto the OAB generation server I found that it too had an old OAB. Where to look now…? Like every good admin I went to the Application Event Log and lo and behold found a set of OABGen errors. Oh Oh…
- OABGen encountered file error ffffffff (internal ID 5050237) while generating the offline address list for address list ‘\Global Address List’. Make sure there is enough disk space available.- \Default Offline Address Book
- OABGen detected that the file ‘\\SERVERNAME\ExchangeOAB\1c3e9f63-a69d-4bef-9c76-26106a110574\21ce54cd-3ac0-4a04-90f5-0defca52bd79-data-1585.lzx’ is corrupted or missing. This indicates data tampering or disk problems. Restore files in this folder from the recent backup or clean up folder content and force a full OAB generation.- \Default Offline Address Book
- OABGen encountered file error ffffffff (internal ID 505026e) while generating the offline address list for address list ‘\Global Address List’. Make sure there is enough disk space available.- \Default Offline Address Book
- OABGen encountered an error while publishing the OAB files to the distribution point ‘\Default Offline Address Book’. Check other logged events to see more information about the problem.- \Default Offline Address Book
Ok it says disk space issues, however the OS is saying that the disk has multiple GB of free space, so that isn’t quite right. Of course I don’t have recent backups of the OAB Generation directory, and creating a brand new OAB isn’t an option due to the global size of the company with >200,000 Outlook clients pulling a 40MB+ OAB… What now…?
I checked online and couldn’t find any more elegant solution that to trash the OAB and create a new one, but not being happy with that option I decided to think a little harder.
I could see the OAB folder D:\Program Files\Microsoft\Exchange Server\ExchangeOAB\1c3e9f63-a69d-4bef-9c76-26106a110574. This folder contains oab.xml that describes the content of the folder and a whole lot of lzx files being the full OAB and the deltas. Now I noticed that the oab.xml file contained file names like 21ce54cd-3ac0-4a04-90f5-0defca52bd79-data-1585.lzx (i.e. all ending with 1584) yet the directory also included files ending with 1585. Hmm I wonder if Exchange is reporting disk full because it is trying to create 1585.lzx files but gets an error since the files already exist… I’ve seen that type of thing before in my development days, and thought surely Exchange isn’t that poorly handling this situation…?
I copied the entire ExchangeOAB folder to a temp location just in case I mucked it up, and then deleted all of the lzx files ending with 1585, and then reran the OAB generation and sure enough it worked!!! All the old deltas stayed in tact and the new 1585 deltas got recreated and the oab.xml file got updated to reflect the new OAB state.
Now as for why Exchange got into this state I can only summize that the server got rebooted just when Exchange was writing out the results of the OAB generation and therefore got itself in a pickle. At least now I know how to recover from this situation without 200,000 Outlook clients all downloading a fresh new OAB of 40MB+…
Good days work, time for a self congratulory drink :o)