I often get asked to upgrade clients’ TFS instances to newer versions, apply updates, etc. and I swear it never “just works”. Well, it did once, and it was a server I had literally JUST upgraded personally to TFS 2012 that no one had touched since My challenges in upgrading TFS instances are rarely due to problems with TFS itself. 99% of the time someone got paranoid about an account having ‘sysadmin’ rights on TFS databases, or Administrator rights on one of the TFS servers, and removed it without telling anyone and suddenly boom! Of course “boom” doesn’t happen until I am in there maintaining the server or performing an upgrade... So, onto my story about update 3.
It seems every time I get error messages back from the TFS install wizard, they’re not documented (don’t get me started on that rant!), no one else has blogged about it, and no one has asked about it in a forum that I can find. So I’m posting it myself so the next person to run into this doesn’t waste a lot of time troubleshooting.
So, I was upgrading from TFS 2012.0 to TFS 2012.3. I upgraded the server from TFS 2010 on SQL 2008 to TFS 2012 on SQL 2012 myself back in January of this year, with just a few minor hiccups. I returned to the client recently to do some customizations to discover they had undone some of the security setup I worked on. :: heavy sigh:: After granting my account all the necessary rights to effectively administer TFS I started working on the upgrade to TFS 2012.3. About 20 minutes in I encountered this fun little error: “Error: TF400167: Installation failed for the package (patch_KB2815416) with the following status 0x80070643”
So to me there were two places to focus for figuring out the issue. 1) figure out what that KB file was and why it was failing to install, and 2) check out the status 0x80070643 and what causes it.
First I tried simply searching on the error message “TF400167 Installation failed for the package”. A lot of unrelated posts were returned, and a few potentially helpful ones in Japanese. Alas I do not speak or read that language. I did see a lot of references to needing to reboot the server in the posts I COULD read, which I did, to no avail.
Then I tried finding any references to KB2815416 and found exactly one, but alas it was not related to the issue I was having. Grrr.
I dug through the error logs, found the one related to that file (KB2815416), and noticed a host of errors including one saying “TFS 2012 timeout while waiting for worker process” and lots of warnings in the TFS logs about issues with the TFS application pool - [Info @22:09:26.796] Waiting for worker process for application pool Microsoft Team Foundation Server Message Queue Application Pool to stop. OK then, Guess I’ll restart the app tier and try to install Update 3 again. Still not working. FRACK!
So now I started focusing on the status being thrown back (0x80070643) and I start finding more helpful posts, though most of them related to much older releases of TFS. Still, gave it a go. So I’d already rebooted the server, restarted IIS, but then I noted Vicky’s comment about necessary permissions of the user. Well, I know the user I am logged in as SHOULD have all of those permissions, because it did back in January when I upgraded to TFS 2012, but we’ve been here before with other clients haven’t we? BINGO! The TFS admin account I was using had its rights revoked on the SQL Server machine and databases. It was a general TFS Admin account that was shared too, not just Angela-consultant account, so that could have been a big problem for someone. Why, WHY does this happen everywhere I go?!? So I added the TFS admin account back to the server admin group, and gave it sysadmin on the TFS databases, and finished up the upgrade nicely. Good thing I get paid by the hour, I guess.
The only other things I had to do was rebuild the data warehouse (through the admin console) and rebuild the cube to get reports back up and running. And even though I manually ran the scheduled backup job after the upgrade successfully, when the exact same job ran overnight it failed with error: “TFS database backup job failed with error: TF30040: The database is not correctly configured. Contact your Team Foundation Server administrator.System.Data.SqlClient.SqlException (0x80131904): Could not find stored procedure 'prc_TfsSetTransactionLogMark'.”
Next step is to recreate the backup job and hope that does the trick.