Team Foundation Server 2010 Performance Tuning – Lessons learned

The 2^nd day after this year’s MVP Summit in Redmond, just 14 hours after I got back from the U.S, I took another flight from Beijing to South China for a job. The job turned out to be one of the most challenging one that I’ve never done, pushing the performance limit of Team Foundation Server.

 

Figure: with Brian Harry @ MVP Summit 2011

Some background information first, the client is one of the biggest who also owns one of the biggest development team all around the world, with over 20,000 developers. The system we are developing has to be able to cope such amount of requests, sometime concurrent requests in a timely fashion. We are using TFS Object Model mainly to implement most of the functionalities, and we initially thought TFS should be able to handle the load without problem because Microsoft has been dogfooding TFS (http://blogs.msdn.com/b/bharry/archive/2009/09/04/dogfooding-tfs-2010-in-devdiv.aspx) for a long time in their even bigger and much more demanding environment, and TFS survived. However, like every story, things will never run as you expected; with all the data access, business logic and interfacing implementations on top of TFS Object Model, the initial end result was kind of disappointing.

Test Case	Before tuning	Target
Create Operation
No concurrency	1200 ms	200 ms
10 concurrency	2700 ms	500 ms
100 concurrency	>3000 ms	1500 ms
Query Operation
No concurrency	234 ms	200 ms
10 concurrency	565 ms	500 ms
100 concurrency	3000 ms	1500 ms

Figure: before tuning the performance was kind of disappointing

As you can see from above, all of the benchmarks are offsetting from the target, some of them are even 5-6 time bigger than the target. These results were collected on a SQL database (Tfs Collection DB) with around 500 million data entries.

Shocking? Isn’t it? Yes, I was when I first saw these. But like every story, there is a happy ending, so here is the result after our tuning.

Test Case	Target	After Tuning
Create Operation
No concurrency	200 ms	87 ms
10 concurrency	500 ms	399 ms
100 concurrency	1500 ms	2500 ms (single server) 1500 ms (NLB)
Query Operation
No concurrency	200 ms	28 ms
10 concurrency	500 ms	32 ms
100 concurrency	1500 ms	200 ms

Figure: The client is happy with the results, we made our target in each case. That said, I think we are at TFS 2010 limits

So, back to the topic, the lessons that we learned:

Lesson 1: Team Work

I put this as the top one like always, software developing is ever a one man job especially when you are dealing a complex system, like TFS. You need to know Windows Server 2008, IIS 7, SQL Server 2008 R2 and TFS web services and Object Model. You need expert in every part of these in order to combine them together to achieve your goal.

During the whole project, I got help from Microsoft, my boss and many other MVPs around the world; including Brian Harry (Microsoft Technical Fellow, father of TFS), Aaron Hallberg (TFS DevTeam), Tiago Pascoal (MVP), Adam Cogan (Microsoft Regional Director, MVP, my boss ;) ), Ramesh Rajagopal (DevDiv from MS Shanghai Dev Center), Julia Liuson (Manager of MS Shanghai Dev Center) and Yongming Yi (MS Technical Specialist) … and there is a long list that I couldn’t include all of them here.

To be short, if you want to do the job right, you need the right people. Having such a great team is essential for the end result.

Lesson 2: Performance testing as earlier as possible

We adopted scrum in the whole project and we have built in unit testing and load testing at the very 1^st sprint of our development cycle; however because of resource limitation, we were not able to achieve that much concurrency and gain enough hardware until we are at the very end of the whole project.

Although we were not able to do tunning along the way, the testing results have been given enough transparency to the client, so when the 1^st official test result on top of the staging hardware out, the client was not too surprise and gave us enough time and understanding. To me, transparency is always the biggest benefit of using agile methodologies; having your client support is sometime the best resource you can have in a project.

Another benefit of having performance testing in early was that I had enough time to contact helpful people and gain enough support beforehand.

Lesson 3: Avoid virtualization in demanding production system

To be continued …

ref: http://www.almnetworks.net/en/post/2011/09/30/Team-Foundation-Server-2010-Performance-Tuning-e28093-Lessons-learned.aspx