The small cost of boxing
Chapter 2: Core foundations: building on C# 1: 2.3.4
Last updated: 5/7/2008
Even at the level of "hundreds of thousands" of boxing operations, in many cases performance of an application will barely be impacted. As a quick test, I wrote a program to box and unbox integers 100 million times. It took less than a second to run on my laptop (and scaled linearly as I increased the number of iterations).
Microbenchmarks are notoriously bad indicators of overall performance for various reasons, but it's worth being guided by the kind of scale involved here. Anything which can be done over a hundred million times per second doesn't need to be avoided too much - and certainly not to the extent of bending your design out of shape without hard evidence.