Taking all-optical packet-switched cross-connect architectures that are highly scalable as a starting point, we investigate how much buffering is required in optical packet routers. In particular, we investigate how the buffering depth scales with the number of ports of the router and how the packet loss and throughput depend on the network load. We treat the buffer as a queuing system with finite storage with a Markovian time evolution. This leads to an analytical expression for the buffer depth. Our results can act as a guideline for designing all-optical packet buffers.