it's hard to say.
You can do your physical anytime... it's not like your physical is going to change much over a few weeks.
My guess is he is busy and trying to buy some time. But if you are in the queue they are tracking you and looking out to see when they can likely fit you into a job.
I'd keep on him. The physical is largely just a check in the block to get done. it doesn't have a huge impact unless you want to be SEAL, EOD, Seabee, Aircrew, SWCC.
The recruiting process is the last time you will have much say on your career so don't be afraid to voice your frustration. But be cool about it... Don't go off on the guy just say you're getting tired of waiting and want to get it done.