Well there is a small issue here in the sense..
At the start when you didn't sign the inventory beacuse you didn't agree- why didn't you raise the issue then- you could have got an intermediary (estate agent etc) to do the inventory for both parties.
I assume you (and the landlord) just ignored the inventory and got on with the tenancy agreement - didn't alarm bells go off then that this could cause potential issues ie ones being experienced now?
Did you take any photos etc of the place at the start?
If not you are in a difficult position as how can you prove that the carpet was not in the condition you state when you moved in - it now becomes your word against his and unfortunately he at the moment in time has your deposit.
All you can do is discuss the issue with him and resolve amicably - try to come to some compromise and ask him to be honest and reasonable.
when he does ask you for payment you as mentioned will dispute - he will ask you to prove you didn't damage the property - reverse it on him and ask him to prove that you did ie ask him to provide you with evidence ie photo's (taken just prior to the tenancy) or an inventory.
If he gets silly then say you never even saw an inventory even though you asked him for one - so you made your own but he didnl't sign (just write up a list of items and condition etc and backdate).
Some landlords are like this unfortunately and it seems you have found one - just learn ffrom it and move on
Not sure If i helped there or not but its my take on it anyway