Modelers are basically a programmer's rendition of what he thinks an amp should sound like.
... and how much he is able to do with the system provided. The problem with current modelers is simply the sampling rate, bandwidth, speed. It's kinda comparable to lower bitrate MP3, you can clearly hear that some stuff gets shaved off and the tone lacks certain "fullness". The audience will usually not hear it, but the player himself will notice that maybe some harmonics are missing, the dynamics are much lower, some notes are more "lifeless" and such - in direct comparison to your "good" real amp. I can hear or rather feel it immediately with the current "lower-end" modellers.
Impuls response and similar models are rather limited. Tube amps are not linear and it's really hard to squash so many parameters into a software model, because you'd need a supercomputer to calculate it all correctly. The main limitation is processing power and AD/DA conversion. To properly reproduce a 20KHz random signal (some harmonic which does "color" the basic tone for example), you will usually need at least 4-8 times the sampling rate, which is simply not there in the current lower-end hardware. Thus we have to work with approximations, which basically means that some stuff gets shaved off. I think it will take at least 2-3 more generations of hardware development to get "there". Like 512 bit conversion and 784KHz sampling, with a separate IR for every loudness/gain step etc. and every picking intensity step for each of those. Or a completely different model based on some other principals.