well i think you are talking about two different things.
for an ornament, there are a certain number of beats specified by the meter so if you put more notes than fit, it has to come from somewhere... so "grace note" would be correct.
but when you mean like a cadenza or an ad libitum...
those do exist "out of time" and are sometimes written with small notes.
if it goes on long enough that there are metric feelings within it, you can even used dashed barlines inside this area that dont count towards the measure numbering.
if it's one instrument doing this, but otherwise playing in a group, the rest of the group would have a single measure with a fermata over a rest here, with the word "cadenza" to let them know they will be waiting for a while.
as for what to call them... its just "cadenza" or "ad lib"
or... are you asking what its called to look up how to make them in some notation software?
in finale, if there are just a few, they are "grace notes" and their time will not add up or affect the measure.
if its the cadenza kind, you can do it two ways.
- within one measure, use the tuplet tool to fit in as many notes as you need and then just change the note size for that section using the mass edit tool and the "change" tab. you can manually break and attach beams with the / button.
- use multiple measures and then make the measure lines dashed for that section and program the measure numbers to stop at the beginning of it and restart at the end. and again make the note size smaller. this one will take more formatting though, to hide the measures in the other parts when you extract.