How do scientists define the height of a mountain on Mars when there is no sea to determine a sea level?

It is a convoluted process [even with sea levels.] Scientists mapped the mean Martian radius and the average atmospheric pressure and developed a new convention of zero elevation defined as the equipotential surface. They are basically using the mean radius of the planet as the surface height.

HAAT - height above average terrain is one type of height statistic.

We have maps of Mars, good enough to figure out the average elevation. That average is called "Mars datum surface". The science definition is (used to be) the elevation where the average atmosphere pressure is 6.105 millibars (the pressure of the triple point of water).

After 2001, the definition was changed a bit, to use the Mars Orbiter Laser Altimeter data.

They Measure from the Base of the Mountain.  You measure from the base of the mountains on Earth and you come up with a higher mountain than Everest.

Mon Kea in Hawaii is the tallest mountain on Earth, as measured from it's base.

They melted the polar caps when they were measuring them for the first time.

This web page answers your question. 

https://www.abc.net.au/science/articles/2013/08/12...

You have a good point .  Our mountains may, in fact, be taller . Those sneaky SOBs .