The perfherder comparison view shows a lot of numbers, but I have no idea if they're times (seconds? ms?) or frequency or scores, so I don't know which side is better.
Some details like "times in ms, lower is better" should be displayed.

Trying to find some info, https://wiki.mozilla.org/TestEngineering/Performance/Talos/Tests seems to imply that different tests have different kinds of measurements, so the explanation should be shown for each test.

Showing the measurement unit for the Base and New columns sounds like a good idea. I already did a PR for this: https://github.com/mozilla/treeherder/pull/7033 .

However there is already a feature in Perfherder that explains which side is better (it is activated if there are enough jobs), so I will not show another message regarding this. You can also get more information if you hover over the numbers.