It’s All Downhill From Here

............ it’s a very confusing saying - some people use it to mean a good thing ( like walking or cycling downhill is much easier ) while others say it in a negative way ( as in you’ve reached the peak, it’s never going to be this good again ).
When hubby says it he means it negatively whereas I always think of it as being positive ........ so which is it supposed to be?