From the 3rd Inca Trail campsite, a gate opens at 5:30AM. If you leave then, you reach the Intipuku (Gate of the Sun overlooking Machu Picchu) by 7AM and get into Machu Picchu itself about 7:30. The daytrippers arrive by train after 10AM.
The 3,500 is mostly Inca-built stairs; maybe that's easier on knees than a steep downhill trail.
That has generally been my situation also. It depends upon how steep the downhill is. I've done OK on steady grades like the Inca Trail and Mt. Whitney. The last one that really bothered me was the Vivian Creek trail down San Gorgonio, which is 5,500 vertical in 7 miles. Whitney is 6,100 vertical in 11 miles.can hike uphill indefinitely, or so it seems. So far it's just the downhills that get me.