source: New York Transit Museum
The 27th Street Depot was the first depot built by the New York & Harlem Railroad while they were building the railroad from Manhattan to Harlem. They acquired land on Fourth Avenue between 26th and 27th Street to set up their depot that included their offices, passenger terminal, the sidings, where trains were parked and, of course, the horse stables. In 1842, when 27th Street Depot was opened, regulations did not allow steam engines to enter the dense residential areas (in Manhattan they were south of 23rd Street), which means they were forced to use horse-drawn cars to reach their stops in the area.
The depot was later serving the New York & New Haven Railroad while the two companies were merging. In 1871 they have moved to a new facility on 42 Street, that later became one of the most iconic landmarks on Manhattan, the Grand Central Terminal (also known as Grand Central Station).
Sources:
1. Grand Central Terminal history website
2. New York Times, February 2013, Before There Was a 'Grand' in Central
source: Office for Metropolitan History