source: New York Transit Museum
Not exactly photo, but image of what the arrival of New York’s first train looked like. It was 26th November 1832 when first train, still horse-drawn car, from New York & Harlem Railroad company arrived to today’s Union Square in the middle of Manhattan.
There are two reasons why was it horse-drawn car - on one hand steam engines were already in use, but there were not enough of them to cover all the needs. The second reason were the regulations that did not allowed steam engines to enter dense residential areas, like Manhattan from 23rd Street south.
Sources:
1. Grand Central Terminal history website