Latte system comprises of a domain specific language (DSL) for specifying Deep Neural Networks (DDNs) and its high performance implementation. Users of Latte specify DNNs by constructing ensembles of neurons and applying connections between them. The Latte compiler synthesizes code from the DNN specification, performs a series of domain specific optimizations, and generates efficient code targeting high performance heterogeneous clusters of Intel multicore and manycore architectures. Unlike prominent library-based frameworks such as Caffe, Latte is not limited to a pre-specified list of network layers. In addition, it can perform cross-layer optimizations such as fusion that provide 3-6x speedup over Caffe for three recent ImageNet challenge winning models. Furthermore, Latte runtime manages the communication of data across nodes in a cluster and across host and accelerators in each node. Overall, the Latte system greatly improves the programmability, performance, and portability of DNNs.