Given a directed graph, there exist a universal operator algebra and universal $C^*$-algebra associated to the directed graph. In this paper we give intrinsic constructions for these objects. We also provide an explicit construction for the maximal $C^*$-algebra of an operator algebra. We discuss uniqueness of the universal algebras for finite graphs, showing that for finite graphs the graph is an isomorphism invariant for the universal operator algebra of a directed graph. We show that the underlying undirected graph is a Banach algebra isomorphism invariant for the universal $C^*$-algebra of a directed graph.