Text analysis tools are increasingly using large language models (LLMs) and knowledge graphs (KGs) to support sensemaking of document datasets too large to read. Yet opportunities for incorporating them into visual document analysis, and how they might affect sensemaking, remain under-explored. In response, we developed a novel visual analytics tool, VisPile, that embeds LLMs and KGs in an interactive document piling interface. With VisPile, we studied how users leveraged LLMs and KGs in conjunction with document piles to support sensemaking of 845 documents. We first conducted a usability study with students (n=17). We discovered LLMs and KGs uniquely support sensemaking and, in conjunction with piles, support novel interaction patterns. We then conducted the same study with intelligence analysts (n=6), providing support for patterns observed. Finally, we synthesized lessons learned about how to effectively incorporate LLMs and KGs into future visual analytics tools for supporting sensemaking during document analysis.