Mapping connections in the neonatal brain can provide insight into the crucial early stages of neurodevelopment that shape brain organisation and lay the foundations for cognition and behaviour. Diffusion MRI and tractography provide unique opportunities for such explorations, through estimation of white matter bundles and brain connectivity. Atlas-based tractography protocols, i.e. a priori defined sets of masks and logical operations in a template space, have been commonly used in the adult brain to drive such explorations. However, rapid growth and maturation of the brain during early development make it challenging to ensure correspondence and validity of such atlas-based tractography approaches in the developing brain. An alternative can be provided by data-driven methods, which do not depend on predefined regions of interest. Here, we develop a novel data-driven framework to extract white matter bundles and their associated grey matter networks from neonatal tractography data, based on non-negative matrix factorisation that is inherently suited to the non-negative nature of structural connectivity data. We also develop a non-negative dual regression framework to map group-level components to individual subjects. Using in-silico simulations, we evaluate the accuracy of our approach in extracting connectivity components and compare with an alternative data-driven method, independent component analysis. We apply non-negative matrix factorisation to whole-brain connectivity obtained from publicly available datasets from the Developing Human Connectome Project, yielding grey matter components and their corresponding white matter bundles. We assess the validity and interpretability of these components against traditional tractography results and grey matter networks obtained from resting-state fMRI in the same subjects. We subsequently use them to generate a parcellation of the neonatal cortex using data from 323 new-born babies and we assess the robustness and reproducibility of this connectivity-driven parcellation.