Background There are no studies done to evaluate the distribution of mammographic breast density and factors associated with it among Pakistani women. Methods Participants included 477 women, who had received either diagnostic or screening mammography at two hospitals in Karachi Pakistan. Mammographic breast density was assessed using the Breast Imaging Reporting and Data System. In person interviews were conducted using a detailed questionnaire, to assess risk factors of interest, and venous blood was collected to measure serum vitamin D level at the end of the interview. To determine the association of potential factors with mammographic breast density, multivariable polytomous logistic regression was used. Results High-density mammographic breast density (heterogeneously and dense categories) was high and found in 62.4% of women. There was a significant association of both heterogeneously dense and dense breasts with women of a younger age group < 45 years (OR 2.68, 95% CI 1.60–4.49) and (OR 4.83, 95% CI 2.54–9.16) respectively. Women with heterogeneously dense and dense breasts versus fatty and fibroglandular breasts had a higher history of benign breast disease (OR 1.90, 95% CI 1.14–3.17) and (OR 3.61, 95% CI 1.90–6.86) respectively. There was an inverse relationship between breast density and body mass index. Women with dense breasts and heterogeneously dense breasts had lower body mass index (OR 0.94 95% CI 0.90–0.99) and (OR 0.81, 95% CI 0.76–0.87) respectively. There was no association of mammographic breast density with serum vitamin D levels, diet, and breast cancer. Conclusions The findings of a positive association of higher mammographic density with younger age and benign breast disease and a negative association between body mass index and breast density are important findings that need to be considered in developing screening guidelines for the Pakistani population.