Levels of human obesity have increased over the past 20 years worldwide, primarily due to changes in diet and activity levels. Although environmental changes are clearly responsible for the increasing prevalence of obesity, individuals may show genetic variation in their response to an obesogenic environment. Here, we measure genetic variation in response to a high-fat diet in a mouse model, an F16 Advanced Intercross Line derived from the cross of SM/J and LG/J inbred mouse strains. The experimental population was separated by sex and fed either a high-fat (42% of energy from fat) or low-fat (15% of energy from fat) diet. A number of phenotypic traits related to obesity and diabetes such as growth rate, glucose tolerance traits, organ weights and fat pad weights were collected and analysed in addition to serum levels of insulin, free fatty acids, cholesterol and triglycerides. Most traits are different between the sexes and between dietary treatments and for a few traits, including adult growth, fat pad weights, insulin and glucose tolerance, the dietary effect is stronger in one sex than the other. We find that fat pad weights, liver weight, serum insulin levels and adult growth rates are all phenotypically and genetically correlated with one another in both dietary treatments. Critically, these traits have relatively low genetic correlations across environments (average r=0·38). Dietary responses are also genetically correlated across these traits. We found substantial genetic variation in dietary response and low cross environment genetic correlations for traits aligned with adiposity. Therefore, genetic effects for these traits are different depending on the environment an animal is exposed to.