We are currently in a golden age for wearables. Rapid proliferation driven by affordability and widespread availability has enabled human sensing at scale, and in naturalistic, day-to-day settings. An expanding array of onboard sensors unobtrusively captures different aspects of human life: from sleep patterns, daily activities and routines, and location, to bio-signals like heart rate and oxygen levels. As a result, there is increasing reliance on these devices for deriving objective views into human health and well-being. In particular, sensing and automatically recognizing activities, i.e., Human Activity Recognition (HAR), is critical for wide ranging applications. A variety of approaches have been developed to summarize recorded movements into representations that can discriminate between activities. Ranging from early hand crafting of statistical descriptors of movements, to, by the mid-2010s, learning task-relevant representations via end-to-end training. The focus of my dissertation is on this core component -- representations, and my contributions are five-fold: first, I show that end-to-end training is not the best option for all HAR scenarios, and instead, unsupervised representation learning can be more advantageous when considering factors relevant to wearable computing. This motivates a pursuit of developing more effective unsupervised methods. Second, I show how enhancing the task design of representation learning via self-supervised learning and cross-modal contrastive training yields powerful representations. Third, I develop an assessment framework to perform a well-rounded evaluation of self-supervised methods, to discover their areas of strength as well as limitations. Fourth, I demonstrate that a combination of self-supervised learning and vector quantization results in the discovery of atomic, recurring sub-movements that can be used for improved analysis of activities. Finally, I expand the scope of learned representations beyond the HAR task to a real-world health scenario -- of discovering longitudinal changes in movements in older adults, and investigate how such changes relate to their cognitive health.