Older adults have increasing difficulty with prospective and retrospective memory, hindering their abilities to perform daily activities and posing stress on caregivers to continuously monitor them. Recent developments in Artificial Intelligence (AI) and sensor technologies offer an opportunity to create systems that monitor older adults, but also provide useful support to ensure their safety and independence. However, current solutions lack inclusion of both, are not context-aware or are not designed for older adults. This work discusses the development of a wearable, voice-based memory assistant, MemPal, that helps older adults live more independently at home and reduce caregiver burden. Using visual context from a wearable camera, the multimodal vision-language system creates a real-time automated activity log for memory support, thereby offering object retrieval assistance, recall of past actions, proactive safety reminders, and an activity summarizer. We report on a quantitative and qualitative study of (N=15) older adults within their own homes that showed improved performance of object finding with audio-based assistance compared to no aid and positive overall user perceptions on the designed system. We discuss future design guidelines to adapt these types of wearable systems to various older adults' needs.