Our recent work explores how multi-modality memory can improve medical agents for visual question answering. By storing and retrieving useful information from medical images, patient queries, and clinical reports, the agent can better understand complex cases and provide more accurate and context-aware responses. This direction shows the potential of memory-enhanced medical AI systems for supporting medical VQA tasks.