A man was arrested after he allegedly stole a woman's purse that held a weeks-old kitten that he later abandoned.