A lipoma is not something to be left alone. This Navy career man was told for 17 years by his doctor that this growing mass did not require removal. “It was just a lipoma.” This thing slowly grew to the point at which when he retired this large lump was visible even in clothing.
The bottom line here is that even if a soft fleshy lump is not cancer, it can grow and produce a problem if ignored for long enough.
My baseline recommendation is that people get these things removed early. On Page 2 is an image of the tumor once it was removed.