It fucking is. It's not meant to fit any non-lesbian's aesthetic. It's a word with history and significance that shouldn't be appropriated.
Edit: this was never intended to be about bi women. Use of femme by no women is a nuanced and divisive issue. My opinion is that you can't be femme while trying to attract men, so it does include some bi women. Regardless, it's synonymous to many lesbians' lives and I don't feel it's wrong to say that femme is a lesbian word.