Sorry to confirm your insecurities but looks matter a lot. Beyond approaching and being normal they are the most important thing determining your value in the casual sex game. And no, being "well-groomed and toned" is not enough; the foundations of visual attractiveness - face and height - are fixed in stone. And while adding 30 lbs of lean muscle to your frame can up your attractiveness by a solid 2 or so points, it takes a timeline of many years to achieve that, so you have to accept that on a "day to day" basis you're stuck with what you have.
That said however, should you still worry about your looks? No. You should just pretend it doesn't matter and keep going. Why? First, because at this early stage you need to focus on building the ability to approach girls in general. Second, by obsessing too much, you will end up undervaluing yourself and psyching yourself out of girls who are actually in your league, as well the occasional girl who's outside of your league but has low looks standards. In the end with enough strong, focused approaches across the board, you can get the most accurate sense of what your league is.
That said however, should you still worry about your looks? No. You should just pretend it doesn't matter and keep going. Why? First, because at this early stage you need to focus on building the ability to approach girls in general. Second, by obsessing too much, you will end up undervaluing yourself and psyching yourself out of girls who are actually in your league, as well the occasional girl who's outside of your league but has low looks standards. In the end with enough strong, focused approaches across the board, you can get the most accurate sense of what your league is.