You are not alone in this, and everyone is capable of making mistakes when it comes to keeping your pool chemicals balanced. Make sure you use a reliable test kit to measure your levels and remember that the ideal level for chlorine is 2.0 - 4.0 ppm. Here are a couple of different solutions to lowering the chlorine in your pool.
If your chlorine levels are slightly over 4.0 or even 5.0, the easiest thing to do is give it time. Don't allow anyone to swim and remove the source of chlorine from your pool. This means removing the chlorine floater from the water, or the chlorine tablet from the skimmer, and turning off the chlorinator. Whatever your source for chlorine is, stop it.
If it is a sunny day, your chlorine levels should lower fairly quickly due to the ultraviolet rays breaking down the chlorine. It can significantly reduce the levels in 2 - 3 hours, so make sure you are monitoring your levels regularly, so you know when to add the chlorine sources back in.
Another method for lowering chlorine levels is, removing some of the water from your pool and replacing it with fresh water. This is a slow process and also requires you to check all your levels, not just your chlorine after you have replaced the water.
If removing massive amounts of water from your pool is not an appealing option, consider adding a chlorine neutralizer. There are two main types of neutralizers, sodium thiosulfate, and hydrogen peroxide. If you choose to go the neutralizer route, make sure you purchase from a pool supply store, as the brown bottle in your medicine cabinet is not sufficient.
Finally, another option is to use an ultraviolet lamp to lower the chlorine. There are several different kinds available ranging from low to medium pressure. You can read more about the types of lamps, here.
Remember, keeping your pool chemicals balanced is part of the regular maintenance for your pool. If not maintained, high chlorine levels will damage your pool.