JPEG-Joint Photographic Experts Group

JPEGs (usually without the E), were designed with photographs in mind. They are capable of handling all of the colours needed and can blend colours without the need for dithering.
JPGs have a lossy way of compressing images. At a low compression value, this is largely not noticeable, but at high compression, an image can become blurry and messy. JPEGs accomplish this by 'forgetting' the details about the image and then hoping to guess at it later when it is displayed. The trick is to find the right marriage between quality and speed.
JPGs lack many of the special abilities of GIFs, like animation and transparency, but as I said, they really are only for photos. Simple graphics with large blocks of colour should not be saved as JPGs because the edges get all smudgy.
As a rule, consider this: use JPGs for photos, use GIFs for everything else. Of course, experimentation is the key to success.


GIF(Graphics Interchange Format)

The GIF format is the most popular on the Internet, mainly because of its small file size. It is ideal for small navigational icons and simple diagrams and illustrations where accuracy is required, or graphics with large blocks of a single colour. The format is loss-less, meaning it does not get blurry or messy. GIFs allow transparency, which is how you can have circular graphics that sit perfectly on whatever colour they're placed.
The 256 colour maximum is sometimes tight, and so it has the option to dither, which means create the needed colour by mixing two or more available colours. They can be interlaced too, which means a low-quality version of the picture appears first, then gets progressively drawn over by the final image. GIFs use a simple technique called LZW compression to reduce the filesizes of images by finding repeated patterns, but this compression never degrades the image quality.
GIFs can also be animated, one of its major draws. This comes in particularly handy for banner advertisements, which is where they get their most use, but they are equally suited to a flashing sign on your site.


PNG-Portable Network Graphics

PNGs are a new format invented specifically for the web, and are here to hopefully take over as the one all-dominant format. It has most of the benefits of both other formats, and fewer of their weaknesses. Its features include transparency, high colour palette size, alpha channels (no idea), and better-than-JPG compression. It also has a helpful 'creator information' thing, where you can attach your name to a picture if you are particularly proud of it. They can, of course be interlaced, and their palettes are saved in a more efficient way than GIFs.